CMS Snapshot Analysis - Compute Snapshot - Fatal Error - Error while executing procedure

*Purpose

This page will help you to troubleshoot the Fatal Error message "Error while executing procedure" that occurs during the Compute Snapshot step when you take a Snapshot.

*Observed in CAST AIP

Release	Yes/No
8.3.x
8.2.x
8.1.x
8.0.x
7.3.x

*Observed on RDBMS

RDBMS	Yes/No
Oracle Server
Microsoft SQL Server
CSS3
CSS2

Action Plan

Open CAST-MS log file that is located in the path described in the page : CAST Management Studio - Information - How to find logs - CastMS Log File
Check if the fatal error message is due to insufficient remaining disk space. The following two errors occurs if this is the case.

ERROR: could not write to hash-join temporary file: No space left on device.
Or ERROR: could not extend file "base/xxxx/XXXX.XX": No space left on device.
These two error messages mean that there is not enough disk space on the Database Server. As a solution, free more disk space and make sure that you are following the prerequisites described in the documentation: Deployment - sizing.

If you see below procedures involved in the error log then, issue is related to OMGTDM extension. In this case you can deactivate this extension or deactivate the metrics related to this extension followed by snapshot.

 ERR: 2021-06-03 14:50:32: Error while executing Procedure
com.castsoftware.java.InternalException:Error while executing Procedure
com.castsoftware.connection.JdbcConnection._executeProcedure(JdbcConnection.java:1335)
com.castsoftware.connection.JdbcConnection.executeProcedure(JdbcConnection.java:1267)
com.castsoftware.dssengine.LocalSite.callSnapshotInitializationProc(LocalSite.java:2929)
com.castsoftware.dssengine.LocalSite.consolidateSnapshot(LocalSite.java:2636)
com.castsoftware.dssengine.LocalSite.importAndConsolidate(LocalSite.java:2066)
com.castsoftware.dssengine.Engine.computeSnapshot(Engine.java:582)
com.castsoftware.pmc.dashboard.dssengine.RunDSSEngine.documentExecute(RunDSSEngine.java:183)
com.castsoftware.mda.impl.MemoryTask.execute(MemoryTask.java:86)
com.castsoftware.java.treatment.Task.run(Task.java:380)
com.castsoftware.java.treatment.Task.runChildren(Task.java:590)
com.castsoftware.pmc.actions.snapshot.SnapshotGenerator.documentExecute(SnapshotGenerator.java:115)
com.castsoftware.mda.impl.MemoryTask.execute(MemoryTask.java:86)
com.castsoftware.java.treatment.Task.run(Task.java:380)
com.castsoftware.java.treatment.Task.runChildren(Task.java:590)
com.castsoftware.mda.impl.MemoryAction.execute(MemoryAction.java:272)
com.castsoftware.pmc.actions.snapshot.GenerateSnapshot.execute(GenerateSnapshot.java:612)
com.castsoftware.mda.impl.MemoryAction.topExecute(MemoryAction.java:265)
com.castsoftware.mda.impl.MemoryAction$MemoryActionTask.documentExecute(MemoryAction.java:383)
com.castsoftware.mda.impl.MemoryTask.execute(MemoryTask.java:86)
com.castsoftware.java.treatment.Task.run(Task.java:380)
com.castsoftware.java.treatment.execution.TaskExecutor$TaskMainThread.run(TaskExecutor.java:103)
java.lang.Thread.run(Thread.java:748)
ERROR: could not write block 40321 of temporary file: No space left on device
Where: SQL statement "insert into APM_WORKTABLE (SNAPSHOT_ID, OBJECT_ID, RESULT, RESULT_2 )
select I_SNAPSHOT_ID,
x.PARENT_ID,
count (x.OBJECT_ID),
sum (x.METRIC_NUM_VALUE)
from
(select distinct mp.PARENT_ID AS PARENT_ID, dmr.OBJECT_ID AS OBJECT_ID, dmr.METRIC_NUM_VALUE AS METRIC_NUM_VALUE
from TMP_LINK_INFO tli
join APM_MODULE_PARENTS mp
on mp.MODULE_ID = tli.PREVIOUS_OBJECT_ID
and MP.SNAPSHOT_ID = I_SNAPSHOT_ID
join DSS_WORK_METRIC_RESULTS dmr
on dmr.OBJECT_ID = tli.NEXT_OBJECT_ID
) x
group by x.PARENT_ID"
PL/pgSQL function omgapm_central_result_std_d(integer,integer,integer,integer) line 41 at SQL statement
SQL statement "select OMGAPM_CENTRAL_RESULT_STD_D(I_SNAPSHOT_ID,I_METRIC_PARENT_ID,I_METRIC_ID,I_METRIC_VALUE_INDEX)"
PL/pgSQL function omgapm_central_result_sum_d(integer,integer,integer,integer) line 10 at SQL statement
PL/pgSQL function omg_atdm_propagate_results(integer,integer,integer,integer) line 44 at assignment
PL/pgSQL function omg_atdm_result_aggregate(integer,integer) line 12 at assignment
PL/pgSQL function omg_atdm_all(integer) line 35 at assignment
SQL statement "select OMG_ATDM_ALL (2)"
PL/pgSQL function dss_execute_named_side_process(integer,character varying,integer) line 64 at EXECUTE
SQL statement "select DSS_EXECUTE_NAMED_SIDE_PROCESS(I_SNAPSHOT_ID, 'ALL', I_SIDE)"
PL/pgSQL function dss_execute_side_process(integer,integer) line 5 at SQL statement
PL/pgSQL function dss_cleanup_compute_metric(integer) line 6 at assignment
org.postgresql.util.PSQLException:ERROR: could not write block 40321 of temporary file: No space left on device
Where: SQL statement "insert into APM_WORKTABLE (SNAPSHOT_ID, OBJECT_ID, RESULT, RESULT_2 )
select I_SNAPSHOT_ID,
x.PARENT_ID,
count (x.OBJECT_ID),
sum (x.METRIC_NUM_VALUE)
from
(select distinct mp.PARENT_ID AS PARENT_ID, dmr.OBJECT_ID AS OBJECT_ID, dmr.METRIC_NUM_VALUE AS METRIC_NUM_VALUE
from TMP_LINK_INFO tli
join APM_MODULE_PARENTS mp
on mp.MODULE_ID = tli.PREVIOUS_OBJECT_ID
and MP.SNAPSHOT_ID = I_SNAPSHOT_ID
join DSS_WORK_METRIC_RESULTS dmr
on dmr.OBJECT_ID = tli.NEXT_OBJECT_ID
) x
group by x.PARENT_ID"

Check if the links between the application, module and technology exist. See the following page for assistance: CMS Snapshot Analysis - Information - How to check on and create links between application module and technology
1. If links needed to be created, then run the snapshot again and see if the problem has been resolved.
If the error from CAST-MS log is same as mentioned below then please perform the optimize and vacuum on the Knowledge and the central bases and rerun the snapshot.
If the above points did not help you, identify the function / stored procedure that is involved in the failure. The solution would be to deactivate this stored procedure if it is used by a quality rule:
1. In CAST MS log file, search for the function that is failing by looking for the statement that starts with "Where" as shown in the below screenshot:
  In the above screenshot, we can see that the stored procedure / function that is failing is : tcc_fp_usr_df_group_rule.
2. Check if the error is due to a stored procedure / function that has been customized for transaction calibration: Refer to the page Transaction calibration, section Customizing Calibration that lists the functions that can be customized. If it is one of these functions, you have to review it.
3. If there is no Function name / Stored procedure in the Cast-MS log file, check which one is the last Function / Stored Procedure with no "End" that has run in the DSS_HISTORY in your Local and Central Databases:
  The compute snapshot is a set of functions and stored procedure (SP) that are run in local database and central database. Most of the Stored Procedure / Functions are logged in the DSS_HISTORY table of the Local and Central Databases.
```
select * from DSS_HISTORY
order by action_date
```
  In DSS_HISTORY, when an action has failed and stopped the Compute Snapshot, the Stored Procedure / Function has a Start and No End as shown above.
  In the above example, the name of the function with a Start but no End is: DIAG_SCOPE_CACS001.
4. If all the functions / stored procedure have an End in the DSS_HISTORY table, check in PostgreSQL Log file the function that has failed. The Postgresql Log is in CAST Management Studio - Information - How to find logs - Getting the CSS Log File.

Check if the Stored Procedure / Function you have identified is for Quality Rule. Check this by running the following query in the Local Database

select coalesce(tech.TECHNO_TYPE_NAME, 'AllTechno' ) as technology_name ,
       coalesce(PT.OBJECT_TYPE_ID,0) as technology_id ,
       T.METRIC_ID ,T.METRIC_NAME,
       coalesce( p.IdProp , 0 ) as Property_ID,     
       coalesce(p.Dsc , ' ') as Property_Name,     
       coalesce(T.SCOPE_ID, 0 ),     
       coalesce(S.SetName,'NONE'),     
       TT.METRIC_SCOPE_PROCEDURE_NAME as detailed_proc_name,     
       VT.METRIC_VALUE_PROCEDURE_NAME as total_proc_name
from DSS_METRIC_TYPE_TREES TT , DSS_METRIC_VALUE_TYPES VT,
     DSS_METRIC_TYPES T      
      left outer join DSS_METRIC_PARAM_TYPES PT  on PT.METRIC_ID   = T.METRIC_ID                                                
                                                and PT.PARAM_INDEX = 1      
      left outer join DSS_TECHNO_DISPLAY_VW tech on tech.TECHNO_TYPE_ID = PT.OBJECT_TYPE_ID      
      left outer join ( select distinct d.Dsc , p1.IdProp                          
                          from PropAttr p1 , PropAttr p2 , ObjDscRef d                         
                         where p1.IdProp  = p2.IdProp                          
                          and p1.AttrNam = 'INF_TYPE'                            
                          and p1.IntVal  = d.InfTyp                           
                          and p2.AttrNam = 'INF_SUB_TYPE'                            
                          and p2.IntVal  = d.InfSubTyp ) p   on p.IdProp = T.PROPERTY_ID      
      left outer join ( select SetId , SetName                          
                         from  SET_Definitions         ) S   on S.SetId = T.SCOPE_ID
where T.METRIC_GROUP        = 1
  and T.METRIC_TYPE != 0  -- Not Deleted
  and TT.METRIC_ID          = T.METRIC_ID + 1
  and VT.METRIC_ID          = T.METRIC_ID
  and VT.METRIC_VALUE_INDEX = 2
  and exists ( select 1 from DSS_METRIC_TYPE_TREES x where x.METRIC_ID = T.METRIC_ID)
  and (TT.METRIC_SCOPE_PROCEDURE_NAME = '<Stored Procedure / The Function>'    -- For instance : DSS_PROPAGATE_RESULTS_TO_APP
       or VT.METRIC_VALUE_PROCEDURE_NAME = '<Stored Procedure / The Function>')  -- For instance : DSS_PROPAGATE_RESULTS_TO_APP

The Result of the Query contains the metric / Quality Rule name.

If the query returns at least a row, it means that the stored procedure that is failing is executed to compute results of specific rule. If this is the case the Solution is to deactivate the corresponding quality rule as explained in the page Assessment Model parameters and exclusions using CAST Management Studio at the section Activating/Deactivating a Metric/Quality Rule, then run the Snapshot again by skipping analysis.
Deactivating a quality rule will prevent the Metric / Quality Rule to be calculated. Then, no violations related to this Quality Rule will be shown in the Dashboard.
If there is no row, it is not a quality rule that has failed.

If the snapshot fails with below showed error as "Error while executing Update: Drop table PMC_TMP_SUBSET3" then, you can relaunch the snapshot skipping analysis. There is no précised root cause for this and relaunching snapshot will help in resolving this issue.
Check if the error message is already handled by the below pages
If none of the above points help you to solve the issue, contact Cast Technical Support and provide the following Relevant Input to reproduce the issue and find out if the procedure responsable on the Fatal error can be updated.

Relevant Input

Sherlock (CAST Support Tool (CST) - alias Sherlock) : Options to select are Export Logs, Export Configuration Files, and Export Database.
PostrgreSQL log file dating from the fatal error issue.
Explanation about the problem, screenshots showing the issue.

Notes/Comments

Related Pages