Configuring Neo4j buffer size

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

In some situations you may face an error when attempting to import applications into CAST Imaging (see Export and import the Application data) as follows:

ERROR Import: error while importing objects, aborting import error=objects: failed to run queries to import objects: run: error running Creation of Object to Property Links cypher for APPLICATION_NAME:

exec: unable to run cypher query: Neo4jError: Neo.DatabaseError.Statement.ExecutionFailed (Tried to read a field larger than buffer size 10000000.

A common cause of this is that a field has an unterminated quote and so will try to seek until the next quote, which ever line it may be on. This should not happen if multi-line fields are disabled, given that the fields contains no new-line characters. This field started at /var/lib/neo4j/import/csv/neo4j/APPLICATION_NAME_is_cfg_object_properties.csv:6891)

This error indicates that a property in the exported data has exceeded the default Neo4j maximum of 10MB - and that the application data failed to be imported correctly.

Configuring front-end web server to reverse proxy for multiple CAST Imaging instances

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

It is possible to configure a front-end web server (such as Nginx or Apache) to act as a reverse proxy for accessing multiple installed instances of CAST Imaging. For example your organization may have multiple instances of CAST Imaging and would like to access them with a simple URL as follows:

Prerequisites

Install front-end web server

✅

Install a front-end web server that can handle the proxy pass mechanism. CAST recommends Nginx installed on Linux, however, any other web server that can handle this mechanism can be used, for example Apache.

Configuring http headers for source code viewer

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

In some situations you may attempt to view the source code of an object but the source code is never displayed despite the soure code data source being configured correctly and other objects displaying their source code without issue. On further investigation, a “Request header is too large” error is also seen in the sourcodeservice.log file, as follows:

2024-05-15 09:12:13.289 INFO 5200 --- [nio-9980-exec-5] o.apache.coyote.http11.Http11Processor : Error parsing HTTP request header Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level.java.lang.IllegalArgumentException: Request header is too large

This error indicates that CAST Imaging was unable to display the source code because the source code of the object in question is too large and exceeds the size authorized in the API call between the CAST Imaging UI and the back-end database. It is possible to update the authorized file size to allow the source code of larger objects to be displayed. This is described below.

Configuring huge transaction or data call graph threshold

Mon, 01 Jan 0001 00:00:00 +0000

You may see the following message (Huge Data Call Graph/Transaction, content is not available) when looking through the list of available transactions or call graphs:

This indicates that during the data export/import process, the exporter tool identified that the transaction/data call graph contained more than 10,000 objects, and therefore stopped the processing action for this specific item. In other words, the detail of this transaction/data call graph has not been exported and is therefore unavailable.

Configuring import batch size

Mon, 01 Jan 0001 00:00:00 +0000

Note

This information is only valid for ≥ 2.0.0-beta5.

Info

Summary: how to configure application data import batch size to improve import performance when dealing with large amounts of data.

Introduction

During the application data import process (see Export and import the Application data), CAST Imaging is configured to import data in a batch of 1000 rows and then perform a commit. If you have allocated additional memory to Neo4j (see Optimize Neo4j memory allocation) it is possible to increase the default import batch size so that data commits are performed less, therefore increasing overall import performance.

Configuring the audit trail

Mon, 01 Jan 0001 00:00:00 +0000

Info

Summary: how to configure the audit trail.

Introduction

Out of the box in ≥ 2.0.0, CAST Imaging is configured automatically to log user actions via an audit trail feature. In previous releases, the audit trail is disabled out of the box and will need manually enabling (see below). Audit trail log files are located here:

Microsoft Windows traditional installer:
%PROGRAMFILES%\CAST\ImagingSystem\logs\audit-trail.log

Docker Installer extension:
logs\audit-trail.log

What is logged in the audit trail?

Any of the following GET/PUT/DELETE API actions are logged in the audit trail log:

Configuring the custom aggregation node limit

Mon, 01 Jan 0001 00:00:00 +0000

Info

Summary: how to configure custom aggregation node limit.

Introduction

Out of the box in ≥ 2.17.1, CAST Imaging is configured automatically to limit the number of nodes in a custom aggregation to 100,000 (one hundred thousand) when the custom aggregation is configured to use an existing CAST supplied aggregation (such as CAST Taxonomy, Module, Service, etc.). If the custom aggregation exceeds this value, then an error is displayed when the aggregation is created:

Customizing session timeout

Mon, 01 Jan 0001 00:00:00 +0000

Info

Summary: how to customize the session timeout feature.

Overview

Session timeout functions as follows:


1	If there is no user activity (API calls) for a default period of 30 minutes, the login session will timeout. When this occurs, the following alert message will be displayed:
2	By default a notification period of 20 seconds will then start before the session fully expires and the user will need to re-login.
3	After a default period of 20 hours, a session will timeout regardless of whether there is continuous user activity during this period.

Customizing the timeout

You can alter all session timeout periods by editing the following files. On completion, save the filesand ensure that you restart all CAST Imaging services/containersso that the changes are taken into account.

Dealing with failed to get graph data error

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

When a module contains a very large number of objects (for example you have configured one module for the entire application - a Full Content module - see Application - Config - Modules) CAST Imaging can sometimes fail to load the module, i.e. it will timeout before all the objects are correctly loaded. This issue manifests itself with the following error “Failed to get graph data” or “Failed to get Transaction list”:

Export data from one CAST Imaging instance and import into another

Mon, 01 Jan 0001 00:00:00 +0000

Info

Summary: Instructions explaining how to export application data from one CAST Imaging and import it into another - i.e. to transfer data between CAST Imaging instances.

Introduction

If you have multiple CAST Imaging instances and would like to be able to export application data from one CAST Imaging and import it into another, you can do so following the instructions below. This feature is mainly aimed at supporting pilots and assessments and also for System Integrators who support CAST Imaging for multiple customers. The tool uses a two-step action:

Optimize Neo4j disk usage - cleanup and disable JVM crash dumps

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

If Neo4j crashes, the Java Virtual Machine will automatically create a hs_err_pidxxxx.mdmp crash file in the Neo4j_data location. These .mdmp files can be extremely large and can therefore consume a significant amount of disk space. To resolve this issue there are two options:

Clean up the .mdmp files

The .mdmp files can safely be deleted from disk since they are not used or required by CAST Imaging.

Disable JVM crash dumps in neo4j.conf file

If you want to prevent the creation of the .mdmp crash dump files, edit the neo4j.conf file located here:

Optimize Neo4j disk usage - transaction log retention

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

By default Neo4j is set to retain transaction logs (used for database backup operations) for 7 days, however, transaction logs can grow significantly in size, increasing the overall size of the graph.db file and therefore reduce overall performance. If you are facing disk space/performance issues due to the size of the graph.db file, you can change the transaction log retention time.

See https://neo4j.com/docs/operations-manual/current/configuration/transaction-logs/ for more information on how to set different values.

Optimize Neo4j memory allocation

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

By default Neo4j is set to use the following RAM:


CAST Imaging ≥ 2.5.0	8GB of RAM memory (initial and maximum)
CAST Imaging ≤ 2.4.x	3GB of RAM memory (initial and maximum)

If you are facing performance issues, it is possible to modify the allocated memory. See also https://neo4j.com/docs/operations-manual/current/performance/memory-configuration/#_considerations for more information.

How do performance issues manifest themselves?

In general, a sure sign that you need to modify the RAM allocated to Neo4j is when objects fail to load in CAST Imaging - particularly if there are a large number of objects to display - this is because the query used to fetch the objects for display is not completing before a timeout occurs. This timeout will manifest itself as a 504 (gateway timeout) error in the browser. Allocating additional RAM to Neo4j can help reduce the time required to complete the query.

Reconfiguring the front end service to use an alternative unsecure port

Mon, 01 Jan 0001 00:00:00 +0000

Info

Summary: How to change the port on which the front end service runs (CAST Imaging System - imaging-fe Windows Service or nginx Docker container).

Introduction

CAST Imaging front end runs by default on port 80 in ≤ 2.0.0-beta5 or port 8083 (or the first free port available after 8083) in ≥ 2.0.0-beta6. If this port is already used by another pre-existing service you can reconfigure the service to use an alternative (for example 8080).

Reconfiguring the front end service to use a secure port

Mon, 01 Jan 0001 00:00:00 +0000

Info

Note that you should always check that the new port you intend to use is not ALREADY being used by some other service.
See also Reconfiguring internal services to run on a custom port if you wish to reconfigure any of the internal services to run on another custom port.

Edit nginx.conf

CAST Imaging front end runs by default on port 80 in ≤ 2.0.0-beta5 or port 8083 (or the first free port available after 8083) in ≥ 2.0.0-beta6. To reconfigure the service to run over a secure connection (for example port 443), edit the following file:

Reconfiguring internal services to run on a custom port

Mon, 01 Jan 0001 00:00:00 +0000

Info

Summary: How to change the port on which the internal services are configured to run when using a traditional Microsoft Windows installation (not involving Docker).

Introduction

Info

Note that you should always check that the new port you intend to use is not ALREADY being used by some other service.
See also the following documentation if you wish to reconfigure the front-end service (port 80) to run on port 443 or another custom port:
- Reconfiguring the front end imaging-fe Windows service to use an alternative unsecure port
- Reconfiguring the front end service to use a secure port

CAST Imaging has several internal services that are preconfigured to run on specific ports as detailed below:

Performance issues - tips

Mon, 01 Jan 0001 00:00:00 +0000

Info

Summary: some tips/changes to make to improve the performance of your CAST Imaging instance.

Introduction

CAST Imaging is a CPU and RAM intensive application, and this is especially true if you are importing large and complex applications. The following page lists some configuration changes that may help improve performance of CAST Imaging while importing and investigating applications.

Display performance

Optimize RAM allocated to CAST Imaging

By default Neo4j is set to use the following RAM:

CAST Imaging - Advanced configuration on Documentation - V2

Configuring Neo4j buffer size

Introduction

Configuring front-end web server to reverse proxy for multiple CAST Imaging instances

Introduction

Prerequisites

Configuring http headers for source code viewer

Introduction

Configuring huge transaction or data call graph threshold

Configuring import batch size

Introduction

Configuring the audit trail

Introduction

What is logged in the audit trail?

Configuring the custom aggregation node limit

Introduction

Customizing session timeout

Overview

Customizing the timeout

Dealing with failed to get graph data error

Introduction

Export data from one CAST Imaging instance and import into another

Introduction

Optimize Neo4j disk usage - cleanup and disable JVM crash dumps

Introduction

Clean up the .mdmp files

Disable JVM crash dumps in neo4j.conf file

Optimize Neo4j disk usage - transaction log retention

Introduction

Optimize Neo4j memory allocation

Introduction

Reconfiguring the front end service to use an alternative unsecure port

Introduction

Reconfiguring the front end service to use a secure port

Edit nginx.conf

Reconfiguring internal services to run on a custom port

Introduction

Performance issues - tips

Introduction

Display performance

Optimize RAM allocated to CAST Imaging