Created by James Hurrell, last modified on Jun 14, 2023

Introduction

Before you can start to use CAST Imaging, you will need to:

  • export the Application data that will be displayed in CAST Imaging from a CAST Analysis Service schema
  • import the CSV format data (resulting from the export process) into the CAST Imaging

An Automation tool is provided with the CAST Imaging that can run both actions in one go, or each action separately (from 2.1.2-funcrel of CAST Imaging ETL). The tool can also be used to re-import new data for an existing Application - in this case a data merge is performed - so new objects may be visible and existing objects may be removed or updated. See instructions for your operating system below.

  • Note that this tool is only available for CAST Imaging:
    • ≥ 1.8.x (Microsoft Windows via traditional installer)
    • ≥ 1.11.x (Docker for Linux/Windows)
  • Import actions can take a long time for very large applications. Please avoid restarting services during an application import - this may cause corruption in the Neo4j database.
  • Please ensure you are aware of the free disk space requirements as detailed in Performance issues - tips, particularly if your application is very large.

Microsoft Windows installed via traditional installer

Step 1 - obtain the tool(s)

The tool is shipped with CAST Imaging and is also available as a standalone download - see CAST Imaging ETL (CAST recommends using this where possible to benefit from the most recent updates and bug fixes). The tool MUST be executed from its own folder (i.e. create a batch file in the same folder as the .exe) otherwise it will fail to run correctly:

%PROGRAMFILES%\CAST\ImagingSystem\imagingservice\etl-automation\etl-automation.exe

Step 2 - configure .json file

Edit the .json file to match your environment. A template .json file is provided here:

%APPDATA%\CAST\ImagingSystem\imagingservice\etl_config.json

.json file syntax:

PORT

Enter the port number for the target CAST Storage Service/PostgreSQL instance:

CSS42284
CSS32282
CSS22280
HOST

Enter the network host name of the server on which the CAST Storage Service/PostgreSQL instance is located. This CAST Storage Service/PostgreSQL instance will host the Analysis Service schema which contains the required Application data.

localhost can be used if the CAST Storage Service/PostgreSQL instance is located on the current server.

SCHEMA

Name(s) of the Analysis Service schema containing the Application you would like to export/import into CAST Imaging. Three options are available:

  • single schema = "SCHEMA" : "myapplication_local",
  • multiple schemas = "SCHEMA" : ["myapplication_local", "myapplication2_local"],
  • all schemas on the CAST Storage Service/PostgreSQL instance =  "SCHEMA" : "all",

You can view the name of the Analysis Service schema as follows:

OUTPUTDIR

Full path to a temporary output folder in which to store the exported items - these items are not removed when the process is completed, i.e. a record of the export is retained. This path needs to be escaped:

"OUTPUTDIR" : "D:\\Imaging\\Output\\",
DATABASEAllows you to specify the CAST Storage Service/PostgreSQL database in which the schemas listed in SCHEMAS are stored. By default this is set to "postgres", so unless you have manually changed this, there is no need to specify it in config.json.
EXPORT
Available from CAST Imaging ETL 2.1.2-funcrel and CAST Imaging 2.2.0-funcrel.

Choose whether you want the tool to perform the export step or not, using either true or false:

  • If you set it to false (and IMPORT set to true) then applications listed in the APPS parameter will be imported into CAST Imaging. The exported ZIP files containing the relevant application data must already be present in the location defined in OUTPUTDIR.
  • If you set it to true (and IMPORT set to false) then applications listed in the APPS parameter will be exported to ZIP file and stored in the location defined in OUTPUTDIR.
  • If you set it to true (and IMPORT set to true), then the two step export/import action will be performed.
IMPORT
Available from CAST Imaging ETL 2.1.2-funcrel and CAST Imaging 2.2.0-funcrel.
  • If you set it to false (and EXPORT set to true) then applications listed in the APPS parameter will be exported to ZIP file and stored in the location defined in OUTPUTDIR.
  • If you set it to true (and EXPORT set to false) then applications listed in the APPS parameter will be imported into CAST Imaging. The exported ZIP files containing the relevant application data must already be present in the location defined in OUTPUTDIR.
  • If you set it to true (and EXPORT set to true), then the two step export/import action will be performed.
APPS
Available from CAST Imaging ETL 2.1.2-funcrel and CAST Imaging 2.2.0-funcrel.

Names of the applications you want to export/import. Use the following syntax:

  • single application = "APPS" : "myapp1",
  • multiple schemas = "APPS" : ["myapp1", "myapp2"],
  • all schemas on the CAST Storage Service/PostgreSQL instance =  "APPS" : "all",
IMAGING_HOSTNAME

Enter the network host name/IP address of the server on which CAST Imaging has been installed, e.g.: localhost.

Note that if your CAST Imaging instance is running in SSL mode, then you should enter the same hostname as specified in the SSL certificate, otherwise the import may fail.
IMAGING_PORT


≥ 2.5.x

This corresponds to the port on which the front end of CAST Imaging is running. By default the .json file will contain 8083 (default port), but if you have customized the port (for example you are using https protocol) please ensure that you change it.

≤ 2.4.x

This corresponds to the port on which the CAST Imaging System - imaging-ETL Windows Service runs. By default this is set to 9001 and therefore the etl_config.json will be prefilled with 9001. Unless you have changed the port for the CAST Imaging System - imaging-ETL service, please leave this as is.

IMAGING_DOMAIN

≥ 2.5.x

This corresponds to the tenant into which you would like to import the application. If you have not created any tenants, then please leave this option empty - this will import the application into the "default" tenant. Otherwise, please specify the tenant name:

  • "IMAGING_DOMAIN": "my_tenant",

See Admin Center - Multi tenant panel for more information about tenants.

IMAGING_ADMIN_USER

≥ 2.5.x

This option is no longer required and is replaced by TOKEN.

≤ 2.4.x

Name of a user with the admin role (see Admin Center - Users panel).

TOKEN

≥ 2.5.x

This field requires an API Key - this key can be generated when logged in to CAST Imaging and is valid for the current user - therefore, ensure that the user has permission to import Applications (typically this is a user with the ADMIN or SUPER ADMIN roles). See Admin Center - API key generation and usage for more information about generating an API Key.

Note that token values from previous releases will no longer function.

≤ 2.4.x

Enter the value for the option proxy.config.token.values located in the file:

%APPDATA%\CAST\ImagingSystem\login\application.properties

The value provided in the .json file supplied with CAST Imaging should already match, but please check that this is the case. For example:

USERNAMEEnter the CAST Storage Service/PostgreSQL user login (e.g.: operator) for the instance on which the applications you want to export/import are stored.
PASSWORDEnter the CAST Storage Service/PostgreSQL password corresponding to the user entered in the USERNAME filed, e.g.: CastAIP.
LOG_PATH

Full path to a folder in which the process logs will be stored - the path does not need to exist. This path needs to be escaped as shown below. A file called etl_automation_<guid>.log will be created in the location:

"LOG_PATH" : "D:\\Imaging\\Output\\Log\\"

Example .json file to run export and import steps in one go:

≥ 2.5.x
{
"PORT" : 2282,
"HOST" : "MY_CSS_SERVER",
"SCHEMA" : ["myapplication_local", "myapplication2_local"],
"OUTPUTDIR" : "D:\\Imaging\\Output\\",
"DATABASE": "",
"EXPORT" : true,
"IMPORT": true,
"APPS" : ["myapp1", "myapp2"],
"IMAGING_HOSTNAME" : "localhost",
"IMAGING_PORT" : 8083,
"IMAGING_DOMAIN": ["my_tenant"],
"TOKEN" : "umaKngIi.lW5OIlwlnLQ5Bc6Tn8bzdcHbrwLd",
"USERNAME" : "operator",
"PASSWORD" : "CastAIP",
"LOG_PATH" : "D:\\Imaging\\Output\\Log\\"
}
≤ 2.4.x
{
"PORT" : 2282,
"HOST" : "MY_CSS_SERVER",
"SCHEMA" : ["myapplication_local", "myapplication2_local"],
"OUTPUTDIR" : "D:\\Imaging\\Output\\",
"DATABASE": "",
"EXPORT" : true,
"IMPORT": true,
"APPS" : ["myapp1", "myapp2"],
"IMAGING_HOSTNAME" : "localhost",
"IMAGING_PORT" : 9001,
"IMAGING_ADMIN_USER" : "admin",
"TOKEN" : "D5ED6A406775FC71B8D2A978883E8ED4",
"USERNAME" : "operator",
"PASSWORD" : "CastAIP",
"LOG_PATH" : "D:\\Imaging\\Output\\Log\\"
}

Step 3 - Run the tool

To run the tool use the following syntax, this will perform the extraction and run the import all in one go:

etl-automation.exe -CONFIG <path_to_.json>

This tool MUST be executed from its own folder otherwise it will fail to run correctly. Therefore if you prefer to create a batch file to run this tool, the .bat file must be stored in the folder alongside the .exe file.

Step 4 - Grant access

When you have completed an import of the Application data, you will need to grant access to this data to your users. See Admin Center - Users panel.

Docker for Linux/Windows

Step 1 - obtain the tool(s)

≥ v. 1.12.0-beta4

The tools are provided with the Docker images, therefore as long as CAST Imaging has been installed, you can proceed.

≤ v. 1.12.0-beta3

Click here to expand...

The tools are not provided with the Docker images. Instead, download the three assets from https://github.com/CAST-Extend/com.castsoftware.imaging.dockersetup/releases for your currently installed release:

  • etl-automation
  • exporter
  • automation_scripts.zip

Place the two binaries (etl-automation and exporter) in the etl-automation folder of your installation directory, where the etl_config.json file (shipped with the installation) file is located, for example:

sudo mv /home/my_user/Downloads/exporter /opt/CAST/container-support/etl-automation
sudo mv /home/my_user/Downloads/etl-automation /opt/CAST/container-support/etl-automation

Unzip automation_scripts.zip in the etl-automation folder of your installation directory, where the etl_config.json file is located, for example:

cd /opt/CAST/container-support/etl-automation
sudo unzip /home/my_user/Downloads automation_scripts.zip

Make sure the two binaries have execute permissions:

sudo chmod +x etl-automation exporter

Step 2 - configure .json file

Edit the etl_config.json file to match your environment, for example:

Docker Installer (located in the unzipped extension folder): etl-automation\etl_config.json

Linux Docker Installer (deprecated): /opt/CAST/container-support/etl-automation/etl_config.json

.json file syntax:

PORT

Enter the port number for the target CAST Storage Service/PostgreSQL instance:

CSS42284
CSS32282
CSS22280
HOST

Enter the network host name of the server on which the CAST Storage Service/PostgreSQL instance is located. This CAST Storage Service/PostgreSQL instance will host the Analysis Service schema which contains the required Application data.

localhost can be used if the CAST Storage Service/PostgreSQL instance is located on the current server.

SCHEMA

Name(s) of the Analysis Service schema containing the Application you would like to export/import into CAST Imaging. Three options are available:

  • single schema = "SCHEMA" : "myapplication_local",
  • multiple schemas = "SCHEMA" : ["myapplication_local", "myapplication2_local"],
  • all schemas on the CAST Storage Service/PostgreSQL instance =  "SCHEMA" : "all",

You can view the name of the Analysis Service schema as follows:

OUTPUTDIR

Full path to a temporary output folder in which to store the exported items - these items are not removed when the process is completed, i.e. a record of the export is retained. CAST recommends using the following location, which is located within the CAST Imaging "etl" container (Linux containers are used even when running docker on Windows):

"OUTPUTDIR" : "/opt/imaging/imaging-etl/logs",
DATABASEAllows you to specify the CAST Storage Service/PostgreSQL database in which the schemas listed in SCHEMAS are stored. By default this is set to "postgres", so unless you have manually changed this, there is no need to specify it in config.json.
EXPORT
Available in Docker images with the tag 2.1.2.

Choose whether you want the tool to perform the export step or not, using either true or false:

  • If you set it to false (and IMPORT set to true) then applications listed in the APPS parameter will be imported into CAST Imaging. The exported ZIP files containing the relevant application data must already be present in the location defined in OUTPUTDIR.
  • If you set it to true (and IMPORT set to false) then applications listed in the APPS parameter will be exported to ZIP file and stored in the location defined in OUTPUTDIR.
  • If you set it to true (and IMPORT set to true), then the two step export/import action will be performed.
IMPORT

Available in Docker images with the tag 2.1.2.

  • If you set it to false (and EXPORT set to true) then applications listed in the APPS parameter will be exported to ZIP file and stored in the location defined in OUTPUTDIR.
  • If you set it to true (and EXPORT set to false) then applications listed in the APPS parameter will be imported into CAST Imaging. The exported ZIP files containing the relevant application data must already be present in the location defined in OUTPUTDIR.
  • If you set it to true (and EXPORT set to true), then the two step export/import action will be performed.
APPS

Available in Docker images with the tag 2.1.2.

Names of the applications you want to export/import. Use the following syntax:

  • single application = "APPS" : "myapp1",
  • multiple schemas = "APPS" : ["myapp1", "myapp2"],
  • all schemas on the CAST Storage Service/PostgreSQL instance =  "APPS" : "all",
IMAGING_HOSTNAME

Enter the network host name/IP address of the server on which CAST Imaging has been installed, e.g.: localhost.

Note that if your CAST Imaging instance is running in SSL mode, then you should enter the same hostname as specified in the SSL certificate, otherwise the import may fail.
IMAGING_PORT

≥ 2.5.x

This corresponds to the port on which the front end of CAST Imaging is running. By default the .json file will contain 8083 (default port), but if you have customized the port (for example you are using https protocol) please ensure that you change it.

≤ 2.4.x

This corresponds to the port on which the CAST Imaging System - imaging-ETL Windows Service runs. By default this is set to 9001 and therefore the etl_config.json will be prefilled with 9001. Unless you have changed the port for the CAST Imaging System - imaging-ETL service, please leave this as is.

IMAGING_DOMAIN

≥ 2.5.x

This corresponds to the tenant into which you would like to import the application. If you have not created any tenants, then please leave this option empty - this will import the application into the "default" tenant. Otherwise, please specify the tenant name:

  • "IMAGING_DOMAIN": "my_tenant",

See Admin Center - Multi tenant panel for more information about tenants.

IMAGING_ADMIN_USER

≥ 2.5.x

This option is no longer required and is replaced by TOKEN.

≤ 2.4.x

Name of a user with the admin role (see Admin Center - Users panel).

TOKEN

≥ 2.5.x

This field requires an API Key - this key can be generated when logged in to CAST Imaging and is valid for the current user - therefore, ensure that the user has permission to import Applications (typically this is a user with the ADMIN or SUPER ADMIN roles). See Admin Center - API key generation and usage for more information about generating an API Key.

Note that token values from previous releases will no longer function.

≤ 2.4.x

Enter the value for the option proxy.config.token.values located in the file:

/opt/CAST/container-support/login/application.properties

The value provided in the .json file supplied with CAST Imaging should already match, but please check that this is the case. For example:

USERNAMEEnter the CAST Storage Service/PostgreSQL user login (e.g.: operator) for the instance on which the applications you want to export/import are stored.
PASSWORDEnter the CAST Storage Service/PostgreSQL password corresponding to the user entered in the USERNAME filed, e.g.: CastAIP.
LOG_PATH

Full path to a log file in which the process logs will be stored. CAST recommends using the following location, which is located within the CAST Imaging "etl" container (Linux containers are used even when running docker on Windows):

"LOG_PATH" : "/opt/imaging/imaging-etl/logs"

If you use any other location for the LOG_PATH option, logs will be generated within the CAST Imaging "etl" container and you will therefore need to access the location using docker exec:

docker exec -it etl /bin/bash; cd path/location

Example .json file to run export and import steps in one go:

≥ 2.5.x
{
"PORT" : 2282,
"HOST" : "MY_CSS_SERVER",
"SCHEMA" : ["myapplication_local", "myapplication2_local"],
"OUTPUTDIR" : "/opt/imaging/imaging-etl/logs",
"DATABASE": "",
"EXPORT" : true,
"IMPORT": true,
"APPS" : ["myapp1", "myapp2"],
"IMAGING_HOSTNAME" : "localhost",
"IMAGING_PORT" : 8083,
"IMAGING_DOMAIN": ["my_tenant"],
"TOKEN" : "umaKngIi.lW5OIlwlnLQ5Bc6Tn8bzdcHbrwLd",
"USERNAME" : "operator",
"PASSWORD" : "CastAIP",
"LOG_PATH" : "/opt/imaging/imaging-etl/logs"
}
≤ 2.4.x
{
"PORT" : 2282,
"HOST" : "MY_CSS_SERVER",
"SCHEMA" : ["myapplication_local", "myapplication2_local"],
"OUTPUTDIR" : "/opt/imaging/imaging-etl/logs",
"DATABASE": "",
"EXPORT" : true,
"IMPORT": true,
"APPS" : ["myapp1", "myapp2"],
"IMAGING_HOSTNAME" : "localhost",
"IMAGING_PORT" : 9001,
"IMAGING_ADMIN_USER" : "admin",
"TOKEN" : "D976986DHGDJ987987978",
"USERNAME" : "operator",
"PASSWORD" : "CastAIP",
"LOG_PATH" : "/opt/imaging/imaging-etl/logs"
}

Step 3 - Run the tool

To run the tool use the following syntax, this will perform the extraction and run the import all in one go:

≥ v. 1.12.0-beta4

Docker Installer (located in the unzipped extension folder):
./imaging --configfile etl_config.json --importapp

Linux Docker Installer (deprecated):
imaging --configfile etl_config.json --importapp

≤ v. 1.12.0-beta3

Click here to expand...
sudo ./etl-automation -CONFIG etl_config.json

Step 4 - Grant access

When you have completed an import of the Application data, you will need to grant access to this data to your users. See Admin Center - Users panel.

Tips

Error importing the Application when it already exists

If your import action continually fails when the Application already existing in CAST Imaging, CAST highly recommends deleting the Application from CAST Imaging (see Admin Center - Application management panel) and then re-importing it. Note that deleting the Application will also remove all customization that you have added.

[WinError 10054] An existing connection was forcibly closed by the remote host

If you are attempting to export from a PostgreSQL instance installed on Microsoft Windows (i.e. CAST Storage Service) and your application is very large, you may find that the export process fails with the error which is caused by a timeout:

[WinError 10054] An existing connection was forcibly closed by the remote host

By default, on PostgreSQL instances installed on Microsoft Windows, the PostgreSQL instance timeout is set to 2hrs, therefore any export process that takes longer than 2hrs will fail. If you encounter this error, it is possible to modify the timeout setting using the postgresql.conf file located here:

%PROGRAMFILES%\CAST\CastStorageService<version>\db_data\

Edit this file with a text editor and find the tcp_keepalives_idle entry:

# - TCP settings -
# see "man tcp" for details

#tcp_keepalives_idle = 0		# TCP_KEEPIDLE, in seconds;
					# 0 selects the system default

By default, this entry is commented out and will be ignored - this means that the default value of 0 applies - i.e. the Microsoft Windows system default of 2hrs. To set a longer timeout interval, uncomment the entry and set required interval in seconds (above 2hrs). For example to set a timeout of 6hrs:

# - TCP settings -
# see "man tcp" for details

tcp_keepalives_idle = 21600		# TCP_KEEPIDLE, in seconds;
					# 0 selects the system default

Save the file and then restart the CAST Storage Service instance to apply the change.