What file storage requirements do I need?

Overview

CAST Imaging requires file storage as listed below:

Folder Default path on Microsoft Windows install Default path on Docker install Remarks
delivery Shared network folder Shared network folder Contains reference information about the application and the various source code deliveries that have occurred for a given application.
deploy Shared network folder Shared network folder Used to store certain types of source code which require further extraction and processing before analysis, such as .castextraction files or PDS dump files.
common-data Shared network folder Shared network folder A common location used for node related data (logs, backup, sherlock, source code upload and other folders used by the node).
Logs Node: %PROGRAMDATA%\CAST\CAST\Logs Node: usr/share/CAST/CAST/Logs Used for storing all technical logs produced by the node with regard to code delivery/analysis activities. One sub-folder folder will be created per application onboarded in CAST Imaging. The log files will contain the path and the name of the source file. For some warning messages (e.g.: syntax error), the path of the source code involved in the warning is shown in the log.
LISA / LTSA Node: %PROGRAMDATA%/CAST/CAST/CASTMS/LISA and %PROGRAMDATA%/CAST/CAST/CASTMS/LTSA Node: usr/share/CAST/CAST/CASTMS/LISA and usr/share/CAST/CAST/CASTMS/LTSA Location to store temporary files generated during the analysis process on each node:

- Large Intermediate Storage Area (LISA) - cleaned on completion of each analysis.
- Large Temporary Storage Area (LTSA) - is permanent and contains preprocessed source code files, archive files for analysis etc.

Some technology extensions need to preprocess the source code before performing the analysis. In this case, the source code that will be analyzed is stored in the LISA folder, not in the Deploy folder.
extensions Node: %PROGRAMDATA%/CAST/CAST/Extensions usr/share/CAST/Extensions Used for storing CAST technology extensions obtained by CAST Imaging Console from https://extend.castsoftware.com external link. Must always be located on the node machine.

Storage folder distribution strategy

The following can be configured (see deployment modes):

  • Local distribution only (only for single machine deployment)
  • Mixed local and network distribution (distributed deployment)
  • Network distribution only (distributed deployment)

We also describe the relationship between performance and scalability depending on the way you choose to deploy the storage folders. In addition, we suggest some required disk space recommendations (minimum recommendations and advised configuration for a large number of applications) depending on the chosen storage folder distribution strategy.

Local distribution (only for single machine deployment)

Better performance / lower scalability

All storage folders are located on the local machine:

The limiting factor for this deployment is that all these folders can grow very quickly during an analysis, therefore, they can potentially overload the local disks on the machine.

Disk space recommendations

  • Minimum recommendation: 256GB
  • Run multiple applications in parallel: 1TB minimum

Mixed local and network distribution (distributed deployment)

Balance between storage and performance: medium performance / medium scalability

The delivery/deploy/common-data and the logs folders are located on a shared network drive. Other folders are located on the local machine - i.e. the node:

The aim of using a shared network folder for the delivery/deploy/common-data and the logs folders is to accommodate the potentially large amount of storage that is required and to provide easy access to the files, particularly the logs folder. The size of these folders can grow very quickly during an analysis (in particular the delivery folder which stores multiple versions of a given application’s source code).

This configuration reduces the risk of overloading the local machine’s disk space, however, performance during analysis will be slightly reduced since the logs folder is located on the shared network drive, and an analyzer will require slightly more time to write to it during an analysis ((logs) can be located on the Node machine if required).

Disk space recommendations

  • Minimum recommendation:

    • Network storage (delivery/deploy/common-data/logs): 170GB
    • Local storage (LISA/LTSA/extensions): 86GB

  • Run multiple applications in parallel:

    • Network storage (delivery/deploy/common-data/logs): 700GB minimum
    • Local storage (LISA/LTSA/extensions): 300GB minimum

Network distribution only (distributed deployment)

Lower performance / better scalability

All the storage folders, except “extensions” (which must always be located on the node), are located on a shared network drive:

This configuration will rarely overload the local disk space, however, performance during analysis is slower compared to other storage strategies for the following reasons:

  • The time required to load the source code in to memory from a network drive is inevitably longer.
  • Slower log file updates
  • Temporary files will be generated more slowly.

Disk space recommendations

  • Minimum recommendation:

    • Network storage (delivery/deploy/common-data/logs/LISA/LTSA): 256GB
    • Local storage (extensions): 50GB

  • Run multiple applications in parallel:

    • Network storage (delivery/deploy/common-data/logs/LISA/LTSA): 1TB minimum
    • Local storage (extensions): 50GB minimum