This documentation is not maintained. Please refer to doc.castsoftware.com/technologies to find the latest updates.

Summary: This document provides information about the extension providing UNIX Shell support.

Extension ID

com.castsoftware.shell

What's new?

Please see Shell 1.3 - Release Notes for more information.

Description

This extension provides support for applications written using UNIX Shell languages.

Although this extension is officially supported by CAST, please note that it has been developed within the technical constraints of the CAST Universal Analyzer technology and to some extent adapted to meet specific customer needs. Therefore the extension may not address all of the coding techniques and patterns that exist for the target technology and may not produce the same level of analysis and precision regarding e.g. quality measurement and/or function point counts that are typically produced by other CAST AIP analyzers.

In what situation should you install this extension?

If your application contains source code written using UNIX Shells and you want to view these object types and their links with other objects, then you should install this extension.

Supported UNIX shells

This version of the extension provides partial support for:

UNIX shellSupported
Bourne shell (bsh/sh/shell)(tick)
Bourne-Again Shell (bash)(tick)
C shell (csh)(tick)
KornShell (ksh)(tick)
Secure Shell (ssh)(tick)
Tenex C Shell (tcsh)(tick)

Function Point, Quality and Sizing support

This extension provides the following support:

  • Function Points (transactions): a green tick indicates that OMG Function Point counting and Transaction Risk Index are supported
  • Quality and Sizing: a green tick indicates that CAST can measure size and that a minimum set of Quality Rules exist
Function Points
(transactions)
(tick)
Quality and Sizing(tick)

AIP Core compatibility

This extension is compatible with:

AIP Core release

Extension release

Supported

8.3.x

≥ 1.0.1(tick)

Supported DBMS servers

This extension is compatible with the following DBMS servers:

DBMSSupported
CSS / PostgreSQL(tick)

Prerequisites

(tick)An installation of any compatible release of AIP Core (see table above)

Download and installation instructions

The extension will be automatically downloaded and installed in AIP Console when you deliver Shell code (i.e. any file with an extension listed below). You can also manually install the extension using the Application - Extensions interface.

Prepare and deliver the source code

Once the extension is downloaded and installed, you can now package your source code and run an analysis. The process of preparing and delivering your source code is described below:

Source code preparation

Only files with following extensions will be analyzed:

  • *.bash
  • *.bsh
  • *.csh
  • *.ksh
  • *.sh
  • *.shell
  • *.ssh
  • *.tsch

If any files intended for other applications are included in the delivery and which are renamed to supported Shell extensions, the following effects might be observed during an analysis:

  • inconsistent objects may get created
  • end of string ''' not found errors
  • if the file is binary: Invalid UTF-8 sequence found in text to be matched or searched for a regular expression

Source code preprocessing 

Shell source code needs to be preprocessed so that CAST can understand it and analyze it correctly. This code preprocessing is actioned automatically when an analysis is launched or a snapshot is generated (the code is preprocessed before the analysis starts). The Shell Preprocessor log file is stored in the following location:

%PROGRAMDATA%\CAST\CAST\Logs\<application_name>\Execute_Analysis_<guid>\com.castsoftware.shell.<_extension_version>.prepro_YYYYMMDDHHMMSS.log


Note that the LISA folder will be used to analyze the preprocessed files.

Deliver the source code

Analysis configuration and execution

There are no analysis/technology configuration options available for Shell, however you should check that at least one Shell analysis unit has been created as shown below.

AIP Console exposes the technology configuration options once a version has been accepted/imported, or an analysis has been run. Click Universal Technology (3) in the Config (1) > Analysis (2) tab to display the available options for your Shell source code:

Then choose the relevant Analysis Unit (1) to view the configuration:

Logging mechanism

Analysis log files

Analysis logs are stored in the default locations.

Shell Preprocessor

Shell Preprocessor log files (the preprocessor is launched automatically during an analysis) are stored in the following locations:

LocationLog file name
%PROGRAMDATA%\CAST\CAST\Logs\<unique_application_id>\com.castsoftware.shell.prepro_<ExtensionVersion>_<YYYYMMDDHHMMSS>.log

What results can you expect?

Objects

IconDescription

Shell Project

Shell Program

Shell Function, Shell Special Function

Note that these objects are note displayed in CAST Imaging's Reduced or Simplified Call Graph view.

Shell Variable

Shell Call to Java Program

Shell Call to Generic Program
Amazon Web Services

Shell AWS Lambda Function

Shell Call to AWS Lambda Function

Shell Call to AWS Unknown Lambda Function

Shell AWS S3 Bucket

Shell AWS Unknown S3 Bucket

Shell AWS DynamoDB Database

Shell AWS DynamoDB Table

Shell AWS Unknown DynamoDB Table

Shell AWS SNS Publisher, Shell AWS SQS Publisher

Shell AWS SNS Subscriber, Shell AWS SQS Receiver

Shell AWS Unknown SNS Publisher, Shell AWS Unknown SQS Publisher

Shell AWS Unknown SNS Subscriber, Shell AWS Unknown SQS Receiver

Shell GET service request

Shell POST service request

Shell PUT service request

Shell DELETE service request

Shell SMS, Shell Email
Source ObjectLink TypeTarget Object
Shell ProgramcallProgLinkShell Program
Shell ProgramincludeShell Program
Shell ProgramcallLinkShell Function/Special Function
Shell Function/Special FunctioncallLinkShell Program
Shell Function/Special FunctioncallLinkShell Function/Special Function
Shell Function/Special Function/Programuse select, update, insert, deletedata functions
Shell Function/Special Function/ProgramcallLinkShell Call to Java Program
Shell Function/Special Function/ProgramcallLinkShell Call to Generic Program
Shell Function/Special Function/ProgramcallLinkSQL Table
Shell Function/Special Function/ProgramcallLinkSQL Script

Embedded SQL

Shell allows the use of here-document to write SQL queries, for example:

#!/bin/sh
/opt/solid/bin/solsql db db <<abc
select count(*) from table_1;
select count(*) from table_2;
exit;
!;
abc

The Shell extension can use these queries to create links.

Call to SQL script

The Shell extension allows links to SQL script when used with a dependency to the SQL analyzer

For example:

script.sh
#!/bin/env bash

sqlplus script.sql
script.sql
CREATE TABLE ...

In Enlighten

Links to external programs

The following call pattern are supported

  • ./exe
  • /path/to/exe
  • python
  • java
  • nohup
  • eval
  • wlst.sh
  • runcobol | runcobol.exe

The extension does support links between Shell programs and Java objects (for example methods). Links will be created between these technologies.

The extension manages call to Java classes and .jar files. For .jar files no links will go further as .jar aren't handle by any extensions, the link is purely informative.

Basic Case

...
/bin/java UpdateRepartiteurs
...

wil generate the following diagram

Function Case

...
MajRepartiteurs ()
{
	...
	java UpdateRepartiteurs
	...
}
...

will generate the following diagram

The extension does support links between Shell programs and COBOL objects (for example programs). Links will be created between these technologies.

Basic Case

COBOL file

IDENTIFICATION DIVISION.
PROGRAM-ID. HELLO.
* simple hello world program
PROCEDURE DIVISION.
    DISPLAY 'Hello world!'.

SHELL script

./hello

will generate the following diagram

The extension does support links between Shell programs and Python objects (for example methods). Links will be created between these technologies.

The following script

...
python foo.py
...

will generate the following diagram, assuming that foo.py exists and has been analyzed

Support for AWS CLI

Lambda

Supported APILink typeCallerCalleeRemarks
aws lambda invokecallLinkShell ProgramShell Call to AWS Lambda Function
aws lambda create-functionN/AN/AN/ACreates a Shell AWS Lambda Function with handler and runtime properties (when resolved).

An example of a command invoking a lambda named my-function.

$ aws lambda invoke --function-name my-function

When analyzing it, a Shell Call to AWS Lambda function object is created. This object will be linked to lambda functions objects (which may be created by other extensions) having the same name during application level analysis stage by the web services linker extension. When the name of an invoked Lamba function is not evaluated (either because of the absence of information or technical limitations) a Shell Call to AWS Unknown Lambda Function is created instead. A maximum of one call for each lambda (including unknowns) is created per caller.

An example of a command creating an AWS Lambda of name my-function where the handler is a javascript method.

$ aws lambda create-function --function-name my-function --zip-file fileb://function.zip --handler index.handler --runtime nodejs12.x --role arn:aws:iam::123456789012:role/lambda-ex

When analyzing it, a Shell AWS Lambda Function object is created. It contains the runtime and handler properties (here nodejs12.x and index.handler, respectively). The linking from the lambda function to the handler function is then carried out during application level analysis stage by one of the following extensions (depending on the runtime):

Runtime

Extension

Minimum required extension release

Javacom.castsoftware.awsjava 1.2.0-alpha3 (AWS Java 1.2 - Release Notes#alpha3)
.NETcom.castsoftware.awsdotnet1.0.0-alpha5 (AWS .NET 1.0 - Release Notes#alpha5)
Pythoncom.castsoftware.python1.4.0-beta7 (Python 1.4 - Release Notes#beta7)
Node.js

com.castsoftware.nodejs 

2.7.0-beta3 (when the handler is written in .js) (Node.js - 2.7 - Release Notes#beta3)

TypeScriptcom.castsoftware.typescript 

1.9.0-alpha1 (when the handler is written in .ts) (TypeScript and Frameworks)

Below we illustrate the expected results with a few aws commands. The code samples used are adaptations from available examples in official AWS documentation pages.

Example 

In the following code we enclose three cli calls to aws lambda services in three different shell functions:

CreateFunction() {
  aws lambda create-function --function-name my-lambda --runtime nodejs10.x --zip-file fileb://my-function.zip --handler my-function.handler --role arn:aws:iam::123456789012:role/service-role/MyTestFunction-role-tges6bf4
}

InvokeFunction1() {
  aws lambda invoke --function-name my-function --payload '{ "name": "Bob" }' response.json
}

# it contains an unresolved variable
InvokeFunction2() {
  aws lambda invoke --function-name $VarFunctionName --payload '{ "name": "Bob" }' response.json
}


As a result of the code above, a AWS Lambda function is created, and two calls to a AWS Lambda functions (one resolved and one unresolved).

The Lambda function object contains the runtime and handler properties (see details above in the section) necessary for linking to the respective handler in different technologies.

S3

There are at least two different APIs for manipulation of S3 buckets. The s3api command is used for low level manipulation where s3 has a higher-level functionality.

API (s3)Link typeCallerCalleeRemarks
aws s3 cpuseSelectLink, useInsertLinkShell Program or FunctionShell AWS S3 Bucket
aws s3 mbN/AN/AN/ACreates a bucket
aws s3 mvuseDeleteLink, useInsertLinkShell Program or FunctionShell AWS S3 Bucket
aws s3 lsuseSelectLinkShell Program or FunctionShell AWS S3 BucketWhen no bucket is specified, a link to each buckets is created
aws s3 presigncallLinkShell Program or FunctionShell AWS S3 BucketNo URL creation is modelized.
aws s3 rbcallLinkShell Program or FunctionShell AWS S3 Bucketrb deletes an empty bucket (so the creation  of a useDeleteLink is excluded).
aws s3 rmuseDeleteLinkShell Program or FunctionShell AWS S3 Bucket
aws s3 syncuseSelectLink, useInsertLinkShell Program or FunctionShell AWS S3 Bucket
aws s3 websitecallLinkShell Program or FunctionShell AWS S3 Bucket

In the absence of explicit bucket creation, references to buckets in other commands are used to create table objects.

API (s3api) with CRUD linksLink typeCallerCallee
aws s3apget-objectuseSelectLinkShell Program or FunctionShell AWS S3 Bucket

aws s3apget-object-torrent

aws s3api select-object-content
aws s3api list-objects
aws s3api list-objects-v2
aws s3api list-parts
aws s3api delete-bucketuseDeleteLinkShell Program or Function

Shell AWS S3 Bucket

aws s3api delete-object
aws s3api delete-objects
aws s3apput-objectuseInsertLinkShell Program or FunctionShell AWS S3 Bucket
aws s3api restore-object
aws s3apupload-part
aws s3api copy-objectuseSelectLink,
useInsertLink
Shell Program or FunctionShell AWS S3 Bucket

For the list of commands acting on a bucket that are modelized with a callLink, expand the list below

Click here to expand other methods...
API Link typeCallerCallee
  • abort-multipart-upload
  • complete-multipart-upload
  • create-multipart-upload
  • delete-bucket-analytics-configuration
  • delete-bucket-cors
  • delete-bucket-encryption
  • delete-bucket-intelligent-tiering-configuration
  • delete-bucket-inventory-configuration
  • delete-bucket-lifecycle
  • delete-bucket-metrics-configuration
  • delete-bucket-ownership-controls
  • delete-bucket-policy
  • delete-bucket-replication
  • delete-bucket-tagging
  • delete-bucket-website
  • delete-object-tagging
  • delete-public-access-block
  • get-bucket-accelerate-configuration
  • get-bucket-acl
  • get-bucket-analytics-configuration
  • get-bucket-cors
  • get-bucket-encryption
  • get-bucket-intelligent-tiering-configuration
  • get-bucket-inventory-configuration
  • get-bucket-lifecycle-configuration
  • get-bucket-location
  • get-bucket-logging
  • get-bucket-metrics-configuration
  • get-bucket-notification-configuration
  • get-bucket-ownership-controls
  • get-bucket-policy
  • get-bucket-policy-status
  • get-bucket-replication
  • get-bucket-request-payment
  • get-bucket-tagging
  • get-bucket-versioning
  • get-bucket-website
  • get-object-acl
  • get-object-attributes
  • get-object-legal-hold
  • get-object-lock-configuration
  • get-object-retention
  • get-object-tagging
  • get-public-access-block
  • head-bucket
  • head-object
  • list-bucket-analytics-configurations
  • list-bucket-intelligent-tiering-configurations
  • list-bucket-inventory-configurations
  • list-bucket-metrics-configurations
  • list-buckets
  • list-multipart-uploads
  • list-object-versions
  • put-bucket-accelerate-configuration
  • put-bucket-acl
  • put-bucket-analytics-configuration
  • put-bucket-cors
  • put-bucket-encryption
  • put-bucket-intelligent-tiering-configuration
  • put-bucket-inventory-configuration
  • put-bucket-lifecycle-configuration
  • put-bucket-logging
  • put-bucket-metrics-configuration
  • put-bucket-notification-configuration
  • put-bucket-ownership-controls
  • put-bucket-policy
  • put-bucket-replication
  • put-bucket-request-payment
  • put-bucket-tagging
  • put-bucket-versioning
  • put-bucket-website
  • put-object-acl
  • put-object-legal-hold
  • put-object-lock-configuration
  • put-object-retention
  • put-object-tagging
  • put-public-access-block
  • wait
  • write-get-object-response
callLinkShell Program or FunctionShell AWS S3 Bucket

Example 

In the following code we show a few calls to s3 and s3api commands.

Hello () {
   aws s3 cp test.txt s3://bucket1/test2.txt
}

MoveFile () {
   aws s3 mv s3://bucket19/test.txt s3://${BUCKET}/
}

GetObject () {
   aws s3api get-object --bucket text-content --key dir/my_images.tar.bz2 my_images.tar.bz2
}

GetObjectUnresolved () {
   aws s3api get-object --bucket $UnknownVar --key dir/my_images.tar.bz2 my_images.tar.bz2
}

The resulting buckets and links:

If the name of a bucket is not resolved, an unknown object is created as shown above. Note that a maximum of one unknown (SHELL) bucket will be created per project. However, if the name of the bucket is partially resolved, a normal bucket object will be created, with the unresolved name fragments replaced by the "?" character (example not shown here).

SQS

The supported API:

APILink TypeCallerCallee
aws sqs send-messagecallLinkShell Program or Function

Shell AWS SQS Publisher

aws sqs send-message-batchcallLink
aws sqs receive-messagecallLinkShell AWS SQS ReceiverShell Program or Function (optional)


Example 

In the following code we send a message with the Send function and we retrieve the message in the Receive function:

Send () {
   aws sqs send-message --queue-url https://sqs.us-east-1.amazonaws.com/80398EXAMPLE/MyQueue --message-body "Information about the largest city in Any Region." --delay-seconds 10 --message-attributes file://send-message.json
}

Receive () {
   aws sqs receive-message --queue-url https://sqs.us-east-1.amazonaws.com/80398EXAMPLE/MyQueue --attribute-names All --message-attribute-names All --max-number-of-messages 10
}

The results in Enlighten show the created objects with name equal to the Queue url:

The callLink links between the Publisher and the respective Receivers are created by the Web Services Linker extension during application level based on the Queue name registered in the queueName property. The handler of the receiver is the function Receive itself, responsible of treating the received data.

When the evaluation of the queue name fails, a Shell AWS Unknown SQS Publisher (or Receiver) object will be created.

SNS

APILink TypeCallerCallee
aws sns publishcallLink

Shell Program or Function

Shell AWS SNS Publisher, Shell SMS

aws sns publish_batchcallLink
aws sns subscribecallLink

Shell AWS SNS Publisher

Shell SMS, Shell Email


The supported protocols are the following:

Protocol

Object/s created

Name of the object

email/email-jsonShell Emailan Email   (the email addresses are not evaluated)
http/httpsShell POST service requestthe url (evaluated from the endpoint)
lambdaShell Call to AWS Lambda Functionthe name of the lambda function (evaluated from the endpoint)
smsShell SMSan SMS   (the SMS numbers are not evaluated)
sqs

Shell AWS SQS Publisher

the name of the queue (evaluated from the endpoint)

The callLink links between the Publisher and the respective Subscribers are created by the Web Services Linker extension during application level.

For each method a maximum of one subscriber per given topic will be created as shown in the image above.

Example 1

In the following code we have to functions, one publishing into a topic, and the other one subscribing to different notification endpoints of different protocols.

Publish () {
   aws sns publish --topic-arn "arn:aws:sns:us-west-2:123456789012:my-topic" --message file://message.txt
}

Receive() {
	aws sns subscribe --topic-arn arn:aws:sns:us-west-2:123456789012:my-topic --protocol email --notification-endpoint my-email@example.com
	aws sns subscribe --topic-arn arn:aws:sns:us-west-2:123456789012:my-topic --protocol sms --notification-endpoint 123456789
	aws sns subscribe --topic-arn arn:aws:sns:us-west-2:123456789012:my-topic --protocol sqs --notification-endpoint arn:partition:service:region:account-id:queueName
	aws sns subscribe --topic-arn arn:aws:sns:us-west-2:123456789012:my-topic --protocol lambda --notification-endpoint fooarn:function:lambda_name:v2
	aws sns subscribe --topic-arn arn:aws:sns:us-west-2:123456789012:my-topic --protocol http --notification-endpoint http://foourl
}

The callLink links between the Publisher and the respective Subscribers are created by the Web Services Linker extension during application level.


For each method a maximum of one subscriber per given topic will be created as shown in the image above.

When the evaluation of the topic name fails, a Shell AWS Unknown SNS Publisher (or Subscriber) object will be created.

Example 2

We can also have direct sms deliveries from calls to publish API command:

PublishSMS () {
	aws sns publish --message "Hello world!" --phone-number +1-555-555-0100
}

resulting in the direct callLink to the Shell SMS object:

DynamoDB

Please refer: DynamoDB support for Shell source code

Structural Rules

The following structural rules are provided:

You can also find a global list here: https://technologies.castsoftware.com/rules?sec=t_1016000&ref=||.

Limitations/known issues

Script file import

This extension does not support the "import" of functions defined in other script files using "sourcing"  (see https://www.mkssoftware.com/docs/man1/dot.1.asp), therefore no links are expected between functions declared in other script files Note that "sourcing" is supported when used in variable definitions.

Deployment folder path

The deployment folder path should contain only ASCII characters (due to the way the command line text is passed in a Windows operating system). Names of folders and files inside the deployment folder can contain non-ASCII characters.

Links to database objects

When your Shell scripts contain references to database objects in the server side element of an application and these references use fully qualified names for the database object, no links will be created between Shell and the database objects when you have used the SQL Analyzer extension to analyze the SQL if SQL Analyzer does not create database objects with same name. If you have used the SQL analyzers embedded in CAST AIP to analyze the SQL, then links will be created as normal. This is a known issue.

Metrics Assistant (embedded in CAST AIP) limitations

Searches not limited only to embedded SQL

The MA (Metric Assistant) which is used for metric search cannot search only in embedded SQL. Some Shell rules may be affected by this limitation and may produce false violations.

Cannot calculate metric excluding comments

The MA (Metric Assistant) which is used for metric search cannot search correctly while excluding comments especially if comments start or end adjacent to the keyword. If such a condition exists, random false violations may occur.

Shell embedded strings

Shell code allows string to be embedded in strings as shown in the code sample below. Currently, the Shell extension (and other Universal Analyzer type extensions) will consider this as one continuous string. Because we do not have any way to identify perfect end string patterns in this case, we cannot find the end of string and therefore the file will be skipped during the analysis.

echo '              
Outer string ;              
cat '$file03'             
Another outer string              
'$id'             
' | $command

KSH: guessing of ending single\double quote

Note that this limitation is no longer applicable to Shell ≥ 1.0.10.

KSH supports the guessing of ending single/double quotes. The Shell extension supports this when the string is in single line, however, it is not supported when the string is in a multiple line, for example:

export OUTPUT=`basename $SOME_VARIABLE | $AWK 'FS="-" {
  i=3
  tmpMachineName= "mach_"$2
  while ( i <= NF){
    tmpMachineName=tmpMachineName"-"$i;
    i++;
  }
  print(tmpMachineName);
}`

Multi-line document markers

Note that this limitation is no longer applicable to Shell ≥ 1.0.10.

When a document marker is in a multi-line string, the Shell extension will not be able detect that it is in a string, for example:

some_multiline_string="a;b;c;d;\
e;f;g;h;\
x;y;z<<;strong text"

In both of these cases, the file will be skipped and logs will contain the warning: "File Skipped".