1. Check if the Forms version installed on the machine is supported by CAST by looking at the supported language section in release notes: http://doc.castsoftware.com/help/topic/73x/Supported-Languages_568426668.html. If it is not, then solution is to use supported version. If Forms version is supported, go to 2nd point mentioned below.
2. Try removing the last FMB/PLL loaded before the crash, until it does not crash anymore, or there is no more FMBs/PLLs to extract from.
3. It is the same approach as with the crashes in the analyzers (CMS Snapshot Analysis - Information - Overview of the Dichotomy process ), when we try to identify the culprit source files by dichotomy.
4. If you suspect it is due to the JVM, or other environment settings, then you can do the contrary: you exclude all the files but one, just to check the hypothesis.

Note: To remove an FMB/PLL, Just rename it to exclude it from the extraction; the recommended renaming is to add a supplemental extension "excluded", or ".castexcluded" => it helps identifying the set of files to to be excluded when the extraction succeeds at last.

If the issue is not general one, then go to next section i.e. Check if the identified FMBs/DLLs are valid

If the you are able to open the FMBs/PLLs in Forms Builder and still getting extraction message such as Invalid file, then problem maybe related with Forms API
To confirm the same try to package the same FBBs/PLLs in Different machine where Forms is installed.
If it works in different machine, then it is confirmed that the problem is due the Forms API of the last machine. If not then report to support

If the error is encountered while extraction, please refer the section : Refer the page as per your Extraction error/message

If none of the cases matches your problem please contact Cast Technical Support .