CAST Imaging System - Database - Neo4j - Error - Database not starting results in errors like applications not loading or internal server error

Purpose

When the Neo4j database is not up, you may face several problems such as 'Internal server error' when applying a key, applications not loading, errors like 'Index clean jobs can't be added after collector start'.

Imaging Version

Release	Yes/No
2.x

Details

When the Neo4j database is not up, you can have several problems such as 'Internal server error' when applying a key, applications not loading, and errors like 'Index clean jobs can't be added after collector start'.
1. First try to restart the neo4j database service and see if it comes up. You may need to access the Neo4j database and confirm that all the sub-databases are up as well
2. It can also be that the neo4j database may be in quarantine mode. This can happen after a system crash or abort or a panic due to space issues. To resolve, please do the following:
  1. Run the following command on the neo4j database:
    1. This should be done in the system database.
      CALL dbms.quarantineDatabase('neo4j',false)
    2. You may have to execute the following command before the one above to get into the system database :
      use system
    3. You may need to clear up transaction logs prior to the steps above. If in the debug.log in the imaging logs, you see something like this:
      1. Fail to read transaction log version XXXXX
        which would mean that you need to remove the file <Neo4j_data>/transactions/neo4j/neostore.transaction.db.XXXXX and then issue the commands above to move the database out of quarantine mode
  2. Restart the neo4j service
3. The neo4j database may also be doing some recovery operations after an unexpected operation, crash, or abort. These operations may take some time. You can generally see this with messages like this in the debug.log file:
  1. Called db.clearQueryCaches(): Query caches successfully cleared of XX queries
  2. Schema index cleanup job closed
  3. [o.n.k.r.Recovery] [neo4j/6e6ac477] XX% completed
4. If the database is corrupted, then please follow the steps below. Symptoms of this are the database not starting, applications not loading, or errors like 'Index clean jobs can't be added after collector start'.
  1. Stop the neo4j databases and take an offline dump
  2. Restore the offline dump using --force option, so that old data will be removed in the databases.
  3. Restart the neo4j database service
If you do not find the information you are looking for or solution for your problem, please contact CAST Technical Support

Notes/comments