Database upgrades get extensive testing and validation since it is a critical component for all customers. However, there are times when the upgrade fails and this article covers the key things to do.
Prepare. Upgrades require planning. The Workgroup Server, management application, and security application all need to be stopped. You do not want the databases to be in use during upgrade. Once stopped, take a full backup of the databases so that you can always come back to this point. Failures such as running out of resources in the database usually happen in the middle of changes. Trying to restart at the exact point of failure can be difficult.
Use NSQJDBCMK. Nsqjdbcmk is the intended way to run the upgrade. An upgrade can be a combination of table changes (DDL) as well as new contents (DML). Some companies will allow only the DBA to make changes. One option that is often permitted is to use 2 IDs. One, for normal operation, that has permissions to change the contents, and a second, which is only used by nsqjdbcmk, that has the required administrative rights. Otherwise, it is possible for a DBA to run the scripts directly, noting they use a special command separator. Using manual tools such as script editors work, but create a chance that something will be missed.
Test. Most customers have a test system that they upgrade first. This provides a degree of validation but it is rare that this test is a full representation of usage. A more robust approach is to restore a copy of the production databases and do a trial upgrade run against them. If this is successful as a test, it increases the chances of success for the real upgrade.
Open a support ticket. Open a support ticket right away if a problem occurs. Unless the error is something like "expired password", trying to analyze the error on your own may be difficult.
Restore if required. If the upgrade fails, be ready to execute your plan to back out the changes. Depending on the cause of the failure, database resources, product defects or user errors, the resolution make require some time. Since upgrades are typically done in a change window there may not be time to correct the error.