Known issues with upgrades to your instance

See an overview of workarounds for issues that impact the upgrade process for GitHub Enterprise Server, or impact your instance after you complete an upgrade.

In this article

About known issues with GitHub Enterprise Server upgrades

GitHub is aware of the following issues that could impact upgrades to new releases of GitHub Enterprise Server. For more information, see "Known issues" in the GitHub Enterprise Server release notes.

GitHub strongly recommends regular backups of your instance's configuration and data. Before you proceed with any upgrade, back up your instance, then validate the backup in a staging environment. For more information, see "Configuring backups on your instance" and "Setting up a staging instance."

Elasticsearch Upgrade

As part of upgrading GitHub Enterprise Server to version 3.13 or later, the Elasticsearch service will be upgraded. GitHub strongly recommends following the guidance in "Preparing for the Elasticsearch upgrade in GitHub Enterprise Server 3.13."

Undecryptable records

If you are upgrading from GitHub Enterprise Server 3.11 or 3.12 to 3.13, or from 3.12 to 3.14, you may run into an issue with undecryptable records due to missing required keys for decryption. The only solution is to delete the undecryptable records. The type of records impacted by this issue are 2FA records, that means you might need to ask users to re-enable two-factor authentication (2FA).

Before upgrading

If you are upgrading from GitHub Enterprise Server 3.11 or 3.12 to 3.13, or from 3.12 to 3.14, you can run the encryption diagnostics script to identify the undecryptable records ahead of time. This will give you the opportunity to understand the impact and plan for it.

  1. Download the encryption diagnostics script. You can use a command like curl -L -O https://gh.io/ghes-encryption-diagnostics to download the script.
  2. Save the script to the /data/user/common directory on the appliance.
  3. Follow the instructions at the top of the script and execute it on the appliance. If there are any undecryptable records, they are logged in /tmp/column_encryption_records_to_be_deleted.log. Any records logged here means that the system was not able to find the keys for them and hence was not able to decrypt the data in those records.

At this stage, please note that these records will be deleted as part of the process. The script will warn you about the users who will need to re-enroll into 2FA after the upgrade. The impacted users' handles are logged in /tmp/column_encryption_users_to_have_2fa_disabled.log. These users will need to be re-enrolled into 2FA.

If the script runs into unexpected issues, you will be prompted to contact GitHub Support. Errors related to these issues will be logged in /tmp/column_encryption_unexpected_errors.log. If you are in a dire situation and are unable to have users re-enroll into 2FA, contact GitHub Support for help.

During the upgrade

In case you did not have the opportunity to run the encryption diagnostics script ahead of time, there are mechanisms in the product to help you. The pre-flight checks during the upgrade process will detect undecryptable records and log them in /tmp/column_encryption_records_to_be_deleted.log. The sequence will warn you of the users who will need to re-enable 2FA after the upgrade. The impacted users records are logged in /tmp/column_encryption_users_to_have_2fa_disabled.log.

If undecryptable records are detected, you will be prompted whether you want to proceed with the upgrade or not. If you proceed, the upgrade process deletes the undecryptable records. Otherwise, the upgrade process will exit.

If you have any questions during the upgrade, you can reach out to GitHub Support. Once you have had the time and opportunity to understand the impact, you can retrigger the upgrade.