Migrating Elasticsearch indices to GitHub Enterprise Server 2.14 or later
To prepare for an upgrade to GitHub Enterprise Server 2.14, you'll need to migrate your indices to Elasticsearch 5.6 with our migration script.
In this article
- Search indices
- Webhook indices
- Available indices
- Preparing a GitHub Enterprise Server 2.12 or 2.13 appliance
GitHub Enterprise Server 2.14 includes an upgrade to Elasticsearch 5.6. Before upgrading to GitHub Enterprise Server 2.14 or later from 2.12 or 2.13, we recommend you download, install, and run the Elasticsearch migration tools, so your largest indices are migrated online while your appliance still has online access.
Search indices
The migration script checks for any search indices first while the appliance is online. Migrating search indices can take a few minutes to a few days, depending on their size. For an example of large indices, these indices took a couple of days to migrate in our test environment.
admin@ip-172-31-2-141:~$ curl -s http://localhost:9200/_cat/indices?v | sort -n -k 6
green open blog-1 1 0 0 0 144b 144b
green open projects-1 1 0 0 0 144b 144b
green open registry-packages-1 1 0 0 0 144b 144b
green open showcases-1 1 0 0 0 144b 144b
health status index pri rep docs.count docs.deleted store.size pri.store.size
green open pull-requests-1 1 0 1 0 9.3kb 9.3kb
green open wikis-1 1 0 2 0 5kb 5kb
green open hookshot-logs-2018-05-29 5 0 25 0 124.2kb 124.2kb
green open repos-1 1 0 1638 1 1.4mb 1.4mb
green open gists-1 1 0 3531 64 291.9kb 291.9kb
green open audit_log-1-2018-06-1 1 0 11108 0 3mb 3mb
green open users-1 1 0 19866 56 2.7mb 2.7mb
green open hookshot-logs-2018-05-31 5 0 20000 0 33.4mb 33.4mb
green open hookshot-logs-2018-06-04 5 0 20000 0 32.6mb 32.6mb
green open issues-1 1 0 26405 6 82.8mb 82.8mb
green open hookshot-logs-2018-05-30 5 0 119744 0 196.8mb 196.8mb
green open audit_log-1-2018-05-1 1 0 191664 0 50mb 50mb
green open code-search-1 1 0 6932626 44 42.9gb 42.9gb
green open commits-1 1 0 63753587 1485 45.4gb 45.4gb
The search indices start with:
- blog-
- code-search-
- commits-
- gists-
- issues-
- labels-
- marketplace-listings-
- non-marketplace-listings-
- projects-
- pull-requests-
- registry-packages-
- repos-
- showcases-
- topics-
- users-
Webhook indices
After the migration script rebuilds the necessary search indices online, the script will check if any webhook indices need to be rebuilt. If you've run your appliance with GitHub Enterprise Server 2.12 or 2.13 for 14 days or longer, then you likely will not need your webhook indices rebuilt since webhook indices have a default retention policy of seven days. If you're updating your appliance from GitHub Enterprise 2.11 or earlier, then you may need to rebuild the webhook indices.
If any webhook indices need to be rebuilt, then you'll be prompted to enable maintenance mode before the script can rebuild the webhook indices. Although migrating webhook indices requires some downtime, large maintenance windows or downtime is not necessary.
The webhook indices start with hookshot-logs-.
Available indices
You can see available indices on your appliance using curl.
admin@ip-172-31-2-141:~$ curl -s http://localhost:9200/_cat/indices?v | sort -n -k 6
green open blog-1 1 0 0 0 144b 144b
green open projects-1 1 0 0 0 144b 144b
green open registry-packages-1 1 0 0 0 144b 144b
green open showcases-1 1 0 0 0 144b 144b
health status index pri rep docs.count docs.deleted store.size pri.store.size
green open pull-requests-1 1 0 1 0 9.3kb 9.3kb
green open wikis-1 1 0 2 0 5kb 5kb
green open hookshot-logs-2018-05-29 5 0 25 0 124.2kb 124.2kb
green open repos-1 1 0 1638 1 1.4mb 1.4mb
green open gists-1 1 0 3531 64 291.9kb 291.9kb
green open audit_log-1-2018-06-1 1 0 11108 0 3mb 3mb
green open users-1 1 0 19866 56 2.7mb 2.7mb
green open hookshot-logs-2018-05-31 5 0 20000 0 33.4mb 33.4mb
green open hookshot-logs-2018-06-04 5 0 20000 0 32.6mb 32.6mb
green open issues-1 1 0 26405 6 82.8mb 82.8mb
green open hookshot-logs-2018-05-30 5 0 119744 0 196.8mb 196.8mb
green open audit_log-1-2018-05-1 1 0 191664 0 50mb 50mb
green open code-search-1 1 0 6932626 44 42.9gb 42.9gb
green open commits-1 1 0 63753587 1485 45.4gb 45.4gb
Preparing a GitHub Enterprise Server 2.12 or 2.13 appliance
If you upgrade to GitHub Enterprise Server 2.14 or later without running the migration tools, the existing Elasticsearch indices may be invalid and won't work correctly. To run the Elasticsearch migration script, your GitHub Enterprise Server appliance must be running GitHub Enterprise 2.12 or 2.13.
Warning:
- Using GitHub Enterprise Server Backup Utilities will destroy old Elasticsearch indices not compatible with 5.X after restoring. In this case, manual reindexing could be necessary.
- If GitHub Enterprise Server is configured for High Availability, the migration script must run while replication is still running. The changes must be allowed to fully replicate to the other appliance before starting the upgrade. If replication is not running while the migration script runs, your Elasticsearch indexes may become invalid.
-
Authenticate to the primary appliance with High Availability enabled using SSH.
-
Download and install the migration script to the appliance:
$ wget https://github-enterprise.s3.amazonaws.com/util/es-5x-transition-tools.tar.gz $ sudo tar -C / -xvf es-5x-transition-tools.tar.gzIf you manage a GitHub Enterprise Server Cluster, authenticate to one of the Elasticsearch server nodes using SSH and install the migration tools there. Locate them using:
$ ghe-cluster-each -r elasticsearch -p ghe-test-data-0 ghe-test-data-1 ghe-test-data-2 -
Run the migration script:
$ /usr/local/share/enterprise/ghe-es-5x-migration -rNote: If you have
webhookindices to migrate, after running the online migrations, you'll be prompted to enable maintenance mode. -
If you’re running a GitHub Enterprise Server Cluster, follow the official upgrade documentation for single VMs or High Availability environments or the cluster upgrade guide. For more information, see "Upgrading GitHub Enterprise Server" or "Upgrading a cluster".