About evacuation of cluster nodes running data services
In a cluster configuration for GitHub Enterprise Server, you may need to take an individual node offline. For example, you may need to replace the node's virtual machine (VM). If the node you want to replace operates in the storage tier, GitHub recommends that you first evacuate the node's data services. Evacuation ensures that the remaining nodes contain the minimum expected copies of the data.
For more information about nodes and service tiers for GitHub Enterprise Server, see "About cluster nodes."
Warnings:
-
To avoid data loss during replacement of a node, GitHub strongly recommends evacuation of the applicable data services on the node before you take the node offline.
-
To ensure redundancy for any data service on your cluster, copies of data should exist on at least three nodes. For example, when four or more nodes store Git data, during evacuation, evacuated repository data will move from the node you're evacuating to the other three nodes. If you only have three nodes that store data for a service, evacuation of one node could fail and result in under-replicated data.
Evacuating a cluster node running data services
If you plan to take a node offline and the node runs any of the following roles, evacuate each applicable service before taking the node offline.
| Service | Data |
|---|---|
git-server | Repositories |
pages-server | Site builds for GitHub Pages |
storage-server |
|
-
SSH into any node in your cluster. For more information, see "Accessing the administrative shell (SSH)."
-
To find the UUID of the node to evacuate, run the following command. Replace HOSTNAME with the node's hostname. You'll use the UUID in subsequent commands.
ghe-config cluster.HOSTNAME.uuid -
For each applicable service on the node, to determine the initial data counts, run the following commands. For each command, replace UUID with the UUID from the previous step.
-
git-server:-
Command:
ghe-spokes evac-status git-server-UUID -
Relevant output:
Networks,Gists
-
-
pages-server:-
Command:
echo "select count(*) from pages_replicas where host = 'pages-server-UUID'" | ghe-dbconsole -y
-
-
storage-server:-
Command:
ghe-storage evacuation-status storage-server-UUID -
Relevant output:
Remaining item(s)
-
-
-
To evacuate an applicable service on the node, run the following commands. For each command, replace UUID with the UUID from the earlier step.
-
git-server:-
Command (replace REASON FOR EVACUATION with the reason for evacuation):
ghe-spokes server evacuate git-server-UUID 'REASON FOR EVACUATION'
-
-
pages-server:-
Command:
ghe-dpages evacuate pages-server-UUID
-
-
storage-server:-
Take the node's service offline by running the following command.
ghe-storage offline storage-server-UUID -
Evacuate the node by running the following command.
ghe-storage evacuate storage-server-UUID
-
-
-
To monitor evacuation of a service while GitHub Enterprise Server copies the data, run the following commands. For each command, replace UUID with the UUID from the earlier step.
Warning: Do not shut down the node until evacuation is complete. Evacuation is complete when the data counts reach zero, which means that all data is safely stored on other nodes.
-
git-server:ghe-spokes evac-status git-server-UUID -
pages-server:echo "select count(*) from pages_replicas where host = 'pages-server-UUID'" | ghe-dbconsole -y -
storage-server:ghe-storage evacuation-status storage-server-UUID
-
-
After evacuation completes for the service, shut down the node.