Evacuating a cluster node running data services

If a node in your GitHub Enterprise Server cluster runs services that store distributed data, you can ensure redundancy as you prepare to replace the node by evacuating the node's data.

Who can use this feature?

GitHub determines eligibility for clustering, and must enable the configuration for your instance's license. Clustering requires careful planning and additional administrative overhead. For more information, see "About clustering."

In this article

About evacuation of cluster nodes running data services

In a cluster configuration for GitHub Enterprise Server, you may need to take an individual node offline. For example, you may need to replace the node's virtual machine (VM). If the node you want to replace operates in the storage tier, GitHub recommends that you first evacuate the node's data services. Evacuation ensures that the remaining nodes contain the minimum expected copies of the data.

For more information about nodes and service tiers for GitHub Enterprise Server, see "About cluster nodes."

Warning

To avoid data loss during replacement of a node, GitHub strongly recommends evacuation of the applicable data services on the node before you take the node offline.
To ensure redundancy for any data service on your cluster, copies of data should exist on at least three nodes. For example, when four or more nodes store Git data, during evacuation, evacuated repository data will move from the node you're evacuating to the other three nodes. If you only have three nodes that store data for a service, evacuation of one node could fail and result in under-replicated data.

Evacuating a cluster node running data services

If you plan to take a node offline and the node runs any of the following roles, evacuate each applicable service before taking the node offline.

Service	Data
`git-server`	Repositories
`pages-server`	Site builds for GitHub Pages
`storage-server`	Data stored in repositories using Git Large File Storage Avatar images File attachments from comments in the web UI Release archives

SSH into any node in your cluster. For more information, see "Accessing the administrative shell (SSH)."
To find the UUID of the node to evacuate, run the following command. Replace HOSTNAME with the node's hostname. You'll use the UUID in subsequent commands.
```
ghe-config cluster.HOSTNAME.uuid
```
For each applicable service on the node, to determine the initial data counts, run the following commands. For each command, replace UUID with the UUID from the previous step.
- git-server:
  - Command:
```
ghe-spokesctl server status git-server-UUID
```
  - Relevant output: NETWORKS, GISTS
- pages-server:
  - Command:
```
echo "select count(*) from pages_replicas where host = 'pages-server-UUID'" | ghe-dbconsole -y
```
- storage-server:
  - Command:
```
ghe-storage evacuation-status storage-server-UUID
```
  - Relevant output: Remaining item(s)
To evacuate an applicable service on the node, run the following commands. For each command, replace UUID with the UUID from the earlier step.
- git-server:
  - Command (replace REASON FOR EVACUATION with the reason for evacuation):
```
ghe-spokesctl server set evacuating git-server-UUID 'REASON FOR EVACUATION'
```
- pages-server:
  - Command:
```
ghe-dpages evacuate pages-server-UUID
```
- storage-server:
  1. Take the node's service offline by running the following command.
```
ghe-storage offline storage-server-UUID
```
  2. Evacuate the node by running the following command.
```
ghe-storage evacuate storage-server-UUID
```
To monitor evacuation of a service while GitHub Enterprise Server copies the data, run the following commands. For each command, replace UUID with the UUID from the earlier step.

Warning

Do not shut down the node until evacuation is complete. Evacuation is complete when the data counts reach zero, which means that all data is safely stored on other nodes.
- git-server:
```
ghe-spokesctl server evac-status git-server-UUID
```
- pages-server:
```
echo "select count(*) from pages_replicas where host = 'pages-server-UUID'" | ghe-dbconsole -y
```
- storage-server:
```
ghe-storage evacuation-status storage-server-UUID
```
After evacuation completes for the service, shut down the node.