Monitoring and troubleshooting self-hosted runners

Note: GitHub Actions was available for GitHub Enterprise Server 2.22 as a limited beta. The beta has ended. GitHub Actions is now generally available in GitHub Enterprise Server 3.0 or later. For more information, see the GitHub Enterprise Server 3.0 release notes.

For more information about upgrading to GitHub Enterprise Server 3.0 or later, see "Upgrading GitHub Enterprise Server."
For more information about configuring GitHub Actions after you upgrade, see the documentation for GitHub Enterprise Server 3.0.

Note: GitHub-hosted runners are not currently supported on GitHub Enterprise Server. You can see more information about planned future support on the GitHub public roadmap.

Checking the status of a self-hosted runner

A self-hosted runner can be located in either your repository, organization, or enterprise settings on your GitHub Enterprise Server instance. To manage a self-hosted runner, you must have the following permissions, depending on where the self-hosted runner was added:

User repository: You must be the repository owner.
Organization: You must be an organization owner.
Organization repository: You must be an organization owner, or have admin access to the repository.
Enterprise: You must be a GitHub Enterprise site administrator.

In your organization or repository, navigate to the main page and click Settings.
In the left sidebar, click Actions.
Under "Self-hosted runners", you can view a list of registered runners, including the runner's name, labels, and status.

The status can be one of the following:
- Idle: The runner is connected to GitHub Enterprise Server and is ready to execute jobs.
- Active: The runner is currently executing a job.
- Offline: The runner is not connected to GitHub Enterprise Server. This could be because the machine is offline, the self-hosted runner application is not running on the machine, or the self-hosted runner application cannot communicate with GitHub Enterprise Server.

Reviewing the self-hosted runner application log files

You can monitor the status of the self-hosted runner application and its activities. Log files are kept in the _diag directory, and a new one is generated each time the application is started. The filename begins with Runner_, and is followed by a UTC timestamp of when the application was started.

For detailed logs on workflow job executions, see the next section describing the Worker_ files.

Reviewing a job's log file

The self-hosted runner application creates a detailed log file for each job that it processes. These files are stored in the _diag directory, and the filename begins with Worker_.

Using journalctl to check the self-hosted runner application service

For Linux-based self-hosted runners running the application using a service, you can use journalctl to monitor their real-time activity. The default systemd-based service uses the following naming convention: actions.runner.<org>-<repo>.<runnerName>.service. This name is truncated if it exceeds 80 characters, so the preferred way of finding the service's name is by checking the .service file. For example:

$ cat ~/actions-runner/.service
actions.runner.octo-org-octo-repo.runner01.service

You can use journalctl to monitor the real-time activity of the self-hosted runner:

$ sudo journalctl -u actions.runner.octo-org-octo-repo.runner01.service -f

In this example output, you can see runner01 start, receive a job named testAction, and then display the resulting status:

Feb 11 14:57:07 runner01 runsvc.sh[962]: Starting Runner listener with startup type: service
Feb 11 14:57:07 runner01 runsvc.sh[962]: Started listener process
Feb 11 14:57:07 runner01 runsvc.sh[962]: Started running service
Feb 11 14:57:16 runner01 runsvc.sh[962]: √ Connected to GitHub
Feb 11 14:57:17 runner01 runsvc.sh[962]: 2020-02-11 14:57:17Z: Listening for Jobs
Feb 11 16:06:54 runner01 runsvc.sh[962]: 2020-02-11 16:06:54Z: Running job: testAction
Feb 11 16:07:10 runner01 runsvc.sh[962]: 2020-02-11 16:07:10Z: Job testAction completed with result: Succeeded

To view the systemd configuration, you can locate the service file here: /etc/systemd/system/actions.runner.<org>-<repo>.<runnerName>.service. If you want to customize the self-hosted runner application service, do not directly modify this file. Follow the instructions described in "Configuring the self-hosted runner application as a service."

Using launchd to check the self-hosted runner application service

For macOS-based self-hosted runners running the application as a service, you can use launchctl to monitor their real-time activity. The default launchd-based service uses the following naming convention: actions.runner.<org>-<repo>.<runnerName>. This name is truncated if it exceeds 80 characters, so the preferred way of finding the service's name is by checking the .service file in the runner directory:

% cat ~/actions-runner/.service
/Users/exampleUsername/Library/LaunchAgents/actions.runner.octo-org-octo-repo.runner01.plist

The svc.sh script uses launchctl to check whether the application is running. For example:

$ ./svc.sh status
status actions.runner.example.runner01:
/Users/exampleUsername/Library/LaunchAgents/actions.runner.example.runner01.plist
Started:
379 0 actions.runner.example.runner01

The resulting output includes the process ID and the name of the application’s launchd service.

To view the launchd configuration, you can locate the service file here: /Users/exampleUsername/Library/LaunchAgents/actions.runner.<repoName>.<runnerName>.service. If you want to customize the self-hosted runner application service, do not directly modify this file. Follow the instructions described in "Configuring the self-hosted runner application as a service."

Using PowerShell to check the self-hosted runner application service

For Windows-based self-hosted runners running the application as a service, you can use PowerShell to monitor their real-time activity. The service uses the naming convention GitHub Actions Runner (<org>-<repo>.<runnerName>). You can also find the service's name by checking the .service file in the runner directory:

PS C:\actions-runner> Get-Content .service
actions.runner.octo-org-octo-repo.runner01.service

You can view the status of the runner in the Windows Services application (services.msc). You can also use PowerShell to check whether the service is running:

PS C:\actions-runner> Get-Service "actions.runner.octo-org-octo-repo.runner01.service" | Select-Object Name, Status
Name                                                  Status
----                                                  ------
actions.runner.octo-org-octo-repo.runner01.service    Running

You can use PowerShell to check the recent activity of the self-hosted runner. In this example output, you can see the application start, receive a job named testAction, and then display the resulting status:

PS C:\actions-runner> Get-EventLog -LogName Application -Source ActionsRunnerService

   Index Time          EntryType   Source                 InstanceID Message
   ----- ----          ---------   ------                 ---------- -------
     136 Mar 17 13:45  Information ActionsRunnerService          100 2020-03-17 13:45:48Z: Job Greeting completed with result: Succeeded
     135 Mar 17 13:45  Information ActionsRunnerService          100 2020-03-17 13:45:34Z: Running job: testAction
     134 Mar 17 13:41  Information ActionsRunnerService          100 2020-03-17 13:41:54Z: Listening for Jobs
     133 Mar 17 13:41  Information ActionsRunnerService          100 û Connected to GitHub
     132 Mar 17 13:41  Information ActionsRunnerService            0 Service started successfully.
     131 Mar 17 13:41  Information ActionsRunnerService          100 Starting Actions Runner listener
     130 Mar 17 13:41  Information ActionsRunnerService          100 Starting Actions Runner Service
     129 Mar 17 13:41  Information ActionsRunnerService          100 create event log trace source for actions-runner service

Monitoring the automatic update process

We recommend that you regularly check the automatic update process, as the self-hosted runner will not be able to process jobs if it falls below a certain version threshold. The self-hosted runner application automatically updates itself, but note that this process does not include any updates to the operating system or other software; you will need to separately manage these updates.

You can view the update activities in the Runner_ log files. For example:

[Feb 12 12:37:07 INFO SelfUpdater] An update is available.

In addition, you can find more information in the SelfUpdate log files located in the _diag directory.

Troubleshooting containers in self-hosted runners

Checking that Docker is installed

If your jobs require containers, then the self-hosted runner must be Linux-based and needs to have Docker installed. Check that your self-hosted runner has Docker installed and that the service is running.

You can use systemctl to check the service status:

$ sudo systemctl is-active docker.service
active

If Docker is not installed, then dependent actions will fail with the following errors:

[2020-02-13 16:56:10Z INFO DockerCommandManager] Which: 'docker'
[2020-02-13 16:56:10Z INFO DockerCommandManager] Not found.
[2020-02-13 16:56:10Z ERR  StepsRunner] Caught exception from step: System.IO.FileNotFoundException: File not found: 'docker'

Checking the Docker permissions

If your job fails with the following error:

dial unix /var/run/docker.sock: connect: permission denied

Check that the self-hosted runner's service account has permission to use the Docker service. You can identify this account by checking the configuration of the self-hosted runner in systemd. For example:

$ sudo systemctl show -p User actions.runner.octo-org-octo-repo.runner01.service
User=runner-user

Explore by product

In this article