Enterprise Server 3.14 release notes

Packages have been updated to the latest security versions.

On an instance in a cluster configuration, ghe-repl-promote failed if the primary node was unavailable.

In a high availability configuration, with GitHub Actions, replication would fail on nodes where MSSQL was not configured to run.

The --no-async flag was not implemented for the ghe-cluster-support-bundle command, leading to a potentially increased load.

Pre-receive hook environments with shared memory enabled could not access shared memory at runtime.

For instances hosted on Azure, if a pre-upgrade check failed due to insufficient user disk size, the Management Console displayed an internal server error.

The Enterprise Overview page incorrectly displayed a Beta label, even though it is generally available.

After a user made changes to the isolated subdomain setting, some user assets did not display properly.

On an instance with secret scanning enabled, when selecting repositories for a dry run of an enterprise-level custom pattern, searches for full repository names (ORGANIZATION/REPOSITORY) did not return results.

When adding bypass permissions to a ruleset, the dropdown menu failed to load if one of the suggested actors was an invalid integration.

When creating a pre-receive hook environment, attempts to include an image URL over 255 characters failed with a database error. The maximum length is still 255 characters, but the URL length is now validated before the process starts.

On an instance with GitHub Actions disabled, status check icons on a repositorys commit list failed to render.

Site administrators were unable to use the "Disable repository access" functionality on the site admin dashboard.

Attempting to access the code security settings page for a non-existent enterprise returned a 500 error instead of a 404 error.

Performing a browser back navigation to a pull request now displays up-to-date status checks.

Jekyll-build tooling for GitHub pages could fail when using the jekyll-relative-links plugin, see Failure details.

The removal rate of issues from Git repositories was slower than necessary.

Log output for git maintenance now includes the time taken to complete the maintenance process.

When exporting repositories to blob storage using the migrations REST API endpoint to start an organization migration, the maximum compressed archive size is limited to 90 GB. This is an increase from 30 GB.

Removes the minimum date for the new commit filter bar.

When exporting repositories using the migrations REST API, prior to blob storage upload the tarball is staged in the root volume. For more disk capacity, the tarball will now be staged in the data volume.

During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see Troubleshooting access to the Management Console.

On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1.

In some situations, large .adoc files stored in a repository do not render properly in the web UI. The raw contents are still available to view as plaintext.

Repositories originally imported using ghe-migrator will not correctly track Advanced Security contributions.

Admin stats REST API endpoints may timeout on appliances with many users or repositories. Retrying the request until data is returned is advised.

When following the steps for Replacing the primary MySQL node, step 14 (running ghe-cluster-config-apply) might fail with errors. If this occurs, re-running ghe-cluster-config-apply is expected to succeed.

Running a config apply as part of the steps for Replacing a node in an emergency may fail with errors if the node being replaced is still reachable. If this occurs, shutdown the node and repeat the steps.

If a hotpatch upgrade requires the haproxy-frontend service to be restarted, the restart will hang if there are existing long-lived connections, such as browser web sockets or Git operations. No new connections will be accepted for up to 5 minutes. Any existing unfinished connections at this time will be disconnected.

When restoring data originally backed up from a 3.13 appliance onto a 3.13 appliance, the elasticsearch indices need to be reindexed before some of the data will show up. This happens via a nightly scheduled job. It can also be forced by running /usr/local/share/enterprise/ghe-es-search-repair.

An organization-level code scanning configuration page is displayed on instances that do not use GitHub Advanced Security or code scanning.

When following the steps for Replacing the primary MySQL node, step 14 (running ghe-cluster-config-apply) might fail with errors. If this occurs, re-running ghe-cluster-config-apply is expected to succeed.

In the header bar displayed to site administrators, some icons are not available.

When enabling automatic update checks for the first time in the Management Console, the status is not dynamically reflected until the "Updates" page is reloaded.

When restoring from a backup snapshot, a large number of mapper_parsing_exception errors may be displayed.

LOW: Instance administrators could see tokens used to authenticate against gitauth in plaintext in/var/log/github-audit.log.

Packages have been updated to the latest security versions.

Embedded images in wiki pages were broken.

During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see Troubleshooting access to the Management Console.

On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1.

In some situations, large .adoc files stored in a repository do not render properly in the web UI. The raw contents are still available to view as plaintext.

Repositories originally imported using ghe-migrator will not correctly track Advanced Security contributions.

Admin stats REST API endpoints may timeout on appliances with many users or repositories. Retrying the request until data is returned is advised.

When following the steps for Replacing the primary MySQL node, step 14 (running ghe-cluster-config-apply) might fail with errors. If this occurs, re-running ghe-cluster-config-apply is expected to succeed.

Running a config apply as part of the steps for Replacing a node in an emergency may fail with errors if the node being replaced is still reachable. If this occurs, shutdown the node and repeat the steps.

If a hotpatch upgrade requires the haproxy-frontend service to be restarted, the restart will hang if there are existing long-lived connections, such as browser web sockets or Git operations. No new connections will be accepted for up to 5 minutes. Any existing unfinished connections at this time will be disconnected.

When restoring data originally backed up from a 3.13 appliance onto a 3.13 appliance, the elasticsearch indices need to be reindexed before some of the data will show up. This happens via a nightly scheduled job. It can also be forced by running /usr/local/share/enterprise/ghe-es-search-repair.

An organization-level code scanning configuration page is displayed on instances that do not use GitHub Advanced Security or code scanning.

When following the steps for Replacing the primary MySQL node, step 14 (running ghe-cluster-config-apply) might fail with errors. If this occurs, re-running ghe-cluster-config-apply is expected to succeed.

In the header bar displayed to site administrators, some icons are not available.

When enabling automatic update checks for the first time in the Management Console, the status is not dynamically reflected until the "Updates" page is reloaded.

When restoring from a backup snapshot, a large number of mapper_parsing_exception errors may be displayed.

Services may respond with a 503 status due to an out of date haproxy configuration. This can usually be resolved with a ghe-config-apply run.

Attempting to stop replications after stopping GitHub Actions on a GitHub Enterprise Server instance would fail, reporting that MSSQL was not responding. The can be avoided by start MSSQL prior to stopping replication /usr/local/share/enterprise/ghe-nomad-jobs queue /etc/nomad-jobs/mssql/mssql.hcl

Customers performing a feature version upgrade to 3.13.6 or 3.14.3 may experience issues with database migrations due to data issues during database conversions.

During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see Troubleshooting access to the Management Console.

On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1.

In some situations, large .adoc files stored in a repository do not render properly in the web UI. The raw contents are still available to view as plaintext.

Repositories originally imported using ghe-migrator will not correctly track Advanced Security contributions.

Admin stats REST API endpoints may timeout on appliances with many users or repositories. Retrying the request until data is returned is advised.

When following the steps for Replacing the primary MySQL node, step 14 (running ghe-cluster-config-apply) might fail with errors. If this occurs, re-running ghe-cluster-config-apply is expected to succeed.

Running a config apply as part of the steps for Replacing a node in an emergency may fail with errors if the node being replaced is still reachable. If this occurs, shutdown the node and repeat the steps.

If a hotpatch upgrade requires the haproxy-frontend service to be restarted, the restart will hang if there are existing long-lived connections, such as browser web sockets or Git operations. No new connections will be accepted for up to 5 minutes. Any existing unfinished connections at this time will be disconnected.

When restoring data originally backed up from a 3.13 appliance onto a 3.13 appliance, the elasticsearch indices need to be reindexed before some of the data will show up. This happens via a nightly scheduled job. It can also be forced by running /usr/local/share/enterprise/ghe-es-search-repair.

An organization-level code scanning configuration page is displayed on instances that do not use GitHub Advanced Security or code scanning.

When following the steps for Replacing the primary MySQL node, step 14 (running ghe-cluster-config-apply) might fail with errors. If this occurs, re-running ghe-cluster-config-apply is expected to succeed.

In the header bar displayed to site administrators, some icons are not available.

When enabling automatic update checks for the first time in the Management Console, the status is not dynamically reflected until the "Updates" page is reloaded.

When restoring from a backup snapshot, a large number of mapper_parsing_exception errors may be displayed.

Services may respond with a 503 status due to an out of date haproxy configuration. This can usually be resolved with a ghe-config-apply run.

Attempting to stop replications after stopping GitHub Actions on a GHES instanstance would fail, reporting that MSSQL was not responding. The can be avoided by start MSSQL prior to stopping replication /usr/local/share/enterprise/ghe-nomad-jobs queue /etc/nomad-jobs/mssql/mssql.hcl.

When operating in a high availability configuration, running ghe-repl-promote on a replica node will fail if the original primary cannot be reached by the replica node. This is because the ghe-repl-promote script attempts to decommission all Elasticsearch nodes other than the promoted node, however these requests are made to the original primary node which is no longer reachable. The error message written to the terminal will be similar to:

Maintenance mode has been enabled for active replica <REPLICA_HOSTNAME>
{"message": "No server is currently available to service your request. Sorry about that. Please try resubmitting your request and contact your local GitHub Enterprise site administrator if the problem persists."}
jq: error (at :3): Cannot index string with string "node"

If this occurs, workaround this issue by running the following command — this changes the ghe-repl-promote script in place:

sudo sed -i.bak -e '/for node_hostname in/i if ! $forced; then' -e '/^  done/a fi' /usr/local/bin/ghe-repl-promote

Then re-run the updated ghe-repl-promote script.

[Updated: 2024-11-29]

Elasticsearch packages have been updated to the latest security versions.

Packages have been updated to the latest security version.

HIGH: An attacker could bypass SAML single sign-on (SSO) authentication with the optional encrypted assertions feature, allowing unauthorized provisioning of users and access to the instance, by exploiting an improper verification of cryptographic signatures vulnerability in GitHub Enterprise Server. This is a follow up fix for CVE-2024-9487 to further harden the encrypted assertions feature against this type of attack. Please note that encrypted assertions are not enabled by default. Instances not utilizing SAML SSO, or utilizing SAML SSO authentication without encrypted assertions, are not impacted. Additionally, an attacker would require direct network access as well as a signed SAML response or metadata document to exploit this vulnerability.

HIGH: An attacker with Enterprise Administrator access to the GitHub Enterprise Server instance could escalate privileges to SSH root access. This is achieved by exploiting the pre-receive hook environment to bypass symlink checks in the ghe-firejail path and execute malicious scripts. GitHub has requested CVE ID CVE-2024-10007 for this vulnerability, which was reported via the GitHub Bug Bounty program. [Updated: 2024-11-07]

When saving settings in the Management Console, the configuration run would stop if the enterprise-manage process was restarted.

On an instance with GitHub Actions enabled, some maintenance tasks could fail due to incomplete upgrade steps during previous upgrades to new releases of GitHub Enterprise Server.

A repeated error message concerning connectivity to port 6002 was emitted to the system logs when GitHub Actions was enabled.

The initial setup certificate generation in AWS took longer than expected due to fallback to private IPs. The time for this fallback has been reduced.

The ghe-support-bundle generation would fail when the aqueduct-lite service is down.

If the primary instance was unreachable, running ghe-repl-stop --force on a replica would fail during the config apply run.

Administrators in the SCIM private beta (versions < 3.14) that decided to upgrade their private beta appliance see an incorrectly checked box in the "SCIM Configuration" section of the Enterprise settings authentication security page in 3.14.

Certain URLs may have caused a 500 error on instances that use the mandatory message feature logging.

When restoring from a backup, repositories that had been deleted in the last 90 days were not completely restored.

For instances that use secret scanning, custom messages for push protection set by the enterprise did not display to users.

Restoring Git repositories using backup-utils occasionally failed.

Enterprise installations experienced unpredictable repository search results due to the default 4,000 repository limit. A relaxed repository filter mode, which includes all single-tenant organization repositories and bypasses the limit, has been introduced. Administrators can enable this mode using ghe-config app.github.enterprise-repo-search-filter-enabled true && ghe-config-apply.

Running config-apply became stuck under certain circumstances due to a misconfiguration with Packages and Elasticsearch.

Audit log events for secret scanning alerts incorrectly displayed a blank secret type when generated for a custom pattern.

Some customers upgrading to 3.14 may experience issues with undecryptable records during the upgrade. This issue has now been resolved. We recommend you read Undecryptable records.

When connecting to an appliance via SSH, a notification about upcoming root disk changes displays.

During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. See Troubleshooting access to the Management Console.

On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1.

In some situations, large .adoc files stored in a repository do not render properly in the web UI. The raw contents are still available to view as plaintext.

Repositories originally imported using ghe-migrator will not correctly track GitHub Advanced Security contributions.

Admin stats REST API endpoints may timeout on appliances with many users or repositories. Retrying the request until data is returned is advised.

When following the steps for Replacing the primary MySQL node, step 14 (running ghe-cluster-config-apply) might fail with errors. If this occurs, re-running ghe-cluster-config-apply is expected to succeed.

Running a config apply as part of the steps for Replacing a node in an emergency may fail with errors if the node being replaced is still reachable. If this occurs, shutdown the node and repeat the steps.

If a hotpatch upgrade requires the haproxy-frontend service to be restarted, the restart will hang if there are existing long-lived connections, such as browser web sockets or Git operations. No new connections will be accepted for up to 5 minutes. Any existing unfinished connections at this time will be disconnected.

When restoring data originally backed up from a 3.13 appliance onto a 3.13 appliance, the Elasticsearch indices need to be reindexed before some of the data will show up. This happens via a nightly scheduled job. It can also be forced by running /usr/local/share/enterprise/ghe-es-search-repair.

An organization-level code scanning configuration page is displayed on instances that do not use GitHub Advanced Security or code scanning.

In the header bar displayed to site administrators, some icons are not available.

When enabling automatic update checks for the first time in the Management Console, the status is not dynamically reflected until the "Updates" page is reloaded.

When restoring from a backup snapshot, a large number of mapper_parsing_exception errors may be displayed.

Services may respond with a 503 status due to an out of date haproxy configuration. This can usually be resolved with a ghe-config-apply run.

Customers doing feature version upgrade to 3.14.3 may experience issues with database migrations due to data issues during database conversions. [Added: 2024-11-08]

When operating in a high availability configuration, running ghe-repl-promote on a replica node will fail if the original primary cannot be reached by the replica node. This is because the ghe-repl-promote script attempts to decommission all Elasticsearch nodes other than the promoted node, however these requests are made to the original primary node which is no longer reachable. The error message written to the terminal will be similar to:

Maintenance mode has been enabled for active replica <REPLICA_HOSTNAME>
{"message": "No server is currently available to service your request. Sorry about that. Please try resubmitting your request and contact your local GitHub Enterprise site administrator if the problem persists."}
jq: error (at :3): Cannot index string with string "node"

If this occurs, workaround this issue by running the following command — this changes the ghe-repl-promote script in place:

sudo sed -i.bak -e '/for node_hostname in/i if ! $forced; then' -e '/^  done/a fi' /usr/local/bin/ghe-repl-promote

Then re-run the updated ghe-repl-promote script.

[Updated: 2024-11-29]

A sensitive data exposure in HTML forms was possible in the management console. To mitigate this issue, the "Copy Storage Setting from Actions" functionality was removed from the management console.

MEDIUM: Malicious URLs for SVG assets provided information about a victim user who clicked the URL, allowing an attacker to retrieve metadata belonging to the user and use it to generate a convincing phishing page. This required the attacker to upload malicious SVGs and phish a victim user to click the URL for the uploaded asset. GitHub has requested CVE ID CVE-2024-9539. This vulnerability was reported via the GitHub Bug Bounty program.

HIGH: An attacker could bypass SAML single sign-on (SSO) authentication with the optional encrypted assertions feature, allowing unauthorized provisioning of users and access to the instance, by exploiting an improper verification of cryptographic signatures vulnerability in GitHub Enterprise Server. This was a regression introduced as part of follow-up remediation from CVE-2024-4985, which resulted in a new variant of the vulnerability. Please note that encrypted assertions are not enabled by default. Instances not utilizing SAML SSO, or utilizing SAML SSO authentication without encrypted assertions, are not impacted. Additionally, an attacker would require direct network access as well as a signed SAML response or metadata document. GitHub has requested CVE ID CVE-2024-9487. This vulnerability was reported via the GitHub Bug Bounty program.

A missing configuration value would cause Dependabot to be unable to create group update pull requests.

HAProxy reloading was failure prone, which could lead to failed Git operations. This reloading process has been replaced with a more resilient Systemd process.

This error message mbind: Operation not permitted was repeatedly showing in the /var/log/mysql/mysql.err MySQL logs.

The backup of audit logs could take longer after upgrading to Elasticsearch 8.

An unhandled nil value when configuring Actions storage with AWS S3 via OIDC configuration in the terminal could cause an error.

Users were unable to sign out from gist pages.

On an instance with secret scanning enabled, the custom pattern page would not load because dry run results were tied to a deleted repository.

Suspended users were not always correctly routed to the correct "suspended" page.

The "List teams" API endpoint returned duplicate results when paginating.

When managing the organization permissions required for fine-grained personal access tokens, for custom properties or projects, the Admin access level could not be selected.

A model with no URL could cause a ghe-migrator import to fail.

The ghe-spokesctl status command showed repaired repositories as broken if their network ID changed during the repair (for example, when the repository was detached from it's original network).

Missing URLs on import could lead to migration failures without logging or explanation.

On the security overview dashboard, data initialization could fail when creating new organizations or changing GitHub Advanced Security licensing.

Restore could fail when restoring MySQL using backup-utils.

ghe-remove-node will display the log file location when running in quiet mode.

Pre-receive hook environments can use the clone3() system call.

The creation, deletion, or change in visibility of a gist has been added to the audit log.

During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see Troubleshooting access to the Management Console.

On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1.

In some situations, large .adoc files stored in a repository do not render properly in the web UI. The raw contents are still available to view as plaintext.

Repositories originally imported using ghe-migrator will not correctly track Advanced Security contributions.

The admin stats REST API endpoints may time out on appliances with many users or repositories. Retrying the request until data is returned is advised.

When following the steps for Replacing the primary MySQL node, step 14 (running ghe-cluster-config-apply) might fail with errors. If this occurs, re-running ghe-cluster-config-apply is expected to succeed.

Running a config apply as part of the steps for Replacing a node in an emergency may fail with errors if the node being replaced is still reachable. If this occurs, shutdown the node and repeat the steps.

If a hotpatch upgrade requires the haproxy-frontend service to be restarted, the restart will hang if there are existing long-lived connections, such as browser web sockets or Git operations. No new connections will be accepted for up to 5 minutes. Any existing unfinished connections at this time will be disconnected.

When restoring data originally backed up from a 3.13 appliance onto a 3.13 appliance, the elasticsearch indices need to be reindexed before some of the data will show up. This happens via a nightly scheduled job. It can also be forced by running /usr/local/share/enterprise/ghe-es-search-repair.

An organization-level code scanning configuration page is displayed on instances that do not use GitHub Advanced Security or code scanning.

In the header bar displayed to site administrators, some icons are not available.

When enabling automatic update checks for the first time in the Management Console, the status is not dynamically reflected until the "Updates" page is reloaded.

When restoring from a backup snapshot, a large number of mapper_parsing_exception errors may be displayed.

Services may respond with a 503 status due to an out of date haproxy configuration. This can usually be resolved with a ghe-config-apply run.

Images embedded in wiki pages may stop rendering shortly after being published. [Updated: 2024-10-16]

When operating in a high availability configuration, running ghe-repl-promote on a replica node will fail if the original primary cannot be reached by the replica node. This is because the ghe-repl-promote script attempts to decommission all Elasticsearch nodes other than the promoted node, however these requests are made to the original primary node which is no longer reachable. The error message written to the terminal will be similar to:

Maintenance mode has been enabled for active replica <REPLICA_HOSTNAME>
{"message": "No server is currently available to service your request. Sorry about that. Please try resubmitting your request and contact your local GitHub Enterprise site administrator if the problem persists."}
jq: error (at :3): Cannot index string with string "node"

If this occurs, workaround this issue by running the following command — this changes the ghe-repl-promote script in place:

sudo sed -i.bak -e '/for node_hostname in/i if ! $forced; then' -e '/^  done/a fi' /usr/local/bin/ghe-repl-promote

Then re-run the updated ghe-repl-promote script.

[Updated: 2024-11-29]

The option to "copy Storage settings from Actions" in the Management Console ("GitHub Packages" > "Packages Storage Settings") has been removed. [Updated: 2024-11-20]

MEDIUM: An attacker could steal sensitive information by exploiting a Cross-Site Scripting vulnerability in the repository transfer feature. This exploitation would require social engineering. GitHub has requested CVE ID CVE-2024-8770 for this vulnerability, which was reported via the GitHub Bug Bounty program.

HIGH: A GitHub App installed in organizations could upgrade some permissions from read to write access without approval from an organization administrator. An attacker would require an account with administrator access to install a malicious GitHub App. GitHub has requested CVE ID CVE-2024-8810 for this vulnerability, which was reported via the GitHub Bug Bounty Program. [Updated: 2024-11-07]

On an instance with GitHub Actions enabled, due to an insufficient wait time, MS SQL and MySQL replication could fail with the error message Failed to start nomad service!.

ghe-storage-find was sometimes unable to identify a data disk.

After upgrading the relevant GHES version, the resolvconf service failed to start due to a missing directory.

Some pre-receive hooks using the faccessat2 system call, such as those using Alpine Linux as the base, failed unexpectedly.

When configuring a high availability replica and during the database seeding of a MySQL replica node, restarting the nomad service could time out. Consequently, when MySQL replication attempted to start an error was reported, and setting up replication failed.

On an instance in a cluster configuration, the ghe-cluster-status command returned an error if a soft-deleted repository had a checksum mismatch.

Fixes and improvements for the git core module.

Some repositories could miss spokes information after restoring in a clustering topology due to unrescued exceptions.

In organizations with a large number of repositories, when an administrator used repository properties to target repositories in an organization ruleset, the ruleset index page timed out.

After a user created a Projects Insights chart with time as the X-axis, the chart became hidden and inaccessible.

Fixes a known issue where some links to GitHub Docs from GitHub Enterprise Server may lead to a “Page not found.” Previously, the links incorrectly added enterprise-cloud@latest to the URL.

A bug introduced in 3.12 which prevented the search input in the global navigation from displaying a dropdown of search suggestions has been fixed. The search input functionality prior to 3.12 has been restored, and users are once again able to see and submit suggested search queries, including scope suggestions.

Custom links to other repositories displayed incorrect breadcrumbs.

The Secret Scanning Push Protection custom resource link set at the Enterprise level was not being displayed to users being blocked when pushing secrets to a repository using git through the command line interface.

Following an upgrade, Elasticsearch search migrations are sometimes incorrectly reported as failing in the audit log, even though the migrations completed successfully. [Updated: 2024-09-27]

For instances deployed on Amazon Web Services (AWS), site administrators can configure regional AWS STS endpoints for OIDC from the Management Console.

Site administrators can now configure the instance with NUMA optimizations.

During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see Troubleshooting access to the Management Console.

On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1.

In some situations, large .adoc files stored in a repository do not render properly in the web UI. The raw contents are still available to view as plaintext.

Repositories originally imported using ghe-migrator will not correctly track Advanced Security contributions.

Admin stats REST API endpoints may timeout on appliances with many users or repositories. Retrying the request until data is returned is advised.

When following the steps for Replacing the primary MySQL node, step 14 (running ghe-cluster-config-apply) might fail with errors. If this occurs, re-running ghe-cluster-config-apply is expected to succeed.

Running a config apply as part of the steps for Replacing a node in an emergency may fail with errors if the node being replaced is still reachable. If this occurs, shutdown the node and repeat the steps.

If a hotpatch upgrade requires the haproxy-frontend service to be restarted, the restart will hang if there are existing long-lived connections, such as browser web sockets or Git operations. No new connections will be accepted for up to 5 minutes. Any existing unfinished connections at this time will be disconnected.

When restoring data originally backed up from a 3.13 appliance onto a 3.13 appliance, the elasticsearch indices need to be reindexed before some of the data will show up. This happens via a nightly scheduled job. It can also be forced by running /usr/local/share/enterprise/ghe-es-search-repair.

An organization-level code scanning configuration page is displayed on instances that do not use GitHub Advanced Security or code scanning.

When following the steps for Replacing the primary MySQL node, step 14 (running ghe-cluster-config-apply) might fail with errors. If this occurs, re-running ghe-cluster-config-apply is expected to succeed.

In the header bar displayed to site administrators, some icons are not available.

When enabling automatic update checks for the first time in the Management Console, the status is not dynamically reflected until the "Updates" page is reloaded.

When restoring from a backup snapshot, a large number of mapper_parsing_exception errors may be displayed.

Services may respond with a 503 status due to an out of date haproxy configuration. This can usually be resolved with a ghe-config-apply run.

Images embedded in wiki pages may stop rendering shortly after being published. [Updated: 2024-10-16]

When operating in a high availability configuration, running ghe-repl-promote on a replica node will fail if the original primary cannot be reached by the replica node. This is because the ghe-repl-promote script attempts to decommission all Elasticsearch nodes other than the promoted node, however these requests are made to the original primary node which is no longer reachable. The error message written to the terminal will be similar to:

Maintenance mode has been enabled for active replica <REPLICA_HOSTNAME>
{"message": "No server is currently available to service your request. Sorry about that. Please try resubmitting your request and contact your local GitHub Enterprise site administrator if the problem persists."}
jq: error (at :3): Cannot index string with string "node"

If this occurs, workaround this issue by running the following command — this changes the ghe-repl-promote script in place:

sudo sed -i.bak -e '/for node_hostname in/i if ! $forced; then' -e '/^  done/a fi' /usr/local/bin/ghe-repl-promote

Then re-run the updated ghe-repl-promote script.

[Updated: 2024-11-29]

Instance administration

• On an instance with multiple replica nodes, to start or stop replication for all nodes in a single configuration run, administrators can use the ghe-repl-start-all and ghe-repl-stop-all commands.

Instance services

• Administrators can scale the appliance using generation 2 virtual machines, with support for booting in UEFI mode. This requires deploying a new instance and restoring data onto it. See Using generation 2 virtual machines.

• Nomad has been upgraded to 1.5.17 and Consul has been upgraded to 1.17.4. These services are used in GitHub Enterprise Server to orchestrate containers and configuration.

Identity and access management

• Automated user provisioning via the System for Cross-domain Identity Management (SCIM) standard is available in public beta. Instances that use SAML authentication can enable SCIM to provision user accounts and manage their lifecycle from an identity provider (IdP). You can configure SCIM using an application for supported IdPs, or using the REST API endpoints for SCIM. See Configuring user provisioning with SCIM on GitHub Enterprise Server.

  • If your instance already uses SAML, you will need to configure a new IdP application that supports automated provisioning via SCIM.
  • Existing private beta customers should also reconfigure their implementation with an updated application.
  • During the public beta, we recommend testing SCIM support for your identity system in a non-production instance before adding SCIM to your current setup.

• Organization owners can create and assign custom organization roles, delegating administrative duties to trusted teams and users. See Managing custom organization roles.

• Users can use the account switcher to switch between multiple accounts. See Switching between accounts.

• On an instance that uses built-in authentication, users can use passkeys to sign in securely to GitHub, without needing to input their password. See Authenticating with a passkey.

• Enterprises that use an SSH certificate authority can allow SSH certificates to be used to access user-owned repositories. See Enforcing policies for security settings in your enterprise.

Audit logs

• Every 24 hours, a health check runs for each audit log stream. If a stream is set up incorrectly, an email will be sent to the enterprise owners as notification that their audit log stream is not properly configured.

Secret scanning

• Users can specify which teams or roles have the ability to bypass push protection. This feature is in public beta and subject to change. See About push protection.

• Secret scanning detects secrets leaked in discussions and in pull request titles, bodies, and comments. This feature is in public beta and subject to change. See About secret scanning.

• Secret scanning blocks contributors from uploading files with detected secrets if push protection is enabled for a repository. This feature is in public beta and subject to change.

• Audit log events are created when secret scanning non-provider patterns are enabled or disabled at the repository, organization, or enterprise level.

Code scanning

• Users can create a dedicated code scanning rule to block pull request merges, instead of relying on status checks. This feature is in public beta and subject to change. See Set code scanning merge protection.

• Users can use CodeQL threat model settings for C# to adapt CodeQL's code scanning analysis to detect the most relevant security vulnerabilities in their code. This feature is in public beta and subject to change. See Editing your configuration of default setup.

• Organizations that use default setup for code scanning can use organization-level model packs to extend the coverage of multiple repositories. This feature is in public beta and subject to change. See Editing your configuration of default setup.

• CodeQL can scan Java projects without a build. This feature is in public beta and subject to change.

• This release comes installed with version 2.17.6 of the CodeQL CLI, used in the CodeQL action for code scanning. Significant updates since the default version installed on GitHub Enterprise Server 3.13 include:

  • Support for Java 22, Swift 5.10, TS 5.4, and C# 12
  • New queries for C/C++, Go, Java, and Ruby:
    • cpp/type-confusion: Detects casts to invalid types
    • cpp/iterator-to-expired-container: Detects the creation of iterators owned by temporary objects that are about to be destroyed
    • go/uncontrolled-allocation-size: Detects slice memory allocation with excessive size value
    • java/unvalidated-url-forward: Prevents information disclosure caused by unsafe URL construction
    • rb/insecure-mass-assignment: Detects instances of mass assignment operations accepting arbitrary parameters
    • rb/csrf-protection-not-enabled: Detects cases where Cross-Site Request Forgery protection is not enabled in Ruby on Rails controllers

Dependabot

• Users can consolidate Dependabot pull requests by enabling grouped security updates for related dependencies in a package ecosystem. See About Dependabot security updates.

• Dependabot can access Cargo private registries to provide updates to Rust dependencies. See Guidance for the configuration of private registries for Dependabot.

• Dependabot pauses scheduled jobs after 15 failures. This gives an earlier indication of potential issues while still ensuring that critical security updates continue to be applied without interruption.

• Dependabot uses private registry configurations specified in the dependabot.yml file as expected, even if there is a configuration with target-branch. This ensures that security updates are applied correctly, regardless of your repository's configuration settings. See Configuring access to private registries for Dependabot.

• In the dependabot.yml file, users can apply the same configuration to manifest files from multiple directories using the directories key. Direct strings, glob syntax, and wildcards (*) are all supported for targeting directories. See Dependabot options reference. [Updated: 2024-10-07]

Code security

• The security overview dashboard, with the ability to view secret scanning metrics and trending data for the enablement of security features, is available at the enterprise level. See Viewing security insights.

• The security overview dashboard for organizations is now generally available.

• On the security overview dashboard, users can view alert trends grouped by tool. The group-by option is designed to improve the ability to track and analyze the effectiveness of scanning tools, enabling more strategic decision-making. See Viewing security insights.

• On the security overview dashboard, users can filter by security tool. This feature is in public beta and subject to change.

• In the dependency graph, a software bill of materials (SBOM) generated for a package now includes the package URL for more packages. Previously, the package URL was not included if the manifest file referenced a package with a version range.

GitHub Actions

• For self-hosted GitHub Actions runners on this GitHub Enterprise Server release, the minimum required version of the GitHub Actions Runner application is 2.317.0. See the release notes for this version in the actions/runner repository. If your instance uses ephemeral self-hosted runners and you've disabled automatic updates, you must upgrade your runners to this version of the Runner application before upgrading your instance to this GitHub Enterprise Server release.

• Deployment views across environments are now generally available. Users can pin environments and use additional filters to filter the views. See Viewing deployment history.

GitHub Pages

• Users can configure custom GitHub Actions workflows to build and deploy sites on GitHub Pages. See Configuring a publishing source for your GitHub Pages site.

Repositories

• Users can enhance security by adding deploy keys as a bypass type to rulesets. See Creating rulesets for a repository.

• Users can select Dependabot in the bypass list of a ruleset. See Creating rulesets for a repository.

Projects

• Users can use the auto-close issue workflow to automatically close issues when a project item moves to a specific "completed" status. See Using the built-in automations.

Integrations and extensions

• When authenticating to a native GitHub App or OAuth app, users will be prompted to select which account they want to sign in to using an account picker. Developers of apps can append ?prompt=select_account to their login flow to show users the account picker.

• When using a JSON Web Token (JWT) to authenticate or request an installation token, developers of GitHub Apps can use the app's client ID for the JWT's iss claim. The application ID remains valid, but is considered deprecated.

Complete SCIM payloads are written to the audit log, including SCIM attributes that are not required or supported per API docs. Customers using Okta with SCIM may notice that a placeholder password attribute is among the data passed to audit logs in its current configuration. This placeholder data is associated with Okta’s password synchronization feature that is not expected or required by GitHub. See okta-scim for more information.

Custom firewall rules are removed during the upgrade process.

When enabling automatic update checks for the first time in the Management Console, the status is not dynamically reflected until the "Updates" page is reloaded.

During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see Troubleshooting access to the Management Console.

On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1.

In some situations, large .adoc files stored in a repository do not render properly in the web UI. The raw contents are still available to view as plaintext.

Repositories originally imported using ghe-migrator will not correctly track Advanced Security contributions.

REST API endpoints for admin stats may time out on appliances with many users or repositories. Retrying the request until data is returned is advised.

When following the steps for Replacing the primary MySQL node, step 14 (running ghe-cluster-config-apply) might fail with errors. If this occurs, re-running ghe-cluster-config-apply is expected to succeed.

Running a config apply as part of the steps for Replacing a node in an emergency may fail with errors if the node being replaced is still reachable. If this occurs, shut down the node and repeat the steps.

If a hotpatch upgrade requires the haproxy-frontend service to be restarted, the restart will hang if there are existing long-lived connections, such as browser web sockets or Git operations. No new connections will be accepted for up to 5 minutes. Any existing unfinished connections at this time will be disconnected.

When restoring data originally backed up from a 3.13 appliance, the Elasticsearch indices need to be reindexed before some of the data will show up. This happens via a nightly scheduled job. It can also be forced by running /usr/local/share/enterprise/ghe-es-search-repair.

The global search bar does not have suggestions enabled due to the redesigned navigation and pending new search experience.

Upgrades include an error concerning Error deregistering job for consul-template. This message does not indicate any problems with your install and can be safely ignored.

Some links to GitHub Docs from GitHub Enterprise Server may lead to a "Page not found," because an enterprise-cloud@latest portion is incorrectly added to the URL.

An organization-level code scanning configuration page is displayed on instances that do not use GitHub Advanced Security or code scanning.

In the header bar displayed to site administrators, some icons are not available.

When restoring from a backup snapshot, a large number of mapper_parsing_exception errors may be displayed.

On boot, the resolvconf service may fail to start because the /run/resolvconf directory does not exist when the service attempts to touch a file there, with the error:

/bin/touch: cannot touch '/run/resolvconf/postponed-update': No such file or directory

If this occurs, workaround this issue with the following commands — this change will persist on reboots, but not upgrades:

sudo sed -i.bak \
'/\[Service\]/a ExecStartPre\=\/bin\/mkdir \-p \/run\/resolvconf' \
/etc/systemd/system/resolvconf.service.d/local.conf

sudo systemctl daemon-reload
sudo systemctl start resolvconf 

Services may respond with a 503 status due to an out of date haproxy configuration. This can usually be resolved with a ghe-config-apply run.

When enabling automatic update checks for the first time in the Management Console, the status is not dynamically reflected until the "Updates" page is reloaded.

Following an upgrade, Elasticsearch search migrations are sometimes incorrectly reported as failing in the audit log, even though the migrations completed successfully. [Updated: 2024-09-27]

Images embedded in wiki pages may stop rendering shortly after being published. [Updated: 2024-10-16]

When operating in a high availability configuration, running ghe-repl-promote on a replica node will fail if the original primary cannot be reached by the replica node. This is because the ghe-repl-promote script attempts to decommission all Elasticsearch nodes other than the promoted node, however these requests are made to the original primary node which is no longer reachable. The error message written to the terminal will be similar to:

Maintenance mode has been enabled for active replica <REPLICA_HOSTNAME>
{"message": "No server is currently available to service your request. Sorry about that. Please try resubmitting your request and contact your local GitHub Enterprise site administrator if the problem persists."}
jq: error (at :3): Cannot index string with string "node"

If this occurs, workaround this issue by running the following command — this changes the ghe-repl-promote script in place:

sudo sed -i.bak -e '/for node_hostname in/i if ! $forced; then' -e '/^  done/a fi' /usr/local/bin/ghe-repl-promote

Then re-run the updated ghe-repl-promote script.

[Updated: 2024-11-29]

The Manage GHES API reached feature parity with the Management Console API in GHES 3.12. As a result, we will remove the Management Console API in GitHub Enterprise Server 3.15. For information about updating tooling that relies on the Management Console API, see REST API endpoints for Management Console.

These release notes previously indicated as a known issue that on GitHub Enterprise Server 3.14.0 when log forwarding is enabled, some forwarded log entries may be duplicated. The fix for this problem was already included prior to the release of GitHub Enterprise Server 3.14.0. [Updated: 2024-09-16]

These release notes did not include a note for support of the directories key in dependabot.yml files. [Updated: 2024-10-07]

The "Changes" section indicated that "Pushes that update over 5,000 branches no longer trigger webhooks or GitHub Actions workflows." The change instead affects GitHub Enterprise Server version 3.15. [Updated: 2024-10-30]