Skip to main content

Эта версия GitHub Enterprise Server будет прекращена 2024-06-29. Исправления выпускаться не будут даже при критических проблемах безопасности. Для повышения производительности, повышения безопасности и новых функций выполните обновление до последней версии GitHub Enterprise Server. Чтобы получить справку по обновлению, обратитесь в службу поддержки GitHub Enterprise.

Enterprise Server 3.9 release notes

March 20, 2024

📣 Это не последний выпуск Enterprise Server. Используйте последний выпуск для последних исправлений безопасности, производительности и ошибок.

Warning: A change to MySQL in GitHub Enterprise Server 3.9 and later may impact the performance of your instance. Before you upgrade, make sure you've read the "Known issues" section of these release notes.

3.9.12: Security fixes

  • HIGH: An attacker with an Administrator role in GitHub Enterprise Server could gain SSH root access via remote code execution. GitHub has requested CVE ID CVE-2024-2469 for this vulnerability, which was reported via the GitHub Bug Bounty program.

  • HIGH: An attacker with an editor role in the Management Console could gain SSH access to the instance by command injection when configuring GeoJSON settings. GitHub has requested CVE ID CVE-2024-2443 for this vulnerability, which was reported via the GitHub Bug Bounty program.

  • Packages have been updated to the latest security versions.

3.9.12: Bug fixes

  • In some cases, storage initialization on a new instance launch could cause EBS-backed data volumes to not be detected correctly.

  • Redundant messages caused an increase in the volume of events logged in /var/log/syslog.

  • Administrators could initiate an SSH audit that unknowingly unverified all SSH keys.

  • Attributes used to debug LDAP issues were not included in system logs.

  • On an instance in a cluster configuration with high availability enabled, the ghe-spokesctl command failed when run on a replica node.

  • On an instance with GitHub Actions enabled, GitHub Actions workflows that deployed GitHub Pages sites failed with the following error: Error: Deployment failed, try again later.

  • Organizations using projects (classic) returned an error log about a soon-to-be deprecated MySQL feature when viewing a project.

  • On an instance in a cluster configuration, Jupyter notebooks did not render correctly.

  • In some cases, manual repository maintenance using ghe-spokesctl would fail with the following error: panic: runtime error: invalid memory address or nil pointer dereference.

  • On an instance with a GitHub Advanced Security license, in some cases, when a user deleted a custom pattern for secret scanning, GitHub Enterprise Server failed to close or delete the pattern's alerts.

  • On an instance in a cluster configuration with many nodes, requests to the REST API for managing GitHub Enterprise Server would exceed the instance's HTTP timeouts.

  • In some cases, the codeload service could panic during shutdown and not terminate gracefully.

  • When an administrator set a policy to require two-factor authentication (2FA) for an enterprise, a message incorrectly indicated that users without 2FA enabled on their account would be removed from the enterprise. These users will be removed from repositories and organizations in the enterprise, but not from the enterprise itself.

  • On an instance with a GitHub Advanced Security license, viewing a secret scanning alert as a user without the security manager role would return a 500 error if the alert was generated from a Git tag instead of a normal commit.

  • When using GitHub Enterprise Importer to import repositories, ghost users in archive metadata files would cause an error when generating a list of migration conflicts using ghe-migrator conflicts.

  • Some API endpoints for projects did not properly filter target repositories based on the users access.

  • After an administrator ran ghe-saml-mapping-csv, the output did not include the corresponding SQL query.

  • During a configuration run prompted by the delayed restart of the notebooks service, a container validation warning appeared in system logs.

  • On an instance in a cluster configuration, rebuilds of GitHub Pages sites failed if no replicas of the GitHub Pages data were available (for example, on a newly restored cluster).

  • On an instance with code scanning enabled, upgrades to GitHub Enterprise Server version 3.9 or 3.10 could be slow if a large number of code scanning analyses were present on the instance.

3.9.12: Changes

  • Gists can be deleted using the Purge Gist button on the Deleted Gists page in Staff Tools.

  • People deploying a GitHub Enterprise Server instance in AWS can now deploy in an environment that uses Instance Metadata Service Version 2 (IMDSv2).

  • On an instance in a cluster configuration, MySQL replica nodes can be configured to skip database seeding.

  • The payload for the push webhook event is now limited to 2,048 commits. If there are more than 2,048 commits in a push, the webhook payload for that push will not contain any commits. If you need to fetch commit information, you can use the Commits endpoints of the REST API. For more information, see "События и полезные данные веб-перехватчика" and "Конечные точки REST API для фиксаций."

3.9.12: Known issues

  • Custom firewall rules are removed during the upgrade process.

  • During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

  • If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see "Устранение неполадок с доступом к консоли управления."

  • If an instance is configured to forward logs to a target server with TLS enabled, certificate authority (CA) bundles that a site administrator uploads using ghe-ssl-ca-certificate-install are not respected, and connections to the server fail.

  • When running ghe-config-apply, the process may stall with the message Deployment is running pending automatic promotion.

  • The mbind: Operation not permitted error in the /var/log/mysql/mysql.err file can be ignored. MySQL 8 does not gracefully handle when the CAP_SYS_NICE capability isn't required, and outputs an error instead of a warning.

  • When enabling CodeQL via default setup at scale, some checks related to GitHub Actions are omitted, potentially preventing the process from completing.

  • After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8, to 3.9 or 3.10, MySQL may not start back up. For more information, see "Известные проблемы с обновлением экземпляра."

  • After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8 to 3.9 or 3.10, I/O utilization will increase, and in some cases the instance's performance will be impacted. Reduced performance is due to the database server being upgraded from MySQL 5.7 to MySQL 8.0. For more information, see "Известные проблемы с обновлением экземпляра."

  • In rare circumstances, a small instance with both high availability and GitHub Actions configured may report that MSSQL replication is unhealthy after many upgrades with full upgrade packages. If you encounter this issue, contact GitHub Support.

  • On an instance in a cluster configuration with high availability configured, ghe-config-apply times out while waiting for hookshot-go to start on replica application nodes.

  • On an instance hosted in AWS, system time may lose synchronization with Amazon's servers after an administrator reboots the instance.

  • On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1.

  • In GitHub Enterprise Server 3.10 and later, the requirements for TLS security levels have changed due to an upgrade to containers in the underlying OS. After upgrading from GitHub Enterprise Server 3.9, on an instance with GitHub Actions enabled and a custom TLS certificate, users may experience disruptions with workflow runs if the TLS certificate uses weak encryption. Workflow runs will not trigger, and the following error message will appear in system logs for babeld.

    CA certificate key too weak
    

    To resolve this issue, confirm that your certificate complies with level 2 of the OpenSSL security specification. For more information, see SSL_CTX_set_security_level in the OpenSSL docs. For more information about reviewing your instance's logs, see "Сведения о системных журналах".

    If the error appears in babeld logs because your TLS certificate does not comply with level 2 of the specification, you must create and upload a new certificate with stronger security before you upgrade to GitHub Enterprise Server 3.10 or later. For more information, see "Настройка TLS."

  • On an instance with GitHub Actions enabled, after an upgrade from GitHub Enterprise Server 3.8 or earlier, an internal exception could prevent successful completion of some operations, like upgrades or the configuration of new replica nodes for high availability. If this issue occurs, administrators may see the following error in /data/user/common/ghe-config.log.

    Error occurred while executing servicing step 'Clone datatier login to secondary replica' for component CopyAvailabilityGroupSqlLogins during CopyAvailabilityGroupSqlLogins: Object reference not set to an instance of an object.
    

    To resolve this issue, upgrade to the latest patch release of GitHub Enterprise Server.

  • On an instance in a cluster configuration, restoration of a backup using ghe-restore will exit prematurely if Redis has not restarted properly.

  • The HAProxy version has been updated in this release. You may encounter elevated error rates when HAProxy is upgraded as part of a hotpatch upgrade to a GitHub Enterprise Server instance. These elevated error rates should resolve within 5 minutes of the hotpatch being applied.

    Please note, when performing a hotpatch upgrade to GitHub Enterprise Server version 3.9.7 or higher you will encounter this known issue only if you are hotpatching from GitHub Enterprise Server version 3.9.6 or lower .

  • On an instance with GitHub Actions enabled, Actions workflows that deploy GitHub Pages sites may fail with the following error:

    Error: Deployment failed, try again later.
    

    To fix this issue, connect to any of the instance's nodes using SSH, then run the following commands.

    if [ -d "$CHROOT_PATH/data/pages-untar" ] ; then
      rm -rf "$CHROOT_PATH/data/pages-untar"
    fi
    pages_untar_image_tag="$(cat "$CHROOT_PATH/data/docker-image-tags/pages_untar_image_tag")"
    id="$(docker create "pages-untar:$pages_untar_image_tag")"
    sudo docker cp "$id:/data/pages-untar" "$BASE_PATH/$CHROOT_PATH/data/pages-untar/"
    docker rm "$id"
    

February 29, 2024

📣 Это не последний выпуск исправлений этой серии выпусков, и это не последний выпуск Enterprise Server. Используйте последний выпуск для последних исправлений безопасности, производительности и ошибок.

Warning: A change to MySQL in GitHub Enterprise Server 3.9 and later may impact the performance of your instance. Before you upgrade, make sure you've read the "Known issues" section of these release notes.

3.9.11: Security fixes

  • HIGH: On an instance with GitHub Connect enabled and non-default settings for GitHub Connect configured, an attacker could use an enterprise GitHub Actions download token to fetch private repository data. This token is only accessible to users on the GitHub Enterprise Server instance. To fix this vulnerability, the Actions download token will now be a permissionless token. GitHub has requested CVE ID CVE-2024-1908 for this vulnerability, which was reported via the GitHub Bug Bounty program.

  • Packages have been updated to the latest security versions.

3.9.11: Bug fixes

  • Redundant messages caused increased log volumes in /var/log/syslog.

3.9.11: Known issues

  • Custom firewall rules are removed during the upgrade process.

  • During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

  • If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see "Устранение неполадок с доступом к консоли управления."

  • If an instance is configured to forward logs to a target server with TLS enabled, certificate authority (CA) bundles that a site administrator uploads using ghe-ssl-ca-certificate-install are not respected, and connections to the server fail.

  • When running ghe-config-apply, the process may stall with the message Deployment is running pending automatic promotion.

  • The mbind: Operation not permitted error in the /var/log/mysql/mysql.err file can be ignored. MySQL 8 does not gracefully handle when the CAP_SYS_NICE capability isn't required, and outputs an error instead of a warning.

  • When enabling CodeQL via default setup at scale, some checks related to GitHub Actions are omitted, potentially preventing the process from completing.

  • After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8, to 3.9 or 3.10, MySQL may not start back up. For more information, see "Известные проблемы с обновлением экземпляра."

  • After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8 to 3.9 or 3.10, I/O utilization will increase, and in some cases the instance's performance will be impacted. Reduced performance is due to the database server being upgraded from MySQL 5.7 to MySQL 8.0. For more information, see "Известные проблемы с обновлением экземпляра."

  • In rare circumstances, a small instance with both high availability and GitHub Actions configured may report that MSSQL replication is unhealthy after many upgrades with full upgrade packages. If you encounter this issue, contact GitHub Support.

  • On an instance in a cluster configuration with high availability configured, ghe-config-apply times out while waiting for hookshot-go to start on replica application nodes.

  • On an instance hosted in AWS, system time may lose synchronization with Amazon's servers after an administrator reboots the instance.

  • On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1.

  • In GitHub Enterprise Server 3.10 and later, the requirements for TLS security levels have changed due to an upgrade to containers in the underlying OS. After upgrading from GitHub Enterprise Server 3.9, on an instance with GitHub Actions enabled and a custom TLS certificate, users may experience disruptions with workflow runs if the TLS certificate uses weak encryption. Workflow runs will not trigger, and the following error message will appear in system logs for babeld.

    CA certificate key too weak
    

    To resolve this issue, confirm that your certificate complies with level 2 of the OpenSSL security specification. For more information, see SSL_CTX_set_security_level in the OpenSSL docs. For more information about reviewing your instance's logs, see "Сведения о системных журналах".

    If the error appears in babeld logs because your TLS certificate does not comply with level 2 of the specification, you must create and upload a new certificate with stronger security before you upgrade to GitHub Enterprise Server 3.10 or later. For more information, see "Настройка TLS."

  • On an instance with GitHub Actions enabled, after an upgrade from GitHub Enterprise Server 3.8 or earlier, an internal exception could prevent successful completion of some operations, like upgrades or the configuration of new replica nodes for high availability. If this issue occurs, administrators may see the following error in /data/user/common/ghe-config.log.

    Error occurred while executing servicing step 'Clone datatier login to secondary replica' for component CopyAvailabilityGroupSqlLogins during CopyAvailabilityGroupSqlLogins: Object reference not set to an instance of an object.
    

    To resolve this issue, upgrade to the latest patch release of GitHub Enterprise Server.

  • On an instance in a cluster configuration, restoration of a backup using ghe-restore will exit prematurely if Redis has not restarted properly.

  • The HAProxy version has been updated in this release. You may encounter elevated error rates when HAProxy is upgraded as part of a hotpatch upgrade to a GitHub Enterprise Server instance. These elevated error rates should resolve within 5 minutes of the hotpatch being applied.

    Please note, when performing a hotpatch upgrade to GitHub Enterprise Server version 3.9.7 or higher you will encounter this known issue only if you are hotpatching from GitHub Enterprise Server version 3.9.6 or lower .

  • On an instance with GitHub Actions enabled, Actions workflows that deploy GitHub Pages sites may fail with the following error:

    Error: Deployment failed, try again later.
    

    To fix this issue, connect to any of the instance's nodes using SSH, then run the following commands.

    if [ -d "$CHROOT_PATH/data/pages-untar" ] ; then
      rm -rf "$CHROOT_PATH/data/pages-untar"
    fi
    pages_untar_image_tag="$(cat "$CHROOT_PATH/data/docker-image-tags/pages_untar_image_tag")"
    id="$(docker create "pages-untar:$pages_untar_image_tag")"
    sudo docker cp "$id:/data/pages-untar" "$BASE_PATH/$CHROOT_PATH/data/pages-untar/"
    docker rm "$id"
    

February 13, 2024

📣 Это не последний выпуск исправлений этой серии выпусков, и это не последний выпуск Enterprise Server. Используйте последний выпуск для последних исправлений безопасности, производительности и ошибок.

Warning: A change to MySQL in GitHub Enterprise Server 3.9 and later may impact the performance of your instance. Before you upgrade, make sure you've read the "Known issues" section of these release notes.

3.9.10: Security fixes

  • HIGH: An attacker could gain unauthorized read permission to files by deploying arbitrary symbolic links to a GitHub Pages site with a specially crafted artifact tarball. GitHub has requested CVE ID CVE-2024-1082 for this vulnerability, which was reported via the GitHub Bug Bounty program.

  • HIGH: An attacker with an editor role in the Management Console could gain admin SSH access to the appliance by command injection when configuring SAML settings. GitHub has requested CVE ID CVE-2024-1372 for this vulnerability, which was reported via the GitHub Bug Bounty program.

  • HIGH: An attacker with an editor role in the Management Console could gain admin SSH access to the appliance by command injection when setting an HTTP proxy. GitHub has requested CVE ID CVE-2024-1359 for this vulnerability, which was reported via the GitHub Bug Bounty program.

  • HIGH: An attacker with an editor role in the Management Console could gain admin SSH access to the appliance by command injection into nomad templates when configuring SMTP options. GitHub has requested CVE ID CVE-2024-1378 for this vulnerability, which was reported via the GitHub Bug Bounty program.

  • HIGH: An attacker with an editor role in the Management Console could gain admin SSH access to the appliance by command injection in the actions-console docker container while setting a service URL. GitHub has requested CVE ID CVE-2024-1355 for this vulnerability, which was reported via the GitHub Bug Bounty program.

  • HIGH: An attacker with an editor role in the Management Console could gain admin SSH access to the appliance by command injection in the syslog-ng configuration file. GitHub has requested CVE ID CVE-2024-1354 for this vulnerability, which was reported via the GitHub Bug Bounty program.

  • HIGH: An attacker with an editor role in the Management Console could gain admin SSH access to the appliance by command injection when setting the username and password for collectd configurations. GitHub has requested CVE ID CVE-2024-1369 for this vulnerability, which was reported via the GitHub Bug Bounty program.

  • HIGH: An attacker with an editor role in the Management Console could gain admin SSH access to the appliance by command injection into nomad templates when configuring audit log forwarding. GitHub has requested CVE ID CVE-2024-1374 for this vulnerability, which was reported via the GitHub Bug Bounty program.

  • HIGH: An attacker could create new branches in public repositories, and run arbitrary GitHub Actions workflows with permissions from the GITHUB_TOKEN. GitHub has requested CVE ID CVE-2024-1482 for this vulnerability, which was reported via the GitHub Bug Bounty program.

  • MEDIUM: An attacker could make changes to a user account by taking advantage of a Cross-site Scripting vulnerability in the tag name pattern field in the tag protections UI. Exploitation of this vulnerability required user interaction with malicious javascript on a website along with further social engineering. GitHub has requested CVE ID CVE-2024-1084 for this vulnerability, which was reported via the GitHub Bug Bounty program.

  • LOW: An attacker could decrypt the user section of the enterprise user license list JSON file by using an exposed private key. This vulnerability was reported via the GitHub Bug Bounty program.

  • Packages have been updated to the latest security versions.

3.9.10: Bug fixes

  • On startup, Elasticsearch logged an innocuous JMX MBeans registration error.

  • Hunk headers in C# files did not correctly display changed functions.

  • Pre-receive hook failures were not visible in the administrator audit log. A previous bug fix for this issue was incomplete.

  • When restoring a deleted repository, some metadata associated with the repository, such as packages or project items, did not properly restore.

  • During Git data server maintenance, a process that was ran on unsupported GitHub Enterprise Server topologies created a significant amount of system logs but did not perform any repair work.

3.9.10: Changes

  • The default 30 second webhook delivery HTTP timeout can be configured.

3.9.10: Known issues

  • Custom firewall rules are removed during the upgrade process.

  • During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

  • If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see "Устранение неполадок с доступом к консоли управления."

  • If an instance is configured to forward logs to a target server with TLS enabled, certificate authority (CA) bundles that a site administrator uploads using ghe-ssl-ca-certificate-install are not respected, and connections to the server fail.

  • When running ghe-config-apply, the process may stall with the message Deployment is running pending automatic promotion.

  • The mbind: Operation not permitted error in the /var/log/mysql/mysql.err file can be ignored. MySQL 8 does not gracefully handle when the CAP_SYS_NICE capability isn't required, and outputs an error instead of a warning.

  • When enabling CodeQL via default setup at scale, some checks related to GitHub Actions are omitted, potentially preventing the process from completing.

  • After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8, to 3.9 or 3.10, MySQL may not start back up. For more information, see "Известные проблемы с обновлением экземпляра."

  • After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8 to 3.9 or 3.10, I/O utilization will increase, and in some cases the instance's performance will be impacted. Reduced performance is due to the database server being upgraded from MySQL 5.7 to MySQL 8.0. For more information, see "Известные проблемы с обновлением экземпляра."

  • In rare circumstances, a small instance with both high availability and GitHub Actions configured may report that MSSQL replication is unhealthy after many upgrades with full upgrade packages. If you encounter this issue, contact GitHub Support.

  • On an instance in a cluster configuration with high availability configured, ghe-config-apply times out while waiting for hookshot-go to start on replica application nodes.

  • On an instance hosted in AWS, system time may lose synchronization with Amazon's servers after an administrator reboots the instance.

  • On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1.

  • In GitHub Enterprise Server 3.10 and later, the requirements for TLS security levels have changed due to an upgrade to containers in the underlying OS. After upgrading from GitHub Enterprise Server 3.9, on an instance with GitHub Actions enabled and a custom TLS certificate, users may experience disruptions with workflow runs if the TLS certificate uses weak encryption. Workflow runs will not trigger, and the following error message will appear in system logs for babeld.

    CA certificate key too weak
    

    To resolve this issue, confirm that your certificate complies with level 2 of the OpenSSL security specification. For more information, see SSL_CTX_set_security_level in the OpenSSL docs. For more information about reviewing your instance's logs, see "Сведения о системных журналах".

    If the error appears in babeld logs because your TLS certificate does not comply with level 2 of the specification, you must create and upload a new certificate with stronger security before you upgrade to GitHub Enterprise Server 3.10 or later. For more information, see "Настройка TLS."

  • On an instance with GitHub Actions enabled, after an upgrade from GitHub Enterprise Server 3.8 or earlier, an internal exception could prevent successful completion of some operations, like upgrades or the configuration of new replica nodes for high availability. If this issue occurs, administrators may see the following error in /data/user/common/ghe-config.log.

    Error occurred while executing servicing step 'Clone datatier login to secondary replica' for component CopyAvailabilityGroupSqlLogins during CopyAvailabilityGroupSqlLogins: Object reference not set to an instance of an object.
    

    To resolve this issue, upgrade to the latest patch release of GitHub Enterprise Server.

  • On an instance in a cluster configuration, restoration of a backup using ghe-restore will exit prematurely if Redis has not restarted properly.

  • The HAProxy version has been updated in this release. You may encounter elevated error rates when HAProxy is upgraded as part of a hotpatch upgrade to a GitHub Enterprise Server instance. These elevated error rates should resolve within 5 minutes of the hotpatch being applied.

    Please note, when performing a hotpatch upgrade to GitHub Enterprise Server version 3.9.7 or higher you will encounter this known issue only if you are hotpatching from GitHub Enterprise Server version 3.9.6 or lower .

  • On an instance with GitHub Actions enabled, Actions workflows that deploy GitHub Pages sites may fail with the following error:

    Error: Deployment failed, try again later.
    

    To fix this issue, connect to any of the instance's nodes using SSH, then run the following commands.

    if [ -d "$CHROOT_PATH/data/pages-untar" ] ; then
      rm -rf "$CHROOT_PATH/data/pages-untar"
    fi
    pages_untar_image_tag="$(cat "$CHROOT_PATH/data/docker-image-tags/pages_untar_image_tag")"
    id="$(docker create "pages-untar:$pages_untar_image_tag")"
    sudo docker cp "$id:/data/pages-untar" "$BASE_PATH/$CHROOT_PATH/data/pages-untar/"
    docker rm "$id"
    ``` [Updated: 2024-03-07]
    
  • Redundant messages may cause an increase in the volume of events logged in /var/log/syslog. [Updated: 2024-03-08]

January 30, 2024

📣 Это не последний выпуск исправлений этой серии выпусков, и это не последний выпуск Enterprise Server. Используйте последний выпуск для последних исправлений безопасности, производительности и ошибок.

Warning: A change to MySQL in GitHub Enterprise Server 3.9 and later may impact the performance of your instance. Before you upgrade, make sure you've read the "Known issues" section of these release notes.

3.9.9: Bug fixes

  • The instance incorrectly wrote the output for multiple workloads to /var/log/syslog.log.

  • During periods of high traffic, interruptions in service occurred due to insufficient resource allocations for internal components.

  • When starting up an instance using NVME storage in a cloud other than AWS, the attached data disk was not properly detected.

3.9.9: Known issues

  • Custom firewall rules are removed during the upgrade process.

  • During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

  • If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see "Устранение неполадок с доступом к консоли управления."

  • If an instance is configured to forward logs to a target server with TLS enabled, certificate authority (CA) bundles that a site administrator uploads using ghe-ssl-ca-certificate-install are not respected, and connections to the server fail.

  • When running ghe-config-apply, the process may stall with the message Deployment is running pending automatic promotion.

  • The mbind: Operation not permitted error in the /var/log/mysql/mysql.err file can be ignored. MySQL 8 does not gracefully handle when the CAP_SYS_NICE capability isn't required, and outputs an error instead of a warning.

  • When enabling CodeQL via default setup at scale, some checks related to GitHub Actions are omitted, potentially preventing the process from completing.

  • After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8, to 3.9 or 3.10, MySQL may not start back up. For more information, see "Известные проблемы с обновлением экземпляра."

  • After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8 to 3.9 or 3.10, I/O utilization will increase, and in some cases the instance's performance will be impacted. Reduced performance is due to the database server being upgraded from MySQL 5.7 to MySQL 8.0. For more information, see "Известные проблемы с обновлением экземпляра."

  • In rare circumstances, a small instance with both high availability and GitHub Actions configured may report that MSSQL replication is unhealthy after many upgrades with full upgrade packages. If you encounter this issue, contact GitHub Support.

  • On an instance in a cluster configuration with high availability configured, ghe-config-apply times out while waiting for hookshot-go to start on replica application nodes.

  • On an instance hosted in AWS, system time may lose synchronization with Amazon's servers after an administrator reboots the instance.

  • On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1.

  • In GitHub Enterprise Server 3.10 and later, the requirements for TLS security levels have changed due to an upgrade to containers in the underlying OS. After upgrading from GitHub Enterprise Server 3.9, on an instance with GitHub Actions enabled and a custom TLS certificate, users may experience disruptions with workflow runs if the TLS certificate uses weak encryption. Workflow runs will not trigger, and the following error message will appear in system logs for babeld.

    CA certificate key too weak
    

    To resolve this issue, confirm that your certificate complies with level 2 of the OpenSSL security specification. For more information, see SSL_CTX_set_security_level in the OpenSSL docs. For more information about reviewing your instance's logs, see "Сведения о системных журналах".

    If the error appears in babeld logs because your TLS certificate does not comply with level 2 of the specification, you must create and upload a new certificate with stronger security before you upgrade to GitHub Enterprise Server 3.10 or later. For more information, see "Настройка TLS."

  • On an instance with GitHub Actions enabled, after an upgrade from GitHub Enterprise Server 3.8 or earlier, an internal exception could prevent successful completion of some operations, like upgrades or the configuration of new replica nodes for high availability. If this issue occurs, administrators may see the following error in /data/user/common/ghe-config.log.

    Error occurred while executing servicing step 'Clone datatier login to secondary replica' for component CopyAvailabilityGroupSqlLogins during CopyAvailabilityGroupSqlLogins: Object reference not set to an instance of an object.
    

    To resolve this issue, upgrade to the latest patch release of GitHub Enterprise Server.

  • Restoring backups with ghe-restore on a GHES cluster will exit prematurely if redis has not restarted properly.

  • The HAProxy version has been updated in this release. You may encounter elevated error rates when HAProxy is upgraded as part of a hotpatch upgrade to a GitHub Enterprise Server instance. These elevated error rates should resolve within 5 minutes of the hotpatch being applied.

    Please note, when performing a hotpatch upgrade to GitHub Enterprise Server version 3.9.7 or higher you will encounter this known issue only if you are hotpatching from GitHub Enterprise Server version 3.9.6 or lower .

  • On an instance with GitHub Actions enabled, Actions workflows that deploy GitHub Pages sites may fail with the following error:

    Error: Deployment failed, try again later.
    

    To fix this issue, connect to any of the instance's nodes using SSH, then run the following commands.

    if [ -d "$CHROOT_PATH/data/pages-untar" ] ; then
      rm -rf "$CHROOT_PATH/data/pages-untar"
    fi
    pages_untar_image_tag="$(cat "$CHROOT_PATH/data/docker-image-tags/pages_untar_image_tag")"
    id="$(docker create "pages-untar:$pages_untar_image_tag")"
    sudo docker cp "$id:/data/pages-untar" "$BASE_PATH/$CHROOT_PATH/data/pages-untar/"
    docker rm "$id"
    ``` [Updated: 2024-03-07]
    
  • Redundant messages may cause an increase in the volume of events logged in /var/log/syslog. [Updated: 2024-03-08]

January 16, 2024

📣 Это не последний выпуск исправлений этой серии выпусков, и это не последний выпуск Enterprise Server. Используйте последний выпуск для последних исправлений безопасности, производительности и ошибок.

Warning: A change to MySQL in GitHub Enterprise Server 3.9 and later may impact the performance of your instance. Before you upgrade, make sure you've read the "Known issues" section of these release notes.

3.9.8: Security fixes

  • HIGH: An attacker with access to a Management Console user account with the editor role could escalate privileges through a command injection vulnerability in the Management Console. GitHub has requested CVE ID CVE-2024-0507 for this vulnerability, which was reported via the GitHub Bug Bounty program.

  • HIGH: An attacker could leverage an unsafe reflection vulnerability in GitHub Enterprise Server (GHES) that could lead to reflection injection. This vulnerability could lead to the execution of user-controlled methods and remote code execution. To exploit this bug, an actor would need to be logged into an account on the GHES instance with the organization owner role. GitHub has requested CVE ID CVE-2024-0200 for this vulnerability, which was reported via the GitHub Bug Bounty program.

  • Packages have been updated to the latest security versions.

3.9.8: Bug fixes

  • Support for authenticating to GitHub Enterprise Server using GitHub CLI OAuth App with a device code was unintentionally disabled.

  • During periods of high load, users would see intermittent interruptions to services when upstream services failed internal health checks.

  • On an instance with GitHub Actions enabled, some maintenance tasks could fail due to incomplete upgrade steps during previous upgrades to new releases of GitHub Enterprise Server.

  • Deleting a repository would enqueue unnecessary background jobs that would never complete.

  • When creating a new custom pattern for secret scanning, the "More options" section of the custom pattern form automatically collapsed when a user entered an invalid regex in the post processing expressions (before/after secret match or additional secret requirements).

  • On an instance with a GitHub Advanced Security license and secret scanning enabled, users could experience a 500 error when viewing a secret scanning alert page in cases where the alerted commits belonged to the user and one or more commits could not be found.

  • Members of an enterprise were incorrectly allowed access to the REST API endpoints for Enterprise licensing.

  • On an instance that uses SAML for authentication, an upgrade from GitHub Enterprise Server 3.7 to 3.9 could result in user login failures due to an outdated gem dependency.

3.9.8: Changes

  • To avoid leaking secrets, the logging of all parameters is disabled for Management Console events in enterprise audit logs.

  • More detailed information is logged when a GitHub Enterprise Server upgrade failed due to missing database encryption keys.

  • The branch protection setting to require PR approval of the most recent reviewable push is included in exports from ghe-migrator or the Organization Migrations API.

3.9.8: Known issues

  • Custom firewall rules are removed during the upgrade process.

  • During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

  • If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see "Устранение неполадок с доступом к консоли управления."

  • If an instance is configured to forward logs to a target server with TLS enabled, certificate authority (CA) bundles that a site administrator uploads using ghe-ssl-ca-certificate-install are not respected, and connections to the server fail.

  • When running ghe-config-apply, the process may stall with the message Deployment is running pending automatic promotion.

  • The mbind: Operation not permitted error in the /var/log/mysql/mysql.err file can be ignored. MySQL 8 does not gracefully handle when the CAP_SYS_NICE capability isn't required, and outputs an error instead of a warning.

  • When enabling CodeQL via default setup at scale, some checks related to GitHub Actions are omitted, potentially preventing the process from completing.

  • After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8, to 3.9 or 3.10, MySQL may not start back up. For more information, see "Известные проблемы с обновлением экземпляра."

  • After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8 to 3.9 or 3.10, I/O utilization will increase, and in some cases the instance's performance will be impacted. Reduced performance is due to the database server being upgraded from MySQL 5.7 to MySQL 8.0. For more information, see "Известные проблемы с обновлением экземпляра."

  • In rare circumstances, a small instance with both high availability and GitHub Actions configured may report that MSSQL replication is unhealthy after many upgrades with full upgrade packages. If you encounter this issue, contact GitHub Support.

  • On an instance in a cluster configuration with high availability configured, ghe-config-apply times out while waiting for hookshot-go to start on replica application nodes.

  • On an instance hosted in AWS, system time may lose synchronization with Amazon's servers after an administrator reboots the instance.

  • On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1.

  • In GitHub Enterprise Server 3.10 and later, the requirements for TLS security levels have changed due to an upgrade to containers in the underlying OS. After upgrading from GitHub Enterprise Server 3.9, on an instance with GitHub Actions enabled and a custom TLS certificate, users may experience disruptions with workflow runs if the TLS certificate uses weak encryption. Workflow runs will not trigger, and the following error message will appear in system logs for babeld.

    CA certificate key too weak
    

    To resolve this issue, confirm that your certificate complies with level 2 of the OpenSSL security specification. For more information, see SSL_CTX_set_security_level in the OpenSSL docs. For more information about reviewing your instance's logs, see "Сведения о системных журналах".

    If the error appears in babeld logs because your TLS certificate does not comply with level 2 of the specification, you must create and upload a new certificate with stronger security before you upgrade to GitHub Enterprise Server 3.10 or later. For more information, see "Настройка TLS."

  • On an instance with GitHub Actions enabled, after an upgrade from GitHub Enterprise Server 3.8 or earlier, an internal exception could prevent successful completion of some operations, like upgrades or the configuration of new replica nodes for high availability. If this issue occurs, administrators may see the following error in /data/user/common/ghe-config.log.

    Error occurred while executing servicing step 'Clone datatier login to secondary replica' for component CopyAvailabilityGroupSqlLogins during CopyAvailabilityGroupSqlLogins: Object reference not set to an instance of an object.
    

    To resolve this issue, upgrade to the latest patch release of GitHub Enterprise Server.

  • Restoring backups with ghe-restore on a GHES cluster will exit prematurely if redis has not restarted properly.

  • The HAProxy version has been updated in this release. You may encounter elevated error rates when HAProxy is upgraded as part of a hotpatch upgrade to a GitHub Enterprise Server instance. These elevated error rates should resolve within 5 minutes of the hotpatch being applied.

    Please note, when performing a hotpatch upgrade to GitHub Enterprise Server version 3.9.7 or higher you will encounter this known issue only if you are hotpatching from GitHub Enterprise Server version 3.9.6 or lower .

  • During periods of high traffic, interruptions in service may occur due to insufficient resource allocations for internal components. [Updated 2024-01-23]

  • The settings for enabling scheduled reminders were added unintentionally to this release. Scheduled reminders are not officially supported. [Updated: 2024-02-22]

  • Redundant messages may cause an increase in the volume of events logged in /var/log/syslog. [Updated: 2024-03-08]

3.9.8: Errata

  • These release notes previously indicated that GitHub Enterprise Server 3.9.8 contained fixes for the following issues:

    • An improper authentication vulnerability that affected private mode, CVE-2023-6847
    • An incorrect authorization vulnerability that affected issue comments, CVE-2023-51380

    These fixes were included in GitHub Enterprise Server 3.9.7.

December 21, 2023

📣 Это не последний выпуск исправлений этой серии выпусков, и это не последний выпуск Enterprise Server. Используйте последний выпуск для последних исправлений безопасности, производительности и ошибок.

Warning: A change to MySQL in GitHub Enterprise Server 3.9 and later may impact the performance of your instance. Before you upgrade, make sure you've read the "Known issues" section of these release notes.

3.9.7: Security fixes

  • HIGH: An improper authentication vulnerability was identified in GitHub Enterprise Server that allowed a bypass of private mode by using a specially crafted API request. Private mode is the mechanism that enforces authentication for publicly-scoped resources. For more information, see "Включение частного режима."

    This vulnerability would allow unauthenticated attackers to gain access to various types of resources set as public on the instance. To exploit this vulnerability, an attacker would need network access to the GitHub Enterprise Server instance configured in private mode. This vulnerability was reported via the GitHub Bug Bounty program and assigned CVE-2023-6847.

  • HIGH: An attacker with access to a Management Console user account with the editor role could escalate privileges by making requests to the endpoint used for bootstrapping the instance, and then reset the root site administrator password. GitHub has requested CVE ID CVE-2023-46647 for this vulnerability, which was reported via the GitHub Bug Bounty program.

  • HIGH: A path traversal vulnerability was identified in GitHub Enterprise Server that allowed arbitrary file reading when building a GitHub Pages site. To exploit this vulnerability, an attacker would need permission to create and build a GitHub Pages site on the GitHub Enterprise Server instance. This vulnerability was reported via the GitHub Bug Bounty program and assigned CVE-2023-46645.

  • MEDIUM: An insertion of sensitive information into log file vulnerability was identified in the log files for a GitHub Enterprise Server backend service that could permit an adversary in the middle attack when combined with other phishing techniques. To exploit this, an attacker would need access to the log files for the GitHub Enterprise Server instance, a backup archive created with GitHub Enterprise Server Backup Utilities, or a service which received streamed logs. GitHub has requested CVE ID CVE-2023-6746 for this vulnerability.

  • MEDIUM: Due to an insufficient entropy vulnerability, an attacker could brute force a user invitation to the Management Console. To exploit this vulnerability, an attacker would have needed knowledge that a user invitation was pending. This vulnerability was reported via the GitHub Bug Bounty program and assigned CVE-2023-46648.

  • MEDIUM: An attacker could maintain admin access via a race condition when an organization was converted from a user. GitHub has requested CVE ID CVE-2023-46649 for this vulnerability, which was reported via the GitHub Bug Bounty program.

  • MEDIUM: Due to an improper access control, an attacker could view private repository names by enumerating check run IDs with the "Get a check run" API endpoint. This vulnerability did not allow unauthorized access to any repository content other than the name. GitHub has requested CVE ID CVE-2023-46646 for this vulnerability, which was reported via the GitHub Bug Bounty program.

  • MEDIUM: An attacker could maintain admin access to a transferred repository in a race condition by making a GraphQL mutation to alter repository permissions during the transfer. GitHub has requested CVE ID CVE-2023-6690 for this vulnerability, which reported via the GitHub Bug Bounty program.

  • MEDIUM: An insertion of sensitive information into log file in the audit log in GitHub Enterprise Server was identified that that could allow an attacker to gain access to the Management Console. To exploit this, an attacker would need access to the log files for the GitHub Enterprise Server instance, a backup archive created with GitHub Enterprise Server Backup Utilities, or a service which received streamed logs. GitHub has requested CVE ID CVE-2023-6802 for this vulnerability.

  • MEDIUM: A race condition in GitHub Enterprise Server allowed an outside collaborator to be added while a repository is being transferred. GitHub has requested CVE ID CVE-2023-6803 for this vulnerability, which was reported via the GitHub Bug Bounty program.

  • MEDIUM: Improper privilege management allowed arbitrary workflows to be committed and run using an improperly scoped PAT. To exploit this, a workflow must have already existed in the target repo. GitHub has requested CVE ID CVE-2023-6804 for this vulnerability, which was reported via the GitHub Bug Bounty program.

  • MEDIUM: An incorrect authorization vulnerability was identified that allowed issue comments to be updated with an improperly scoped token. This vulnerability did not allow unauthorized access to any repository content as it also required contents.write and issues.read permissions. This vulnerability was reported via the GitHub Bug Bounty Program and has been assigned CVE-2023-51379.

  • MEDIUM: An incorrect authorization vulnerability was identified that allowed issue comments to be read with an improperly scoped token. This vulnerability was reported via the GitHub Bug Bounty Program and has been assigned CVE-2023-51380.

  • LOW: Pre-receive hooks have been further hardened against shell command injections.

  • LOW: To render interactive maps in an instance's web UI using Azure Maps, GitHub Enterprise Server has migrated from use of an unsecure Azure Maps API token to a more secure access token provided by role-based access control (RBAC) in Entra ID. After upgrading to this release, to re-enable interactive maps, an administrator must reconfigure authentication to Azure Maps in the Management Console. For more information, see "Настройка интерактивных карт."

  • To address scenarios that could lead to denial of service, HAProxy has been upgraded to version 2.8.4.

  • Packages have been updated to the latest security versions.

3.9.7: Bug fixes

  • In rare cases, on an instance with GitHub Actions enabled, a failed check on a deleted repository could cause upgrades to a new version of GitHub Enterprise Server to fail.

  • Threads in the Git proxy service babeld could crash while reading Git packet lines.

  • When an administrator ran the ghe-support-bundle or ghe-cluster-support-bundle command, the -p flag did not produce bundles with log durations as specified. The duration period can now only be specified in days. Additionally, unnecessary files were sanitized by the commands.

  • On an instance in a cluster configuration, site administrators using the ghe-config-apply utility may have seen the extraneous message "Error: Server closed the connection" in the logs for the utility.

  • Some OAuth applications did not have device code flow (DCF) explicitly enabled, which prevented DCF from running correctly.

  • On an instance in a cluster configuration, upgrades could fail due to a background job running during database migration.

  • On an instance with a GitHub Advanced Security license and secret scanning enabled, site administrators using the ghe-secret-scanning command would not see a relevant error message if their input was invalid.

  • On an instance with GitHub Actions enabled, some maintenance tasks could fail due to incomplete upgrade steps during previous upgrades to new releases of GitHub Enterprise Server.

  • On an instance in a high availability configuration, the ghe-repl-teardown command failed when provided with a UUID.

  • Support for authenticating to GitHub Enterprise Server from Visual Studio Code with a device code was unintentionally disabled.

  • In some environments, stale .backup log files could accumulate in the system.

  • On an instance hosted on AWS, when configuring GitHub Packages, virtual-hosted-style AWS S3 URLs would default to path-style URLs if a region-code was included. For more information, see Virtual hosting of buckets in the AWS documentation.

  • In some cases, when an administrator uploaded a custom TLS certificate, the certificate was not correctly installed on the instance.

  • Because the | character was not permitted, administrators could not add an SMTP username to authenticate with the Azure Communication Service.

  • On an instance with a GitHub Advanced Security license, users with the security manager role could not update custom links for push protection using the REST API.

  • On an instance with the dependency graph enabled, some security products were not automatically enabled for new public repositories.

  • Deprecated resource_activity jobs were not processed and accumulated over time in the queue, causing possible memory issues.

  • Pull request review threads at the file level, rather than the individual line level, were not included in exports from ghe-migrator or the Organization Migrations API.

  • After importing a migration archive using ghe-migrator or REST API endpoints for organization migrations, in some cases, some review comments within pull requests were not associated with lines of code.

  • On an instance with a GitHub Advanced Security license and secret scanning enabled, secret scanning alert emails were sent to organization owners even if their email address did not comply with domain restrictions.

  • After a user started a repository transfer, if another user viewed the repository before the transfer finished, the repository overview rendered incorrectly.

  • On an instance with GitHub Connect and unified search enabled, users trying to view the unified search code results would get a 500 error.

  • On an instance with GitHub Actions enabled, users occasionally got a 500 error when viewing a job with a pending deployment.

  • On an instance with GitHub Actions enabled, an issue with GH_TOKEN sometimes prevented GitHub Pages sites from building successfully in workflows.

  • An administrator could enable GitHub Connect on an instance with a license that does not support GitHub Connect.

  • On an instance with GitHub Connect enabled, some system users were incorrectly counted as consuming a license following license sync.

  • The enterprise account pages on some installations rendered very slowly.

  • A user in the process of being converted into an organization could be added as a collaborator on a repository. This resulted in the new organizations owners unexpectedly receiving access to the repository.

  • When using ghe-migrator to import repositories into GitHub Enterprise Server, the conflicts and audit subcommands produced an invalid CSV file due to an extra log line appended to the file.

  • On an instance with subdomain isolation disabled, a notebook could not be loaded due to incorrect asset paths.

  • Running ghe-spokesctl gov info without any arguments caused a panic response.

  • On an instance with a GitHub Advanced Security license and secret scanning enabled, dry runs sometimes incorrectly reported no results for custom patterns.

  • On an instance with a GitHub Advanced Security license and secret scanning enabled, webhooks for alert locations did not contain information about push protection bypasses.

  • On an instance with a GitHub Advanced Security license, code scanning would report an incorrect number of files scanned on the "Tools" status page.

3.9.7: Changes

  • On an instance with Dependabot updates enabled, Dependabot relies on the node installation provided by the actions runner instead of dynamically downloading.

  • When adding a node to an instance, performance is improved during initial database replication.

  • An administrator can run the new ghe-check-background-upgrade-jobs command to ensure all upgrade jobs that run in the background have finished. This allows the administrator to know when they can start the next upgrade to their GitHub Enterprise Server instance.

  • To avoid negative effects on disk utilization, babeld log files have a maximum size of 15 GB.

  • Instance administrators can manage search indices for GitHub Discussions from the site admin dashboard.

  • To improve reliability of release uploads in low-bandwidth environments, the time-to-live (TTL) value of the token for uploading release assets has increased from 1 hour to 3 hours.

  • When using ghe-migrator prepare to import an archive, a missing schema.json file results in an UnsupportedArchive error rather than an UnsupportedSchemaVersion error.

  • The audit log now tracks all failed password attempts individually. Previously, duplicate failed password attempts in sequence within the same day would be grouped into one failed password attempt, with a count field.

3.9.7: Known issues

  • Custom firewall rules are removed during the upgrade process.

  • The GitHub Packages npm registry no longer returns a time value in metadata responses. This was done to allow for substantial performance improvements. We continue to have all the data necessary to return a time value as part of the metadata response and will resume returning this value in the future once we have solved the existing performance issues.

  • During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

  • If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see "Устранение неполадок с доступом к консоли управления."

  • If an instance is configured to forward logs to a target server with TLS enabled, certificate authority (CA) bundles that a site administrator uploads using ghe-ssl-ca-certificate-install are not respected, and connections to the server fail.

  • When running ghe-config-apply, the process may stall with the message Deployment is running pending automatic promotion.

  • The mbind: Operation not permitted error in the /var/log/mysql/mysql.err file can be ignored. MySQL 8 does not gracefully handle when the CAP_SYS_NICE capability isn't required, and outputs an error instead of a warning.

  • When enabling CodeQL via default setup at scale, some checks related to GitHub Actions are omitted, potentially preventing the process from completing.

  • After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8, to 3.9 or 3.10, MySQL may not start back up. For more information, see "Известные проблемы с обновлением экземпляра."

  • After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8 to 3.9 or 3.10, I/O utilization will increase, and in some cases the instance's performance will be impacted. Reduced performance is due to the database server being upgraded from MySQL 5.7 to MySQL 8.0. For more information, see "Известные проблемы с обновлением экземпляра."

  • In rare circumstances, a small instance with both high availability and GitHub Actions configured may report that MSSQL replication is unhealthy after many upgrades with full upgrade packages. If you encounter this issue, contact GitHub Support.

  • On an instance in a cluster configuration with high availability configured, ghe-config-apply times out while waiting for hookshot-go to start on replica application nodes.

  • On an instance hosted in AWS, system time may lose synchronization with Amazon's servers after an administrator reboots the instance.

  • On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1.

  • In GitHub Enterprise Server 3.10 and later, the requirements for TLS security levels have changed due to an upgrade to containers in the underlying OS. After upgrading from GitHub Enterprise Server 3.9, on an instance with GitHub Actions enabled and a custom TLS certificate, users may experience disruptions with workflow runs if the TLS certificate uses weak encryption. Workflow runs will not trigger, and the following error message will appear in system logs for babeld.

    CA certificate key too weak
    

    To resolve this issue, confirm that your certificate complies with level 2 of the OpenSSL security specification. For more information, see SSL_CTX_set_security_level in the OpenSSL docs. For more information about reviewing your instance's logs, see "Сведения о системных журналах".

    If the error appears in babeld logs because your TLS certificate does not comply with level 2 of the specification, you must create and upload a new certificate with stronger security before you upgrade to GitHub Enterprise Server 3.10 or later. For more information, see "Настройка TLS."

  • On an instance with GitHub Actions enabled, after an upgrade from GitHub Enterprise Server 3.8 or earlier, an internal exception could prevent successful completion of some operations, like upgrades or the configuration of new replica nodes for high availability. If this issue occurs, administrators may see the following error in /data/user/common/ghe-config.log.

    Error occurred while executing servicing step 'Clone datatier login to secondary replica' for component CopyAvailabilityGroupSqlLogins during CopyAvailabilityGroupSqlLogins: Object reference not set to an instance of an object.
    

    To resolve this issue, upgrade to the latest patch release of GitHub Enterprise Server.

  • Restoring backups with ghe-restore on a GHES cluster will exit prematurely if redis has not restarted properly.

  • The HAProxy version has been updated in this release. You may encounter elevated error rates when HAProxy is upgraded as part of a hotpatch upgrade to a GitHub Enterprise Server instance. These elevated error rates should resolve within 5 minutes of the hotpatch being applied.

    Please note, when performing a hotpatch upgrade to GitHub Enterprise Server version 3.9.7 or higher you will encounter this known issue only if you are hotpatching from GitHub Enterprise Server version 3.9.6 or lower . [Updated 2024-01-03]

  • During periods of high traffic, interruptions in service may occur due to insufficient resource allocations for internal components. [Updated 2024-01-23]

  • The settings for enabling scheduled reminders were added unintentionally to this release. Scheduled reminders are not officially supported. [Updated: 2024-02-22]

  • Redundant messages may cause an increase in the volume of events logged in /var/log/syslog. [Updated: 2024-03-08]

3.9.7: Deprecations

  • Interactive maps in the web UI no longer allow authentication using an Azure Maps API key

    • To allow users to render interactive maps in an instance's web UI by writing GeoJSON or TopoJSON syntax, GitHub Enterprise Server previously required a potentially unsecure API key for authentication with Azure Maps. If an administrator previously enabled interactive maps on an instance, the feature is disabled upon upgrade to this release.

      To re-enable interactive maps for your instance, you must configure an application on an Entra ID tenant that has access to Azure Maps using role-based access control (RBAC). For more information, see "Настройка интерактивных карт" and the security fixes for this release.

October 24, 2023

📣 Это не последний выпуск исправлений этой серии выпусков, и это не последний выпуск Enterprise Server. Используйте последний выпуск для последних исправлений безопасности, производительности и ошибок.

Warning: A change to MySQL in GitHub Enterprise Server 3.9 and later may impact the performance of your instance. Before you upgrade, make sure you've read the "Known issues" section of these release notes.

3.9.6: Security fixes

  • LOW: Due to an incorrect permission assignment for some configuration files, an attacker with access to a local operating system user account could read MySQL connection details including the MySQL password. [Updated: 2023-11-13]

  • Packages have been updated to the latest security versions.

3.9.6: Bug fixes

  • The ghe-cluster-repl-status command did not display all replication statuses.

  • On an instance in a cluster configuration with high availability enabled, ghe-config-apply timed out while waiting for hookshot-go to start on replica application nodes.

  • SpokesRepairRepoReplicaJob and SpokesSyncCacheReplicaJob jobs failed, causing cache server replicas to not update and potentially prolonging replication issues.

  • /var/log/lastlog was not copied over as a sparse file during ghe-upgrade, which could cause issues by using additional disk space.

  • On an instance in a cluster configuration, when managing maintenance mode using ghe-cluster-maintenance, an erroneous warning appeared that read "Warning: Maintenance mode set on primary, please make sure to set it on any active replica if needed". - | ghe-repl-status did not identify Git replicas in certain incomplete states and incorrectly suggested that a failover could be performed safely. In some cases, this led to data loss during failover.

  • Repository exports using ghe-migrator or the REST API's operation for organization migrations could fail when a large number of commit comments or long commit comments were present.

  • On an instance with a GitHub Advanced Security license and secret scanning enabled, secret scanning suggested incorrect filters when viewing both open and closed alerts.

  • On instances using the private beta of SCIM provisioning, some users were presented with a "single sign-in" hover card.

  • On an instance with multiple nodes, ghe-spokes status did not identify Git replicas in certain incomplete states, causing a false report that replication was in sync and leading to data loss or replication issues during failover.

  • On an instance with GitHub Actions enabled, administrators received a 500 error after attempting to force cancel a workflow run via Staff Tools.

  • On an instance with a GitHub Advanced Security license, repositories within organizations created using the + dropdown menu did not have GitHub Advanced Security features enabled automatically, even if the features should have been enabled.

  • As a security measure, GitHub Pages does not build sites that contain symbolic links except when using custom GitHub Actions workflows. This change strengthens GitHub Pages's symbolic link detection.

  • On an instance with a GitHub Advanced Security license and secret scanning enabled, dry runs sometimes incorrectly reported no results for custom patterns.

3.9.6: Changes

  • Instructions in the "Migrations" section of the Management Console clarify that only standard AWS S3 endpoints are supported when configuring AWS S3 as a blob storage provider for migrations.

  • When running async repository repairs, the output message about scheduling a repair job is more accurate.

  • On an instance in a cluster configuration, administrators can identify the repository networks or gists that are common across a specified set of storage nodes using the spokesctl find-on-replicas command.

3.9.6: Known issues

  • In GitHub Enterprise Server 3.10 and later, the requirements for TLS security levels have changed due to an upgrade to containers in the underlying OS. After upgrading from GitHub Enterprise Server 3.9, on an instance with GitHub Actions enabled and a custom TLS certificate, users may experience disruptions with workflow runs if the TLS certificate uses weak encryption. Workflow runs will not trigger, and the following error message will appear in system logs for babeld.

    CA certificate key too weak
    

    To resolve this issue, confirm that your certificate complies with level 2 of the OpenSSL security specification. For more information, see SSL_CTX_set_security_level in the OpenSSL docs. For more information about reviewing your instance's logs, see "Сведения о системных журналах".

    If the error appears in babeld logs because your TLS certificate does not comply with level 2 of the specification, you must create and upload a new certificate with stronger security before you upgrade to GitHub Enterprise Server 3.10 or later. For more information, see "Настройка TLS." [Updated: 2023-10-26]

  • Custom firewall rules are removed during the upgrade process.

  • During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

  • If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see "Устранение неполадок с доступом к консоли управления."

  • On an instance in a high-availability configuration, passive replica nodes accept Git client requests and forward the requests to the primary node.

  • If an instance is configured to forward logs to a target server with TLS enabled, certificate authority (CA) bundles that a site administrator uploads using ghe-ssl-ca-certificate-install are not respected, and connections to the server fail.

  • When running ghe-config-apply, the process may stall with the message Deployment is running pending automatic promotion.

  • The mbind: Operation not permitted error in the /var/log/mysql/mysql.err file can be ignored. MySQL 8 does not gracefully handle when the CAP_SYS_NICE capability isn't required, and outputs an error instead of a warning.

  • On an instance with subdomain isolation disabled, Mermaid diagrams in the web UI display an "Unable to render rich display" error and fail to render.

  • When enabling CodeQL via default setup at scale, some checks related to GitHub Actions are omitted, potentially preventing the process from completing.

  • After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8, to 3.9 or 3.10, MySQL may not start back up. For more information, see "Известные проблемы с обновлением экземпляра." [Updated: 2023-08-11]

  • After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8 to 3.9 or 3.10, I/O utilization will increase, and in some cases the instance's performance will be impacted. Reduced performance is due to the database server being upgraded from MySQL 5.7 to MySQL 8.0. For more information, see "Известные проблемы с обновлением экземпляра."

  • In rare circumstances, a small instance with both high availability and GitHub Actions configured may report that MSSQL replication is unhealthy after many upgrades with full upgrade packages. If you encounter this issue, contact GitHub Support.

  • On an instance in a cluster configuration with high availability configured, ghe-config-apply times out while waiting for hookshot-go to start on replica application nodes.

  • Jobs in a deprecated queue are not processed and may accumulate over time. These jobs are reflected in the monitor dashboard's "Aqueduct queue depth" graph as an increase in resource_activity. In some cases, a buildup of unprocessed jobs can result in memory exhaustion. If you observe memory exhaustion on your instance and see a high metric for resource_activity, contact GitHub Support.

  • When an administrator uses the -p flag with the ghe-support-bundle utility to collect data for a specific number of hours, the utility erroneously collects more logs than necessary.

  • On an instance hosted in AWS, system time may lose synchronization with Amazon's servers after an administrator reboots the instance. [Updated 2023-11-10]

  • On an instance with GitHub Actions enabled, after an upgrade from GitHub Enterprise Server 3.8 or earlier, an internal exception could prevent successful completion of some operations, like upgrades or the configuration of new replica nodes for high availability. If this issue occurs, administrators may see the following error in /data/user/common/ghe-config.log.

    Error occurred while executing servicing step 'Clone datatier login to secondary replica' for component CopyAvailabilityGroupSqlLogins during CopyAvailabilityGroupSqlLogins: Object reference not set to an instance of an object.
    

    To resolve this issue, upgrade to the latest patch release of GitHub Enterprise Server. [Updated: 2023-12-04]

  • On an instance in a cluster configuration, restoration of a backup using ghe-restore will exit prematurely if Redis has not restarted properly. [Updated 2023-12-05]

  • On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1. [Updated 2023-12-13]

  • During periods of high traffic, interruptions in service may occur due to insufficient resource allocations for internal components. [Updated 2024-01-23]

  • The settings for enabling scheduled reminders were added unintentionally to this release. Scheduled reminders are not officially supported. [Updated: 2024-02-22]

  • Redundant messages may cause an increase in the volume of events logged in /var/log/syslog. [Updated: 2024-03-08]

September 21, 2023

📣 Это не последний выпуск исправлений этой серии выпусков, и это не последний выпуск Enterprise Server. Используйте последний выпуск для последних исправлений безопасности, производительности и ошибок.

Warning: A change to MySQL in GitHub Enterprise Server 3.9 and later may impact the performance of your instance. Before you upgrade, make sure you've read the "Known issues" section of these release notes.

3.9.5: Security fixes

  • HTTP Strict Transport Security (HSTS) is enabled within the Management Console.

  • Packages have been updated to the latest security versions.

3.9.5: Bug fixes

  • On an instance with GitHub Actions enabled, scale sets configured at the enterprise level did not appear for use within the instance's organizations or repositories.

  • On an instance with a GitHub Advanced Security license and secret scanning enabled, secret scanning alerts could fail to show an error message in the UI when a failure occurred closing or reopening the alert.

  • On an instance with a GitHub Advanced Security license and secret scanning enabled, and when using Safari, changing additional match requirements for a custom pattern did not retrigger custom pattern evaluation against a user submitted test string.

  • On an instance with a GitHub Advanced Security license and secret scanning enabled, when token location(s) included a commit that introduced a large change, the page for viewing the alert would load slowly.

  • In some cases, users could reopen a pull request that should not have been able to be reopened.

  • When running the ghe-saml-mapping-csv CLI command in dry run mode, the operation failed with errors.

  • When uploading migration archives to blob storage, the GitHub Enterprise Server instance's outbound web proxy server was not used.

  • On an enterprise with the policy setting that disallows repository admins from enabling/disabling secret scanning, transferring a repository to a new organization that automatically enabled secret scanning wouldn't result in the transferred repository being automatically enabled for secret scanning.

  • When viewing fine-grained personal access tokens, the permissions text for pre-receive hooks was not visible for selection when filtering by permission.

  • When migrating a repository from a GitHub Enterprise Server instance to another location, the ghe-migrator target_url command allows you to record the repository's new location. The new URL is displayed when you visit the main page of the repository in the web interface.

  • On an instance with subdomain isolation disabled, a notebook could not be loaded due to incorrect asset paths.

  • On an instance with subdomain isolation disabled, a notebook could not be loaded due to an extra / character in the URL path.

  • On an instance with a GitHub Advanced Security license and secret scanning enabled, in some cases, custom patterns would erroneously show no results for a dry run.

  • On an instance with GitHub Actions enabled, the software on ephemeral runners would not automatically update to the latest version.

  • On an instance with subdomain isolation disabled, Mermaid diagrams in the web UI display an "Unable to render rich display" error and fail to render.

3.9.5: Changes

  • When listing the node metadata for all nodes using the Manage GitHub Enterprise Server REST API, information about whether a given node is a replica is included.

  • When GitHub Enterprise checks for a new upgrade or hotpatch package, if the check fails the failure details are output to the ghe-update-check log, and the Management Console UI provides a "Check Again" button to rerun the check.

  • When providing data to GitHub Support, GitHub Enterprise Server displays a notice describing how support data is used before uploading the support files.

  • When running async repository repairs, the output message about scheduling a repair job is more accurate.

3.9.5: Known issues

  • In GitHub Enterprise Server 3.10 and later, the requirements for TLS security levels have changed due to an upgrade to containers in the underlying OS. After upgrading from GitHub Enterprise Server 3.9, on an instance with GitHub Actions enabled and a custom TLS certificate, users may experience disruptions with workflow runs if the TLS certificate uses weak encryption. Workflow runs will not trigger, and the following error message will appear in system logs for babeld.

    CA certificate key too weak
    

    To resolve this issue, confirm that your certificate complies with level 2 of the OpenSSL security specification. For more information, see SSL_CTX_set_security_level in the OpenSSL docs. For more information about reviewing your instance's logs, see "Сведения о системных журналах".

    If the error appears in babeld logs because your TLS certificate does not comply with level 2 of the specification, you must create and upload a new certificate with stronger security before you upgrade to GitHub Enterprise Server 3.10 or later. For more information, see "Настройка TLS." [Updated: 2023-10-26]

  • Custom firewall rules are removed during the upgrade process.

  • During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

  • If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see "Troubleshooting access to the Management Console." [Updated: 2023-02-23]

  • On an instance in a high-availability configuration, passive replica nodes accept Git client requests and forward the requests to the primary node.

  • If an instance is configured to forward logs to a target server with TLS enabled, certificate authority (CA) bundles that a site administrator uploads using ghe-ssl-ca-certificate-install are not respected, and connections to the server fail.

  • When running ghe-config-apply, the process may stall with the message Deployment is running pending automatic promotion.

  • The mbind: Operation not permitted error in the /var/log/mysql/mysql.err file can be ignored. MySQL 8 does not gracefully handle when the CAP_SYS_NICE capability isn't required, and outputs an error instead of a warning.

  • When enabling CodeQL via default setup at scale, some checks related to GitHub Actions are omitted, potentially preventing the process from completing.

  • After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8, to 3.9 or 3.10, MySQL may not start back up. For more information, see "Известные проблемы с обновлением экземпляра." [Updated: 2023-08-11]

  • After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8 to 3.9 or 3.10, I/O utilization will increase, and in some cases the instance's performance will be impacted. Reduced performance is due to the database server being upgraded from MySQL 5.7 to MySQL 8.0. For more information, see "Известные проблемы с обновлением экземпляра."

  • In rare circumstances, a small instance with both high availability and GitHub Actions configured may report that MSSQL replication is unhealthy after many upgrades with full upgrade packages. If you encounter this issue, contact GitHub Support.

  • On an instance in a cluster configuration with high availability configured, ghe-config-apply times out while waiting for hookshot-go to start on replica application nodes.

  • Jobs in a deprecated queue are not processed and may accumulate over time. These jobs are reflected in the monitor dashboard's "Aqueduct queue depth" graph as an increase in resource_activity. In some cases, a buildup of unprocessed jobs can result in memory exhaustion. If you observe memory exhaustion on your instance and see a high metric for resource_activity, contact GitHub Support. [Updated: 2023-10-10]

  • When an administrator uses the -p flag with the ghe-support-bundle utility to collect data for a specific number of hours, the utility erroneously collects more logs than necessary. [Updated: 2023-10-13]

  • The settings for enabling scheduled reminders were added unintentionally to this release. Scheduled reminders are not officially supported. [Updated: 2023-10-17]

  • On an instance with GitHub Actions enabled, after an upgrade from GitHub Enterprise Server 3.8 or earlier, an internal exception could prevent successful completion of some operations, like upgrades or the configuration of new replica nodes for high availability. If this issue occurs, administrators may see the following error in /data/user/common/ghe-config.log.

    Error occurred while executing servicing step 'Clone datatier login to secondary replica' for component CopyAvailabilityGroupSqlLogins during CopyAvailabilityGroupSqlLogins: Object reference not set to an instance of an object.
    

    To resolve this issue, upgrade to the latest patch release of GitHub Enterprise Server. [Updated: 2023-12-04]

  • On an instance hosted in AWS, system time may lose synchronization with Amazon's servers after an administrator reboots the instance. [Updated 2023-11-10]

  • On an instance in a cluster configuration, restoration of a backup using ghe-restore will exit prematurely if Redis has not restarted properly. [Updated 2023-12-05]

  • On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1. [Updated 2023-12-13]

  • During periods of high traffic, interruptions in service may occur due to insufficient resource allocations for internal components. [Updated 2024-01-23]

  • Redundant messages may cause an increase in the volume of events logged in /var/log/syslog. [Updated: 2024-03-08]

August 24, 2023

📣 Это не последний выпуск исправлений этой серии выпусков, и это не последний выпуск Enterprise Server. Используйте последний выпуск для последних исправлений безопасности, производительности и ошибок.

Warning: A change to MySQL in GitHub Enterprise Server 3.9 and later may impact the performance of your instance. Before you upgrade, make sure you've read the "Known issues" section of these release notes.

3.9.4: Security fixes

  • An authorization/sensitive information disclosure vulnerability was identified in GitHub Enterprise Server that allowed a fork to retain read access to an upstream repository after the fork's visibility was changed to private. This vulnerability was reported via the GitHub Bug Bounty Program and assigned CVE-2023-23763. [Updated: 2023-09-01]

  • Packages have been updated to the latest security versions.

3.9.4: Bug fixes

  • On an instance with GitHub Actions enabled, scale sets configured at the enterprise level did not appear for use within the instance's organizations or repositories.

  • When an administrator tried to validate blob storage connection settings for GitHub Enterprise Importer in the Management Console using the Test storage settings button, the operation failed.

  • syslog-ng configurations for containerized services caused errors for log forwarding services. The configurations have been removed.

  • When an instance exhausted available memory, in some cases, the system's out-of-memory killer (OOMK) killed the process for dockerd, causing Nomad to fail to recover after systemd restarted Docker.

  • In some cases, when starting a new GitHub Enterprise Server instance, the preflight page indicated that there was no user disk of sufficient size attached.

  • When running the ghe-migrator, certain error messages contained an invalid link to import documentation.

  • On an instance with GitHub Actions enabled, due to mismatched values, users could not easily associate workflow job run IDs from the GitHub Enterprise Server APIs or webhooks with a job in the UI. Workflow job runs now use a new URL pattern of ...actions/runs/job/{job_id}, and job_id matches values from APIs and webhook payloads.

  • Administrators could not see or use the "Migrations" section in an instance's Management Console, which prevented the configuration of blob storage for GitHub Enterprise Importer. [Updated: 2023-08-31]

3.9.4: Known issues

  • In GitHub Enterprise Server 3.10 and later, the requirements for TLS security levels have changed due to an upgrade to containers in the underlying OS. After upgrading from GitHub Enterprise Server 3.9, on an instance with GitHub Actions enabled and a custom TLS certificate, users may experience disruptions with workflow runs if the TLS certificate uses weak encryption. Workflow runs will not trigger, and the following error message will appear in system logs for babeld.

    CA certificate key too weak
    

    To resolve this issue, confirm that your certificate complies with level 2 of the OpenSSL security specification. For more information, see SSL_CTX_set_security_level in the OpenSSL docs. For more information about reviewing your instance's logs, see "Сведения о системных журналах".

    If the error appears in babeld logs because your TLS certificate does not comply with level 2 of the specification, you must create and upload a new certificate with stronger security before you upgrade to GitHub Enterprise Server 3.10 or later. For more information, see "Настройка TLS." [Updated: 2023-10-26]

  • Custom firewall rules are removed during the upgrade process.

  • During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

  • If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see "Troubleshooting access to the Management Console." [Updated: 2023-02-23]

  • On an instance in a high-availability configuration, passive replica nodes accept Git client requests and forward the requests to the primary node.

  • If an instance is configured to forward logs to a target server with TLS enabled, certificate authority (CA) bundles that a site administrator uploads using ghe-ssl-ca-certificate-install are not respected, and connections to the server fail.

  • When running ghe-config-apply, the process may stall with the message Deployment is running pending automatic promotion.

  • The mbind: Operation not permitted error in the /var/log/mysql/mysql.err file can be ignored. MySQL 8 does not gracefully handle when the CAP_SYS_NICE capability isn't required, and outputs an error instead of a warning.

  • On an instance with subdomain isolation disabled, Mermaid diagrams in the web UI display an "Unable to render rich display" error and fail to render.

  • When enabling CodeQL via default setup at scale, some checks related to GitHub Actions are omitted, potentially preventing the process from completing.

  • After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8, to 3.9 or 3.10, MySQL may not start back up. For more information, see "Известные проблемы с обновлением экземпляра." [Updated: 2023-08-11]

  • After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8 to 3.9 or 3.10, I/O utilization will increase, and in some cases the instance's performance will be impacted. Reduced performance is due to the database server being upgraded from MySQL 5.7 to MySQL 8.0. For more information, see "Известные проблемы с обновлением экземпляра."

  • In rare circumstances, a small instance with both high availability and GitHub Actions configured may report that MSSQL replication is unhealthy after many upgrades with full upgrade packages. If you encounter this issue, contact GitHub Support. [Updated: 2023-09-04]

  • On an instance in a cluster configuration with high availability configured, ghe-config-apply times out while waiting for hookshot-go to start on replica application nodes. [Updated: 2023-09-21]

  • On an instance with GitHub Actions enabled, ephemeral self-hosted runners do not automatically update to the latest version. Users will need to manually update the runners to the latest version. [Updated: 2023-09-29]

  • Jobs in a deprecated queue are not processed and may accumulate over time. These jobs are reflected in the monitor dashboard's "Aqueduct queue depth" graph as an increase in resource_activity. In some cases, a buildup of unprocessed jobs can result in memory exhaustion. If you observe memory exhaustion on your instance and see a high metric for resource_activity, contact GitHub Support. [Updated: 2023-10-10]

  • When an administrator uses the -p flag with the ghe-support-bundle utility to collect data for a specific number of hours, the utility erroneously collects more logs than necessary. [Updated: 2023-10-13]

  • The settings for enabling scheduled reminders were added unintentionally to this release. Scheduled reminders are not officially supported. [Updated: 2023-10-17]

  • On an instance with GitHub Actions enabled, after an upgrade from GitHub Enterprise Server 3.8 or earlier, an internal exception could prevent successful completion of some operations, like upgrades or the configuration of new replica nodes for high availability. If this issue occurs, administrators may see the following error in /data/user/common/ghe-config.log.

    Error occurred while executing servicing step 'Clone datatier login to secondary replica' for component CopyAvailabilityGroupSqlLogins during CopyAvailabilityGroupSqlLogins: Object reference not set to an instance of an object.
    

    To resolve this issue, upgrade to the latest patch release of GitHub Enterprise Server. [Updated: 2023-12-04]

  • On an instance hosted in AWS, system time may lose synchronization with Amazon's servers after an administrator reboots the instance. [Updated 2023-11-10]

  • On an instance in a cluster configuration, restoration of a backup using ghe-restore will exit prematurely if Redis has not restarted properly. [Updated 2023-12-05]

  • On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1. [Updated 2023-12-13]

  • During periods of high traffic, interruptions in service may occur due to insufficient resource allocations for internal components. [Updated 2024-01-23]

  • Redundant messages may cause an increase in the volume of events logged in /var/log/syslog. [Updated: 2024-03-08]

August 10, 2023

📣 Это не последний выпуск исправлений этой серии выпусков, и это не последний выпуск Enterprise Server. Используйте последний выпуск для последних исправлений безопасности, производительности и ошибок.

Warning: A change to MySQL in GitHub Enterprise Server 3.9 and later may impact the performance of your instance. Before you upgrade, make sure you've read the "Known issues" section of these release notes.

3.9.3: Security fixes

  • LOW: An incorrect comparison vulnerability was identified in GitHub Enterprise Server that allowed commit smuggling by displaying an incorrect diff in a reopened pull request. To exploit this vulnerability, an attacker would need write access to the repository. This vulnerability was reported via the GitHub Bug Bounty program and was assigned CVE-2023-23766. [Updated: 2023-09-22]

3.9.3: Bug fixes

  • API results were incomplete, and ordering of results was incorrect if asc or desc appeared in lowercase within the API query.

  • The checks in the merge box for a pull request did not always match the the checks for the most recent commit in the pull request.

  • When a site administrator used GitHub Enterprise Importer on versions 3.7 and below to migrate repositories from GitHub Enterprise Server, the system backup size would increase after running many migrations due to storage files not being cleaned up.

  • A collaborator with the "Set the social preview" permission inherited from the "Read" role could not upload the social preview image of a repository.

  • The security settings page for a repository would return an error when enterprise-level runners were assigned to the repository.

  • GitHub Enterprise Server was queuing zip jobs unnecessarily.

  • On an instance configured to use an outbound web proxy server, an administrator could not exclude private domains in this list from the proxy configuration. [Updated: 2023-11-27]

3.9.3: Changes

  • On GitHub Enterprise Server 3.8 and above, a blob storage provider must be configured in the Management Console in order to use the GitHub Enterprise Importer CLI, "startRepositoryMigration" GraphQL API, or "Start an organization migration" REST API. The "Migrations" section in the Management Console was mistakenly removed and has been added back.

  • Administrators can display all repositories in a network with spokesctl by using the repositories subcommand.

  • The secondary abuse rate limits of the GraphQL API are now configurable in the Management Console. [Updated: 2023-09-01]

3.9.3: Known issues

  • After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8 to 3.9 or 3.10, I/O utilization will increase, and in some cases the instance's performance will be impacted. Reduced performance is due to the database server being upgraded from MySQL 5.7 to MySQL 8.0. For more information, see "Известные проблемы с обновлением экземпляра." [Updated: 2023-08-11]

  • After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8, to 3.9 or 3.10, MySQL may not start back up. For more information, see "Известные проблемы с обновлением экземпляра." [Updated: 2023-08-11]

  • In GitHub Enterprise Server 3.10 and later, the requirements for TLS security levels have changed due to an upgrade to containers in the underlying OS. After upgrading from GitHub Enterprise Server 3.9, on an instance with GitHub Actions enabled and a custom TLS certificate, users may experience disruptions with workflow runs if the TLS certificate uses weak encryption. Workflow runs will not trigger, and the following error message will appear in system logs for babeld.

    CA certificate key too weak
    

    To resolve this issue, confirm that your certificate complies with level 2 of the OpenSSL security specification. For more information, see SSL_CTX_set_security_level in the OpenSSL docs. For more information about reviewing your instance's logs, see "Сведения о системных журналах".

    If the error appears in babeld logs because your TLS certificate does not comply with level 2 of the specification, you must create and upload a new certificate with stronger security before you upgrade to GitHub Enterprise Server 3.10 or later. For more information, see "Настройка TLS." [Updated: 2023-10-26]

  • Custom firewall rules are removed during the upgrade process.

  • During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

  • If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see "Troubleshooting access to the Management Console." [Updated: 2023-02-23]

  • On an instance in a high-availability configuration, passive replica nodes accept Git client requests and forward the requests to the primary node.

  • If an instance is configured to forward logs to a target server with TLS enabled, certificate authority (CA) bundles that a site administrator uploads using ghe-ssl-ca-certificate-install are not respected, and connections to the server fail.

  • When running ghe-config-apply, the process may stall with the message Deployment is running pending automatic promotion.

  • The mbind: Operation not permitted error in the /var/log/mysql/mysql.err file can be ignored. MySQL 8 does not gracefully handle when the CAP_SYS_NICE capability isn't required, and outputs an error instead of a warning.

  • On an instance with subdomain isolation disabled, Mermaid diagrams in the web UI display an "Unable to render rich display" error and fail to render.

  • When enabling CodeQL via default setup at scale, some checks related to GitHub Actions are omitted, potentially preventing the process from completing.

  • Administrators cannot set or update the instance's blob storage settings in the Management Console using the "Migrations" settings tab. No matter what values an administrator provides, the following error message will appear: Unable to connect to migrations provider. Please check the form and try again. [Updated: 2023-08-18]

  • In rare circumstances, a small instance with both high availability and GitHub Actions configured may report that MSSQL replication is unhealthy after many upgrades with full upgrade packages. If you encounter this issue, contact GitHub Support. [Updated: 2023-08-24]

  • On an instance in a cluster configuration with high availability configured, ghe-config-apply times out while waiting for hookshot-go to start on replica application nodes. [Updated: 2023-09-21]

  • On an instance with GitHub Actions enabled, ephemeral self-hosted runners do not automatically update to the latest version. Users will need to manually update the runners to the latest version. [Updated: 2023-09-29]

  • Jobs in a deprecated queue are not processed and may accumulate over time. These jobs are reflected in the monitor dashboard's "Aqueduct queue depth" graph as an increase in resource_activity. In some cases, a buildup of unprocessed jobs can result in memory exhaustion. If you observe memory exhaustion on your instance and see a high metric for resource_activity, contact GitHub Support. [Updated: 2023-10-10]

  • When an administrator uses the -p flag with the ghe-support-bundle utility to collect data for a specific number of hours, the utility erroneously collects more logs than necessary. [Updated: 2023-10-13]

  • The settings for enabling scheduled reminders were added unintentionally to this release. Scheduled reminders are not officially supported. [Updated: 2023-10-17]

  • On an instance with GitHub Actions enabled, after an upgrade from GitHub Enterprise Server 3.8 or earlier, an internal exception could prevent successful completion of some operations, like upgrades or the configuration of new replica nodes for high availability. If this issue occurs, administrators may see the following error in /data/user/common/ghe-config.log.

    Error occurred while executing servicing step 'Clone datatier login to secondary replica' for component CopyAvailabilityGroupSqlLogins during CopyAvailabilityGroupSqlLogins: Object reference not set to an instance of an object.
    

    To resolve this issue, upgrade to the latest patch release of GitHub Enterprise Server. [Updated: 2023-12-04]

  • On an instance hosted in AWS, system time may lose synchronization with Amazon's servers after an administrator reboots the instance. [Updated 2023-11-10]

  • On an instance in a cluster configuration, restoration of a backup using ghe-restore will exit prematurely if Redis has not restarted properly. [Updated 2023-12-05]

  • On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1. [Updated 2023-12-13]

  • During periods of high traffic, interruptions in service may occur due to insufficient resource allocations for internal components. [Updated 2024-01-23]

  • Redundant messages may cause an increase in the volume of events logged in /var/log/syslog. [Updated: 2024-03-08]

July 28, 2023

📣 Это не последний выпуск исправлений этой серии выпусков, и это не последний выпуск Enterprise Server. Используйте последний выпуск для последних исправлений безопасности, производительности и ошибок.

Warning: A change to MySQL in GitHub Enterprise Server 3.9 and later may impact the performance of your instance. Before you upgrade, make sure you've read the "Known issues" section of these release notes.

The known issues originally published on 2023-07-28 omitted a number of known issues that still existed. The Known issues section below was updated on 2023-08-08.

3.9.2: Changes

  • Added a pre-upgrade check to validate the GHES version and MySQL configuration before allowing an upgrade to 3.9.

  • Adjusted the timeout threshold for shutting down MySQL to prevent premature termination when upgrading to GHES 3.9.

3.9.2: Known issues

  • After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8 to 3.9, I/O utilization will increase, and in some cases the instance's performance will be impacted. Reduced performance is due to the database server being upgraded from MySQL 5.7 to MySQL 8.0. For more information, see "Известные проблемы с обновлением экземпляра."

  • After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8, to 3.9 or 3.10, MySQL may not start back up. For more information, see "Известные проблемы с обновлением экземпляра." [Updated: 2023-08-11]

  • In GitHub Enterprise Server 3.10 and later, the requirements for TLS security levels have changed due to an upgrade to containers in the underlying OS. After upgrading from GitHub Enterprise Server 3.9, on an instance with GitHub Actions enabled and a custom TLS certificate, users may experience disruptions with workflow runs if the TLS certificate uses weak encryption. Workflow runs will not trigger, and the following error message will appear in system logs for babeld.

    CA certificate key too weak
    

    To resolve this issue, confirm that your certificate complies with level 2 of the OpenSSL security specification. For more information, see SSL_CTX_set_security_level in the OpenSSL docs. For more information about reviewing your instance's logs, see "Сведения о системных журналах".

    If the error appears in babeld logs because your TLS certificate does not comply with level 2 of the specification, you must create and upload a new certificate with stronger security before you upgrade to GitHub Enterprise Server 3.10 or later. For more information, see "Настройка TLS." [Updated: 2023-10-26]

  • The Management Console may get stuck in a loop showing "Checking requirements..." when attempting to download an update. It is safe to proceed the update process manually via the command line.

  • When enabling CodeQL via default setup at scale, some checks related to GitHub Actions are omitted, potentially preventing the process from completing.

  • After restoration of a backup created using GitHub Enterprise Server Backup Utilities 3.7.0, 3.8.0, or 3.9.0, users may not be able to sign into the instance. To fix this issue, plus a bug that was preventing secret scanning encryption keys from being backed up, upgrade your backup host to use GitHub Enterprise Server Backup Utilities 3.9.1 and generate a new full backup using ghe-backup. For more information on using an existing backup, see "Известные проблемы с резервными копиями для вашего экземпляра." [Updated: 2023-07-31]

  • On an instance in a cluster configuration, after you upgrade nodes other than the primary MySQL node and before you upgrade the primary MySQL node, the following output may appear multiple times after you run ghe-config-apply.

    Error response from daemon: conflict: unable to delete IMAGE_ID (cannot be forced) - image is being used by running container CONTAINER_ID
    

    You can safely ignore this message.

  • Custom firewall rules are removed during the upgrade process.

  • On an instance in a high-availability configuration, passive replica nodes accept Git client requests and forward the requests to the primary node.

  • The mbind: Operation not permitted error in the /var/log/mysql/mysql.err file can be ignored. MySQL 8 does not gracefully handle when the CAP_SYS_NICE capability isn't required, and outputs an error instead of a warning.

  • When using an outbound web proxy server, the ghe-btop command may fail in some circumstances with the error "Error querying allocation: Unexpected response code: 401".

  • If an instance is configured to forward logs to a target server with TLS enabled, certificate authority (CA) bundles that a site administrator uploads using ghe-ssl-ca-certificate-install are not respected, and connections to the server fail.

  • When running ghe-config-apply, the process may stall with the message Deployment is running pending automatic promotion.

  • During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

  • If the root site administrator is locked out of the Management Console after failed login attempts, the account will not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see "Устранение неполадок с доступом к консоли управления."

  • On an instance with subdomain isolation disabled, Mermaid diagrams in the web UI display an "Unable to render rich display" error and fail to render. [Updated: 2023-08-18]

  • The "Migrations" section is missing from the Management Console, so it isn't possible to enable, disable, or reconfigure blob storage credentials for migrations. [Updated: 2023-08-18]

  • In rare circumstances, a small instance with both high availability and GitHub Actions configured may report that MSSQL replication is unhealthy after many upgrades with full upgrade packages. If you encounter this issue, contact GitHub Support. [Updated: 2023-08-24]

  • On an instance with GitHub Actions enabled, if shared runner groups are configured for the enterprise, the enterprise security overview page may return a 500 error. You can avoid the issue by trying one of the following workarounds.

    • Add a runner scale set to the enterprise runner group shared with the repositories.
    • Remove access to the enterprise runner group from the affected repositories or organizations.

    [Updated: 2023-09-05]

  • On an instance in a cluster configuration with high availability configured, ghe-config-apply times out while waiting for hookshot-go to start on replica application nodes. [Updated: 2023-09-21]

  • On an instance with GitHub Actions enabled, ephemeral self-hosted runners do not automatically update to the latest version. Users will need to manually update the runners to the latest version. [Updated: 2023-09-29]

  • Jobs in a deprecated queue are not processed and may accumulate over time. These jobs are reflected in the monitor dashboard's "Aqueduct queue depth" graph as an increase in resource_activity. In some cases, a buildup of unprocessed jobs can result in memory exhaustion. If you observe memory exhaustion on your instance and see a high metric for resource_activity, contact GitHub Support. [Updated: 2023-10-10]

  • When an administrator uses the -p flag with the ghe-support-bundle utility to collect data for a specific number of hours, the utility erroneously collects more logs than necessary. [Updated: 2023-10-13]

  • The settings for enabling scheduled reminders were added unintentionally to this release. Scheduled reminders are not officially supported. [Updated: 2023-10-17]

  • On an instance with GitHub Actions enabled, after an upgrade from GitHub Enterprise Server 3.8 or earlier, an internal exception could prevent successful completion of some operations, like upgrades or the configuration of new replica nodes for high availability. If this issue occurs, administrators may see the following error in /data/user/common/ghe-config.log.

    Error occurred while executing servicing step 'Clone datatier login to secondary replica' for component CopyAvailabilityGroupSqlLogins during CopyAvailabilityGroupSqlLogins: Object reference not set to an instance of an object.
    

    To resolve this issue, upgrade to the latest patch release of GitHub Enterprise Server. [Updated: 2023-12-04]

  • On an instance hosted in AWS, system time may lose synchronization with Amazon's servers after an administrator reboots the instance. [Updated 2023-11-10]

  • On an instance in a cluster configuration, restoration of a backup using ghe-restore will exit prematurely if Redis has not restarted properly. [Updated 2023-12-05]

  • On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1. [Updated 2023-12-13]

  • During periods of high traffic, interruptions in service may occur due to insufficient resource allocations for internal components. [Updated 2024-01-23]

  • Redundant messages may cause an increase in the volume of events logged in /var/log/syslog. [Updated: 2024-03-08]

July 18, 2023

📣 Это не последний выпуск исправлений этой серии выпусков, и это не последний выпуск Enterprise Server. Используйте последний выпуск для последних исправлений безопасности, производительности и ошибок.

Warning: A change to MySQL in GitHub Enterprise Server 3.9 and later may impact the performance of your instance. Before you upgrade, make sure you've read the "Known issues" section of these release notes.

The known issues originally published on 2023-07-18 omitted a number of known issues that still existed. The Known issues section below was updated on 2023-08-08.

3.9.1: Security fixes

  • MEDIUM: An attacker with write access to a repository could craft a pull request that would hide commits made in its source branch. This vulnerability was reported via the GitHub Bug Bounty Program and has been assigned CVE-2023-23764. [Updated: 2023-07-26]

  • An attacker with access to the password hash of the root site administrator user for the instance's Management Console could make requests to the password API endpoint from outside of the instance.

  • Packages have been updated to the latest security versions.

  • LOW: An incorrect comparison vulnerability was identified in GitHub Enterprise Server that allowed commit smuggling by displaying an incorrect diff in a re-opened Pull Request. To exploit this vulnerability, an attacker would need write access to the repository. This vulnerability was reported via the GitHub Bug Bounty Program and was assigned CVE-2023-23765.

3.9.1: Bug fixes

  • If MinIO was configured for external blob storage on an instance with GitHub Actions enabled and MinIO was configured for bucket replication, the instance's credential validation with MinIO would occasionally fail.

  • Customers who use Azure Blob store as the remote blob provider to back GitHub Packages would have validation errors if the EndpointSuffix part of their Connection string was anything other than core.windows.net. Now all valid EndpointSuffix are accepted.

  • When a user viewed a Jupyter notebook, GitHub Enterprise Server returned a 500 error code if the instance was configured with a self-signed TLS certificate.

  • After creation of a blob object from the web UI, pre-receive hook events were missing from the instance's audit log.

  • On an instance with an outbound web proxy server configured, the proxy interfered with internal operations that used nomad alloc exec.

  • On an instance in a cluster configuration, the ghe-cluster-balance behaved inconsistently when displaying status or managing jobs with more than one task group.

  • .topojson files would not render correctly, but files that conformed to the TopoJSON spec that used a .geojson extension would render correctly.

  • On an instance configured for LDAP authentication, if the LDAP server sent an empty string for the sshPublicKey attribute, LDAP user sync would fail.

  • REST API endpoints for managing GitHub Enterprise Server are now functional. For more information, see "Manage GitHub Enterprise Server" in the REST API documentation.

  • After creation of a new Management Console user, the Management Console did not display the button to copy the new users invitation.

  • On an instance with Dependabot enabled, in some situations, Dependabot alerts were not updated when a user pushed to a repository.

  • In some cases, pull requests with more than 25 rich-diff renderable files required that users toggle the diff type to correctly render the files over the 25-file limit.

  • In rare circumstances, Git commits signed with SSH keys using the RSA algorithm would incorrectly indicate the signature was invalid.

  • After a migration using GitHub Enterprise Importer, some repository autolink references were created with an incorrect format.

  • In some cases on an instance without a GitHub Advanced Security license, Redis exceeded the maximum default memory allocation, causing 500 errors for the instance's users.

  • On an instance with many organizations, the enterprise security overview page returned a 500 error.

  • On an instance that was not configured to deliver email notifications using SMTP, background jobs to deliver email were enqueued unnecessarily.

  • Users were unable to configure a SSH certificate authority for an organization.

  • An erroneous "Blocked Copilot Repositories" link was visible in site admin pages for organizations.

  • On an instance with GitHub Actions enabled and a GitHub Advanced Security license, repository-level runner scale sets were not accounted for when determining whether default setup for code scanning could be used.

  • Events related to repository notifications did not appear in the audit log.

  • On an instance with a GitHub Advanced Security license and secret scanning enabled, in some cases, a committer would not receive an email notification for a secret scanning alert where push protections were bypassed.

  • On an instance with a GitHub Advanced Security license, if a user filtered by a custom pattern on an organizations "Code & security analysis" page using an invalid query, the entire GitHub Advanced Security disappeared and an error reading "Sorry, something went wrong loading GitHub Advanced Security settings" appeared.

  • On an instance with a GitHub Advanced Security license, if a user browsed to the alerts page for secret scanning without signing in, the instance responded with a 500 error.

  • On an instance with a GitHub Advanced Security license and secret scanning enabled, output from Git for a push blocked by push protection always included an http:// link.

  • On an instance with GitHub Actions enabled, links to http(s)://HOSTNAME/features/actions from the web UI returned a 500 error.

  • If a user added a new item to a projects roadmap view, and the item was outside of the viewport, the view would crash and display "This project failed to load".

  • The audit log reported the incorrect target repository for pre-receive hook failures.

  • Users can add issues and pull requests from any organization to a project, and are no longer limited to the user or organization of the project.

  • On an instance with GitHub Actions enabled and a GitHub Advanced Security license, enterprise-level runner scale sets with the code-scanning label were not sufficient to allow default setup for code scanning.

  • On an instance in a high availability configuration, existing nodes with out-of-sync repositories prevented new nodes from replicating those repositories.

  • On an instance with multiple nodes, ERROR-level "resolver failed" errors no longer appear in system logs when the instance is unable to resolve an offline fileserver. The messages are now DEBUG-level.

  • On an instance with a GitHub Advanced Security license that was also configured for a timezone greater than UTC, the list of secret scanning alerts displayed a "Loading secrets failed" error if a user sorted secrets by date in descending order.

  • Code Scanning workflow runs now only request the code-scanning label so that they can be used with runner scale sets.

3.9.1: Changes

  • On an instance in a cluster configuration, the ghe-cluster-config-check command-line utility will return an affirmative message when no warnings or errors are detected. The affirmative message is "Configuration validation complete. No errors found."

  • During initialization of a cluster configuration, output from the ghe-cluster-config-init command-line utility is improved and simplified.

  • The API endpoint for management of the GitHub Enterprise Server instance was unavailable prior to initial configuration of the instance.

  • The Management Console displays a warning about unexpected consequences that may result from modification of the instance's hostname after initial configuration.

  • On an instance with multiple nodes, internal tooling to repair repositories now attempts to resolve problems within the entire repository network.

  • To supplement a disaster recovery plan for a GitHub Enterprise Server instance in a cluster configuration, an administrator can configure a replica of an entire cluster in a separate datacenter, allowing the cluster to fail over to redundant nodes. For more information, see "Настройка репликации с высоким уровнем доступности для кластера."

3.9.1: Known issues

  • After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8 to 3.9 or 3.10, I/O utilization will increase, and in some cases the instance's performance will be impacted. Reduced performance is due to the database server being upgraded from MySQL 5.7 to MySQL 8.0. For more information, see "Известные проблемы с обновлением экземпляра."

  • After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8, to 3.9 or 3.10, MySQL may not start back up. For more information, see "Известные проблемы с обновлением экземпляра." [Updated: 2023-08-11]

  • In GitHub Enterprise Server 3.10 and later, the requirements for TLS security levels have changed due to an upgrade to containers in the underlying OS. After upgrading from GitHub Enterprise Server 3.9, on an instance with GitHub Actions enabled and a custom TLS certificate, users may experience disruptions with workflow runs if the TLS certificate uses weak encryption. Workflow runs will not trigger, and the following error message will appear in system logs for babeld.

    CA certificate key too weak
    

    To resolve this issue, confirm that your certificate complies with level 2 of the OpenSSL security specification. For more information, see SSL_CTX_set_security_level in the OpenSSL docs. For more information about reviewing your instance's logs, see "Сведения о системных журналах".

    If the error appears in babeld logs because your TLS certificate does not comply with level 2 of the specification, you must create and upload a new certificate with stronger security before you upgrade to GitHub Enterprise Server 3.10 or later. For more information, see "Настройка TLS." [Updated: 2023-10-26]

  • The Management Console may get stuck in a loop showing "Checking requirements..." when attempting to download an update. It is safe to proceed the update process manually via the command line.

  • When enabling CodeQL via default setup at scale, some checks related to GitHub Actions are omitted, potentially preventing the process from completing.

  • After restoration of a backup created using GitHub Enterprise Server Backup Utilities 3.7.0, 3.8.0, or 3.9.0, users may not be able to sign into the instance. To fix this issue, plus a bug that was preventing secret scanning encryption keys from being backed up, upgrade your backup host to use GitHub Enterprise Server Backup Utilities 3.9.1 and generate a new full backup using ghe-backup. For more information on using an existing backup, see "Известные проблемы с резервными копиями для вашего экземпляра." [Updated: 2023-07-31]

  • On an instance in a cluster configuration, after you upgrade nodes other than the primary MySQL node and before you upgrade the primary MySQL node, the following output may appear multiple times after you run ghe-config-apply.

    Error response from daemon: conflict: unable to delete IMAGE_ID (cannot be forced) - image is being used by running container CONTAINER_ID
    

    You can safely ignore this message.

  • Custom firewall rules are removed during the upgrade process.

  • On an instance in a high-availability configuration, passive replica nodes accept Git client requests and forward the requests to the primary node.

  • The mbind: Operation not permitted error in the /var/log/mysql/mysql.err file can be ignored. MySQL 8 does not gracefully handle when the CAP_SYS_NICE capability isn't required, and outputs an error instead of a warning.

  • When using an outbound web proxy server, the ghe-btop command may fail in some circumstances with the error "Error querying allocation: Unexpected response code: 401".

  • If an instance is configured to forward logs to a target server with TLS enabled, certificate authority (CA) bundles that a site administrator uploads using ghe-ssl-ca-certificate-install are not respected, and connections to the server fail.

  • When running ghe-config-apply, the process may stall with the message Deployment is running pending automatic promotion.

  • During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

  • If the root site administrator is locked out of the Management Console after failed login attempts, the account will not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see "Устранение неполадок с доступом к консоли управления."

  • On an instance with subdomain isolation disabled, Mermaid diagrams in the web UI display an "Unable to render rich display" error and fail to render. [Updated: 2023-08-18]

  • The "Migrations" section is missing from the Management Console, so it isn't possible to enable, disable, or reconfigure blob storage credentials for migrations. [Updated: 2023-08-18]

  • In rare circumstances, a small instance with both high availability and GitHub Actions configured may report that MSSQL replication is unhealthy after many upgrades with full upgrade packages. If you encounter this issue, contact GitHub Support. [Updated: 2023-08-24]

  • On an instance with GitHub Actions enabled, if shared runner groups are configured for the enterprise, the enterprise security overview page may return a 500 error. You can avoid the issue by trying one of the following workarounds.

    • Add a runner scale set to the enterprise runner group shared with the repositories.
    • Remove access to the enterprise runner group from the affected repositories or organizations.

    [Updated: 2023-09-05]

  • On an instance in a cluster configuration with high availability configured, ghe-config-apply times out while waiting for hookshot-go to start on replica application nodes. [Updated: 2023-09-21]

  • On an instance with GitHub Actions enabled, ephemeral self-hosted runners do not automatically update to the latest version. Users will need to manually update the runners to the latest version. [Updated: 2023-09-29]

  • Jobs in a deprecated queue are not processed and may accumulate over time. These jobs are reflected in the monitor dashboard's "Aqueduct queue depth" graph as an increase in resource_activity. In some cases, a buildup of unprocessed jobs can result in memory exhaustion. If you observe memory exhaustion on your instance and see a high metric for resource_activity, contact GitHub Support. [Updated: 2023-10-10]

  • When an administrator uses the -p flag with the ghe-support-bundle utility to collect data for a specific number of hours, the utility erroneously collects more logs than necessary. [Updated: 2023-10-13]

  • The settings for enabling scheduled reminders were added unintentionally to this release. Scheduled reminders are not officially supported. [Updated: 2023-10-17]

  • On an instance with GitHub Actions enabled, after an upgrade from GitHub Enterprise Server 3.8 or earlier, an internal exception could prevent successful completion of some operations, like upgrades or the configuration of new replica nodes for high availability. If this issue occurs, administrators may see the following error in /data/user/common/ghe-config.log.

    Error occurred while executing servicing step 'Clone datatier login to secondary replica' for component CopyAvailabilityGroupSqlLogins during CopyAvailabilityGroupSqlLogins: Object reference not set to an instance of an object.
    

    To resolve this issue, upgrade to the latest patch release of GitHub Enterprise Server. [Updated: 2023-12-04]

  • On an instance hosted in AWS, system time may lose synchronization with Amazon's servers after an administrator reboots the instance. [Updated 2023-11-10]

  • On an instance in a cluster configuration, restoration of a backup using ghe-restore will exit prematurely if Redis has not restarted properly. [Updated 2023-12-05]

  • On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1. [Updated 2023-12-13]

  • During periods of high traffic, interruptions in service may occur due to insufficient resource allocations for internal components. [Updated 2024-01-23]

  • Redundant messages may cause an increase in the volume of events logged in /var/log/syslog. [Updated: 2024-03-08]

June 08, 2023

📣 Это не последний выпуск исправлений этой серии выпусков, и это не последний выпуск Enterprise Server. Используйте последний выпуск для последних исправлений безопасности, производительности и ошибок.

For upgrade instructions, see "Обновление GitHub Enterprise Server."

Warning: A change to MySQL in GitHub Enterprise Server 3.9 and later may impact the performance of your instance. Before you upgrade, make sure you've read the "Known issues" section of these release notes.

3.9.0: Features

3.9.0: Changes

  • Field names and destinations for some service logs on GitHub Enterprise Server have changed. If any tooling or processes in your environment rely on specific field names within logs, or log entries in specific files, the following changes may affect you.

    • level is now SeverityText.
    • log_message, msg, or message is now Body.
    • now is now Timestamp.
    • Custom field names such as gh.repo.id or graphql.operation.name use semantic names.
    • Log statements that the instance would previously write to auth.log, ldap.log, or ldap-sync.log now appear in containerized logs for github-unicorn if the statement originated from a web request, or in logs for github-resqued if the statement originated from a background job. For more information about containerized logs, see "Сведения о системных журналах."

    For a full list of field mappings, download the OpenTelemetry attribute mapping CSV. This change is part of GitHub's gradual migration to internal semantic conventions for OpenTelemetry, and additional field names will change in upcoming releases.

  • On a configured instance, the name for the HAProxy service is now haproxy-frontend. Previously, the name was haproxy. Additionally, on an unconfigured instance, there is a new service named haproxy-pre-config. If your instance forwards logs to an external system, update your rules to reflect these changes. For more information, see "Пересылка журналов" article

  • For an instance or organization with 2FA enabled, when a user sets up 2FA, GitHub Enterprise Server suggests an authenticator app (TOTP) by default.

  • When a person with administrative SSH access to an instance submits a support bundle using either the ghe-support-bundle or ghe-cluster-support-bundle utility, a period for log collection specified with the -p or --period no longer requires quotes to enclose the date value. For more information, see "Служебные программы командной строки."

  • To provide additional context within the web interface on an instance where Dependabot alerts are enabled, links to Dependabot alerts in an issue or pull request comment display an improved label and hovercard with alert details.

  • On an instance with Dependabot alerts enabled, people with write or maintain access to a repository can view or act on Dependabot alerts by default. Custom roles, the security manager role, organization permissions, and notification settings are not affected.

  • On an instance with a GitHub Advanced Security license and GitHub Connect enabled for the synchronization of actions from GitHub.com, CodeQL code scanning is up to 16% faster. For more information, see "Настройка сканирования кода для (модуль)."

  • On an instance with a GitHub Advanced Security license and email configured for notifications, users can receive notifications for secret scanning alerts by watching a repository and choosing "All activity" or "Security alerts". To continue receiving notifications for secret scanning alerts in GitHub Enterprise Server 3.9 and later, users must enable email notifications in the web interface at http(s)://HOSTNAME/settings/notifications under "Watching" by choosing "Email".

  • On an instance with a GitHub Advanced Security license, secret scanning alerts display whether detected tokens from GitHub are valid.

  • On an instance with a GitHub Advanced Security license, the enterprise and organization audit logs now display an event when an owner enables or disables a push protection for a custom pattern for a repository, organization, or the enterprise. For more information, see "Просмотр журнала аудита для вашей организации" and "События журнала аудита для вашего предприятия."

  • Users can filter the lists of alerts for Dependabot, code scanning, and secret scanning by repository topic or team in the security overview for an organization. For more information, see "Обзор фильтрации оповещений в области безопасности."

  • In the security overview for an organization, the following improvements apply to the "Security coverage" view during feature enablement.

    • To provide insight into the number of GitHub Advanced Security licenses used, active committers for the repository are visible. For repositories where GitHub Advanced Security is not enabled, the number indicates the number of licenses required to enable the feature.
    • Unsaved changes are now labeled with a "Modified" tag, and the "Save security settings" button now displays the total number of changes to save.
    • While a security feature is being enabled, the "Security coverage" view shows a status of "Updating..." to inform you of the ongoing process.

    For more information, see "Общие сведения о безопасности."

  • In the security overview's "Security risk" and "Security coverage" views, when a user selects a team from the "Team" drop-down or filters by team, results appear for repositories where the team has write or administrative access or has been granted access to security alerts. Previously, users could only view results for repositories where the team had administrative access or had been granted access to security alerts.

  • To provide more context within a project, users can share a deep link to a specific issue in a project to have the issue open in the project's side panel.

  • Organization owners can create up to five custom repository roles. Previously, the limit was three. For more information, see "Пользовательские роли репозитория."

  • When transferring a repository, users can also rename the repository. For more information, see "Передача репозитория."

  • If a user archives a repository, responses from the GraphQL API that include information about the repository now include an archivedAt value with a timestamp representing the archival date.

3.9.0: Backups

  • Before beginning a backup with GitHub Enterprise Server Backup Utilities 3.9.0 and later, the ghe-host-check utility will now perform a preflight check on the backup host to confirm the software version and disk space requirements. For more information, see the 3.9.0 release in the github/backup-utils repository on GitHub.com.

  • GitHub Enterprise Server Backup Utilities 3.9.0 allows administrators to view the progress of backup and restoration operations on the backup host using the ghe-backup-progress utility. For more information, see "Настройка резервных копий в экземпляре."

3.9.0: Known issues

  • After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8 to 3.9 or 3.10, I/O utilization will increase, and in some cases the instance's performance will be impacted. Reduced performance is due to the database server being upgraded from MySQL 5.7 to MySQL 8.0. For more information, see "Известные проблемы с обновлением экземпляра."

  • After an administrator upgrades from GitHub Enterprise Server 3.7 or 3.8, to 3.9 or 3.10, MySQL may not start back up. For more information, see "Известные проблемы с обновлением экземпляра." [Updated: 2023-08-11]

  • In GitHub Enterprise Server 3.10 and later, the requirements for TLS security levels have changed due to an upgrade to containers in the underlying OS. After upgrading from GitHub Enterprise Server 3.9, on an instance with GitHub Actions enabled and a custom TLS certificate, users may experience disruptions with workflow runs if the TLS certificate uses weak encryption. Workflow runs will not trigger, and the following error message will appear in system logs for babeld.

    CA certificate key too weak
    

    To resolve this issue, confirm that your certificate complies with level 2 of the OpenSSL security specification. For more information, see SSL_CTX_set_security_level in the OpenSSL docs. For more information about reviewing your instance's logs, see "Сведения о системных журналах".

    If the error appears in babeld logs because your TLS certificate does not comply with level 2 of the specification, you must create and upload a new certificate with stronger security before you upgrade to GitHub Enterprise Server 3.10 or later. For more information, see "Настройка TLS." [Updated: 2023-10-26]

  • The Management Console may get stuck in a loop showing "Checking requirements..." when attempting to download an update. It is safe to proceed the update process manually via the command line.

  • After restoration of a backup created using GitHub Enterprise Server Backup Utilities 3.7.0, 3.8.0, or 3.9.0, users may not be able to sign into the instance. To fix this issue, plus a bug that was preventing secret scanning encryption keys from being backed up, upgrade your backup host to use GitHub Enterprise Server Backup Utilities 3.9.1 and generate a new full backup using ghe-backup. For more information on using an existing backup, see "Известные проблемы с резервными копиями для вашего экземпляра." [Updated: 2023-07-31]

  • After upgrading an existing instance to GitHub Enterprise Server 3.9, the Manage GitHub Enterprise Server API is unavailable. To enable the API, SSH into the instance and run the following commands.

    Shell
    sudo mkdir -p /data/ghes-manage-gateway/current
    sudo chown -R ghes-manage-gateway:ghes-manage-gateway /data/ghes-manage-gateway/current
    sudo systemctl restart ghes-manage-gateway ghes-manage-gateway-consul
    

    For more information about the Manage GitHub Enterprise Server API, see "Manage GitHub Enterprise Server." [Updated: 2023-06-22]

  • On an instance in a cluster configuration, after you upgrade nodes other than the primary MySQL node and before you upgrade the primary MySQL node, the following output may appear multiple times after you run ghe-config-apply.

    Error response from daemon: conflict: unable to delete IMAGE_ID (cannot be forced) - image is being used by running container CONTAINER_ID
    

    You can safely ignore this message.

  • Custom firewall rules are removed during the upgrade process.

  • On an instance in a high-availability configuration, passive replica nodes accept Git client requests and forward the requests to the primary node.

  • The mbind: Operation not permitted error in the /var/log/mysql/mysql.err file can be ignored. MySQL 8 does not gracefully handle when the CAP_SYS_NICE capability isn't required, and outputs an error instead of a warning.

  • When using an outbound web proxy server, the ghe-btop command may fail in some circumstances with the error "Error querying allocation: Unexpected response code: 401".

  • If an instance is configured to forward logs to a target server with TLS enabled, certificate authority (CA) bundles that a site administrator uploads using ghe-ssl-ca-certificate-install are not respected, and connections to the server fail.

  • When running ghe-config-apply, the process may stall with the message Deployment is running pending automatic promotion.

  • During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

  • If the root site administrator is locked out of the Management Console after failed login attempts, the account will not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see "Устранение неполадок с доступом к консоли управления."

  • On an instance with subdomain isolation disabled, Mermaid diagrams in the web UI display an "Unable to render rich display" error and fail to render. [Updated: 2023-08-18]

  • In rare circumstances, a small instance with both high availability and GitHub Actions configured may report that MSSQL replication is unhealthy after many upgrades with full upgrade packages. If you encounter this issue, contact GitHub Support. [Updated: 2023-08-24]

  • On an instance with GitHub Actions enabled, if shared runner groups are configured for the enterprise, the enterprise security overview page may return a 500 error. You can avoid the issue by trying one of the following workarounds.

    • Add a runner scale set to the enterprise runner group shared with the repositories.
    • Remove access to the enterprise runner group from the affected repositories or organizations.

    [Updated: 2023-09-05]

  • On an instance in a cluster configuration with high availability configured, ghe-config-apply times out while waiting for hookshot-go to start on replica application nodes. [Updated: 2023-09-21]

  • On an instance with GitHub Actions enabled, ephemeral self-hosted runners do not automatically update to the latest version. Users will need to manually update the runners to the latest version. [Updated: 2023-09-29]

  • Jobs in a deprecated queue are not processed and may accumulate over time. These jobs are reflected in the monitor dashboard's "Aqueduct queue depth" graph as an increase in resource_activity. In some cases, a buildup of unprocessed jobs can result in memory exhaustion. If you observe memory exhaustion on your instance and see a high metric for resource_activity, contact GitHub Support. [Updated: 2023-10-10]

  • When an administrator uses the -p flag with the ghe-support-bundle utility to collect data for a specific number of hours, the utility erroneously collects more logs than necessary. [Updated: 2023-10-13]

  • The settings for enabling scheduled reminders were added unintentionally to this release. Scheduled reminders are not officially supported. [Updated: 2023-10-17]

  • On an instance with GitHub Actions enabled, after an upgrade from GitHub Enterprise Server 3.8 or earlier, an internal exception could prevent successful completion of some operations, like upgrades or the configuration of new replica nodes for high availability. If this issue occurs, administrators may see the following error in /data/user/common/ghe-config.log.

    Error occurred while executing servicing step 'Clone datatier login to secondary replica' for component CopyAvailabilityGroupSqlLogins during CopyAvailabilityGroupSqlLogins: Object reference not set to an instance of an object.
    

    To resolve this issue, upgrade to the latest patch release of GitHub Enterprise Server. [Updated: 2023-12-04]

  • On an instance hosted in AWS, system time may lose synchronization with Amazon's servers after an administrator reboots the instance. [Updated 2023-11-10]

  • On an instance in a cluster configuration, restoration of a backup using ghe-restore will exit prematurely if Redis has not restarted properly. [Updated 2023-12-05]

  • On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1. [Updated 2023-12-13]

  • During periods of high traffic, interruptions in service may occur due to insufficient resource allocations for internal components. [Updated 2024-01-23]

  • Redundant messages may cause an increase in the volume of events logged in /var/log/syslog. [Updated: 2024-03-08]

3.9.0: Deprecations