This weekend, GitHub security announced that they are investigating malicious activity, which appears to indicate that credentials associated with Travis-CI and Heroku GitHub integrations were compromised, and are actively used by threat actors to access private information of GitHub tenants which have installed these integrations.
The fact that these applications are very common, combined with the fact that the source from which the information was leaked is not clear, calls for a series of actions from defenders, to make sure the effect of this breach on their environments is minimal.
As we have been assisting many of our clients and partners in coping with the incident, it was important for us to share some of our insights and understandings to assist all defenders currently dealing with this issue.
This writeup covers the following:
- What happened
- Who is potentially affected
- Measures to take to respond to this incident
- Measure to take to prepare for future such incidents
Let’s dive into the details:
What happened
GitHub have shared detailed information about the incident in a blog released by their Security group: https://github.blog/2022-04-15-security-alert-stolen-oauth-user-tokens/
The primary details –
- On April 12, GitHub Security began an investigation that uncovered evidence that an attacker abused stolen OAuth user tokens issued to two third-party OAuth integrators, Heroku and Travis-CI.
- The tokens are believed to have been stolen from a source outside of GitHub, as GitHub do not store these tokens in their original, usable formats.
- It appears that the compromised Heroku and Travis-CI OAuth user tokens were abused to download private repositories belonging to dozens of victim organizations that were using these apps.
- The investigation began after GitHub Security identified unauthorized access to their npm production infrastructure using a compromised AWS API key. The key was obtained by the attacker when they downloaded a set of private npm repositories using a stolen OAuth token from one of the two affected third-party OAuth applications.
- At this point, while npm infrastructure was indeed accessed, GitHub believes the attacker did not modify any packages or gain access to any user account data or credentials.
- GitHub contacted Heroku and Travis-CI to request that they initiate their own security investigations and revoke all OAuth user tokens associated with the affected applications.
- GitHub is currently working to identify and notify all of the known-affected victim users and organizations that discovered through their analysis across GitHub.com.
Who is potentially affected
As the source from which the tokens were compromised has not been published, the full scope of potentially affected organizations is still unclear.
Organizations using Github.com, which have integrations with either Travis-CI or Heroku, should most certainly evaluate the action items and measures detailed below. However, since it is not certain that the scope of compromise is limited just to organizations using Github.com, we recommend all organizations with Travis-CI or Heroku integrations in their SCM to evaluate these measures as well.
It is important to mention that not having Travis-CI or Heroku widely/actively used does not necessarily mean your organization is not affected.
As part of our collaboration with clients and partners around this incident, we have discovered multiple organizations which have set up these integrations as part of a trial/POC and are no longer using them. However – the integration is still active, and the credentials associated with this integration are still valid – meaning they are at risk.
Measures to take to respond to this incident
GitHub have communicated that they will proactively reach out to all organizations their analysis identifies as victims within the upcoming days. However, given the potential impact, we recommend all organizations to take a proactive approach in responding to this incident and carry out the following measures:
- Search for the existence of integrations with Travis-CI and Heroku within your environment. The search should include both Marketplace apps as well as OAuth apps installed in the organization. In GitHub, all relevant information can be found under: Settings → Third-party access and Settings → GitHub Apps.
- Scan all repositories which the Heroku and Travis-CI integrations had permission to for secrets and credentials. If found, it is recommended to rotate these credentials.
- In case stale/inactive Heroku and Travis-CI integration exist, remove them immediately. Ensure both Marketplace apps and OAuth apps are removed and also denied from future installation.
- In case of active Heroku and Travis-CI integrations, consider recreating the integration to revoke the potentially compromised tokens associated with the existing integration.
- Ensure the Travis-CI and Heroku apps are scoped to the exact set of permissions and repositories these integrations are relevant for.
- Monitor your Github.com organization audit logs and user account security logs for unexpected or anomalous activity associated with Travis-CI / Heroku credentials.
Measures to take to prepare for future such incidents
Ensure capabilities to gain continuous visibility around 3rd party services in your CI/CD environment exist. Retain the following information for all methods of 3rd party integrations across your entire CI/CD environment – all the way from code to deployment (SCM,CI, Artifact repository, container registry, CD):
- Method of integration of the 3rd party service
- Granular detail about permission granted to 3rd party service
- Granular details about permission actively used by 3rd party service
Refer to “Ungoverned Usage of 3rd Party Services” in the “Top 10 CI/CD Security Risks” project for additional details about protecting your organization from 3rd party service compromise.
- For GitHub.com organizations, Ensure “OAuth App access restrictions” are enabled to prevent unauthorized installation of 3rd party integrations in the organization.Use visibility over 3rd party usage to periodically remove/reduce the permission for all inactive/stale/overly permissive 3rd party integrations.
- Ensure all relevant visibility artifacts (audit log, security log) are enabled and retained to provide optimal detection and investigation capabilities for future incidents.
- Continuously scan all data sources potentially accessible by 3rd party integrations (e.g. SCM repos, CI pipeline configuration, CI build logs) for credentials and secrets stored in clear text. Rotate all detected credentials and wherever possible refrain from writing clear text credentials/secrets to these artifacts.
