Attackers are always on the lookout to gain access to credentials, which are a critical asset to protect and are widespread throughout the organization. This is highly relevant in CI/CD systems, which are a vital part of the engineering ecosystem. For each pipeline execution, credentials are decrypted and used, leaving them exposed to anyone with the relevant permissions.
As identifying and blocking malicious credential usage in real time is challenging, the security measures that are implemented to secure credentials are often the last line of defense against attacks. This means it’s very important to implement appropriate security controls to ensure credential hygiene in your CI/CD ecosystem.
Inherently, CI/CD systems interact with sensitive systems such as production, source control management (SCM) and back-office. CI/CD pipelines access data from these systems, and create and modify their resources. Therefore, they usually require high permissions on these systems, which are granted to the credentials used by the pipelines.
The risk of Insufficient Credential Hygiene is defined in the “Top 10 CI/CD Security Risks” initiative as “…an attacker’s ability to obtain and use various secrets and tokens spread throughout the pipeline due to flaws having to do with access controls around the credentials, insecure secret management and overly permissive credentials”.
When attackers are able to compromise these credentials, they can then use them to access and attack all of the aforementioned systems.
For example, a common method for attackers to obtain credentials is by poisoning the CI/CD pipeline, and exfiltrating credentials used during the build process. This is known as a Poisoned Pipeline Execution (PPE) attack which is defined as “…the ability of an attacker with access to source control systems – and without access to the build environment, to manipulate the build process by injecting malicious code/commands into the build pipeline configuration, essentially ‘poisoning’ the pipeline and running malicious code as part of the build process.”. PPE is also part of the “Top 10 CI/CD security risks”.
To help you prevent credential theft and secure your environment, we will present three common credential hygiene issues, and discuss the strengths and weaknesses of four of the most popular CI vendors: Jenkins, GitHub Actions, CircleCI and GitLab CI/CD around these issues.
The three issues are:
Unrotated Static Credentials
CI/CD pipelines use cloud provider credentials to deploy new services and artifacts, send notifications, store logs and more. Until recently, most commonly used CI solutions included authentication through cloud providers by storing long-term credentials in the CI system. Usage of long-term credentials poses a risk, because if attackers get hold of them they can potentially use them forever, unless the credentials are manually revoked or the attackers are detected. Furthermore, this method is very limited in an organization’s capability to restrict the usage of this credential to a specific source (unlike other methods – which we describe shortly – in which even if the credentials are compromised – they cannot be used outside of their designated environment).
CI/CD vendors have recently implemented an efficient method of reducing the risk of credential theft through OpenID Connect (OIDC).
How does OIDC work? Basically, you configure your cloud provider to trust your CI system and when your job runs, the CI system generates a JWT token for the job, with claims such as organization name and repository name related to the job. The job presents the token to the cloud provider, and once the cloud provider authenticates the token, it provides the job with an access token based on the claims. The access token is then used for authentication to the cloud provider.
One of the main benefits of OIDC is that the issued token is short lived, preventing long term malicious access to the cloud provider.
Let’s compare the different vendor support for OIDC:
|Vendor||Support for OIDC|
|Jenkins||The OpenID Connect Provider plugin was released in April 2022 (not yet in wide use).|
|GitLab CI/CD||The feature is in alpha stage at the time of writing.|
Note that besides OIDC, another solution for providing short-lived credentials is to run builds on self-hosted agents that inherently implement the usage of temporary credentials, like EC2 instances that are assigned with an instance role.
The downside of this solution is that it’s challenging and expensive to maintain. For example, if you have two pipelines that require different permissions, you’d need two different EC2 instances configured with roles specific to each instance.
Overly Accessible Credentials
To illustrate the lack of credential access control, let’s imagine a scenario where a developer’s account is compromised and the attacker gains “write” access to some repositories. By executing a PPE attack through the repository, the attacker can access credentials by either modifying the pipeline manifest (Direct-PPE) or by modifying code that runs during the build (Indirect-PPE).
If proper credential access controls are in place, even if the attackers have access to a repository, they would be limited to the amount of credentials that they are able to access.
Credential access controls are mechanisms that limit access to credentials by scope, such as repository, branch, user and pipeline step. For example, credentials that are configured for one branch will not be able to be used by a build running on another branch.
The most permissive access configuration is credentials that can be accessed globally by all organization users or by any pipeline. We recommend limiting the access to each credential based on the least privilege principle. There are generally four options to limit access to credentials:
- Option 1: Limit access to users – the build can access the credentials only if specific users triggered the build
- Option 2: Limit access to a repository – only builds for this repository can access the credentials
- Option 3: Limit access to a branch – only builds of specific branches can access the credentials
- Option 4: Limit access to step – only specific steps inside the build can access the credentials. In contrast to the other options, this option intends to limit access to secrets between the different parts of the pipeline, not to limit the users’ permissions to access secrets
The options above can be used to protect against two kinds of scenarios:
- An attacker can change the pipeline manifest without review and execute code in the CI that will access stored credentials through the build, by executing a Direct-PPE attack. This risk can be reduced by implementing the first three options above. While options #1 (users) and #2 (repository) limit the access of an attacker with a compromised source control account, they aren’t restrictive enough to prevent unreviewed access to credentials, which can only be achieved by implementing option #3 (branch), which leverages additional security mechanisms such as branch protection rules that can enforce a review to any push of code.
- An attacker can run malicious code inside the build by modifying files stored in the repository that are run by the build (Indirect-PPE). This risk can be reduced by applying options #1 (users), #3 (branch) and #4 (step).
- An attacker can run malicious code inside the build by compromising a 3rd party tool used by the build (see Top 10’s Ungoverned Usage of 3rd Party Services). This risk can be reduced by applying option #4 (step).
The following table compares between the access controls that each vendor provides, including vendor limitations.
|GitHub Actions||Available when using OIDC credentials||Available||Available when using OIDC credentials or environments (in public repos or Enterprise plan)||Not available|
|CircleCI||Contexts to a group||Available||Not available||Available|
|Jenkins||Available||Available using multibranch pipeline or folder credentials*||Available when configuring a pipeline to run for a specific branch||Available|
|GitLab CI/CD**||Available when using OIDC credentials (alpha)||Available||Available for protected branches||Available|
** It’s important to note that GitLab allows you to retrieve the clear-text value of the stored credentials using its API, which is not a best-practice security wise.
A few interesting insights:
- Jenkins, being an open source solution often considered to be less modern, is the only one that supports all of the access control features, while most modern CIs do not
- As previously mentioned, Limiting access to a specific branch is the only control which can require reviewer approval using branch protection rules, providing an extra layer of security. This feature is not fully available in all CIs
Credentials Exposed in Console Logs
Console logs are usually broadly accessible in organizations and are even visible to the public in certain projects. Credentials can be accidentally exposed through console logs for many reasons, such as credentials used in commands or printed in debug logs.
The first method to prevent credentials being exposed is by masking credentials that are printed to the console log. This feature is offered by all common CIs. Generally, vendors attempt to identify and mask environment variables populated with credentials from the CI credential store.
It’s important to mention that credential masking is intended to prevent accidental exposure of credentials. Intentional exposure of credentials by an attacker would not be prevented by these features, because credentials can be printed in the console log in many forms, for example – by simple encoding.
The second way to prevent credentials from being exposed in console logs is by masking custom-configured strings. This capability masks strings that aren’t necessarily used as credentials, and can therefore be missed by automatic masking mechanisms. It allows users to mask specific command outputs, preventing credentials from being exposed in the log.
In the table below we compare the masking features provided by the vendors.
|Vendor||Credential Masking||Custom Masking|
|GitHub Actions||Available, turned on by default||Available|
|CircleCI||Available, turned on by default||Not Available|
|Jenkins||Available, turned on by default||Available using the Mask Passwords plugin|
|GitLab CI/CD||Available, not turned on by default||Not Available|
For too long, credential hygiene for CI/CD systems was not given the attention it deserves, providing attackers with a convenient method of gaining access to credentials. In this article we have described common credential hygiene issues and the ways in which you can protect your environment.
We have also provided an overview and comparison of the solutions offered by the main vendors. Each vendor has their own strengths and weaknesses, although no solution provides all the capabilities required to fully protect their credentials.
Whether you’re considering which CI/CD vendor you should choose security wise, or wondering if your current vendor provides you with capabilities to achieve credential hygiene for your CI/CD ecosystem, we hope this post helps you in deciding the next moves to defend your crown jewels.
At Cider Security, we assist many organizations by mapping their CI/CD ecosystem and improving their security posture including credential hygiene.