In the past few weeks, our world infrastructure has been under attack. The attack is very simple: Find a dependency package most of the world is using, hack into the developer’s account and update to a new version with some malware inside. Why was this possible? How can we protect ourselves? What are the potential problems and how can we address them?
Why is this possible?
When installing a package in Node.js, NPM (node package manager) automatically downloads all the package dependencies. Those dependencies then download additional dependencies, and so on, and so on, until all the needed dependencies have been acquired/downloaded.
Because keeping all of these dependencies up to date would be difficult and time consuming, NPM gives developers the option of automatically updating to new major/minor versions using special characters prefixed to the version. However, this option opens up the possibility that new dependencies will be upgraded automatically in the next build, which makes it possible to attack dependency owners by adding malicious content in the form of a new updated version.
Each version of a package has 3 sections: Major, Minor and Patch. This means, for example, that version 4.0.12 means:
- Major — 4
- Minor — 0
- Patch — 12
When defining a version we can give different kinds of instructions, which will determine how the dependency will be upgraded. For example, “5.3.1” means “Use the exact version 5.3.1,” while simply writing “*” means “always use the latest version.” Other, intermediate options include:
- 4.x =>Use all upgrades included in major version 4
- >=4 => Major version 4 and above
- ~4.1.2 => Major version 4, Minor version 1, Upgrade only to patch versions 2 and above
- ^4.1.2 => Major version 4, Upgrade to any Minor version above 1 and patch version 2 and above
To get a better feel of how versioning works, you can try out the semgrep version calculator here: https://semver.npmjs.com
The default option when installing a new dependency is the last example shown on this list (^4.1.2) which means that installing automatically upgrades the package to the next minor and patch versions.
Because of this, when hackers gained access to an account that has permissions to update the COA and RC packages and added malware to the package, any new installations downloading the latest “COA” or “RC” version automatically got this compromised version. Luckily for the world, this particular attack was not as stealthy or as comprehensive as it could have been, causing installations in pipelines to begin failing worldwide. This triggered an investigation by npm, and the malwared versions were quickly removed.
This time we were lucky, but next time a more targeted attack, running in stealth under the radar could be performed by taking advantage of this same weakness. This means that one mistake in a commonly used dependency or a calculated malicious act could leave most companies in the world compromised.
How can we protect ourselves?
One important first step is using “npm ci” instead of “npm install.” Both are ways to install, but npm ci is much more strict in its dependency fetching policy. When we use “npm install,” we are telling the system to download whichever new packages the developer requested in the package.json file, including any automatic updates defined by the default semantic versioning.
Npm ci, in contrast, requires the developer to upload a package-lock.json, which contains all exact versions of dependencies and sub-dependencies, thus telling the package manager exactly which dependencies to download. The developer then needs to periodically update the package-lock to reflect valid updates in the dependencies required. When using “npm ci” we are telling the system to download only the specific packages stated in the package-lock.json.
A quick search on github finds 11,294,073 code instances of “npm install” versus 279,920 instances of “npm ci” which means the process in automatic scripts that downloads new repositories without thinking is 40 times more common than the stricter, safer one.
In order to create trusted/reproducible builds we have to use “npm ci” in all stages of the build systems (qa, testing, security, production) to be sure we are aligned with the same versions and nothing changed during the progression through the CICD pipeline.
Unfortunately npm ci comes with problems of its own…
(a) Updating packages becomes a more complex process: Using “npm ci” introduces a new problem, since for each change in the dependency tree the developer would need to update dependency version changes, validate them and upload the newly created package-lock.json.
Adopting this protocol can be hard for large organizations, but they can educate their developers to continually update the package-lock file or use solutions such as “dependabot” or “renovate” which can assist with creating PR to help.
(b) Developers’ machines can still be vulnerable: Updating dependencies on the developers’ machines moves the vulnerability from the CI environments to the developers’ machines. Anyone who needs to run “npm install”, will be vulnerable and can be targeted for dependency attacks on their local machine.
To minimize this risk, organizations can add some preventive measures when developers run “npm install” by proxying their package registries and working only with versions which are above a certain age, for instance only allowing packages older than 1 week.
Furthermore, if they do have the need to run and update in the CI environments, organizations can make sure their systems are protected by preventing access to secrets, preventing access to sensitive systems, hardening pods and killing pods at first sight of suspicious behaviour.
(c) The latest version of npm ci does not verify the lock file correctly:
We recently discovered that NPM 7 does not properly validate the package-lock as expected:
The npm ci command in npm 7.x and 8.x through 8.1.3 proceeds with an installation even if dependency information in package-lock.json differs from package.json. This behavior is inconsistent with the documentation, and makes it easier for attackers to install malware that was supposed to have been blocked by an exact version match requirement in package-lock.json.
https://nvd.nist.gov/vuln/detail/CVE-2021-43616
When writing this blog and researching best practices to protect yourselves we found out that in recent versions of npm, the “npm ci” command does not work as written in the documentation and does not verify the package-lock properly
What should security/devops teams do now?
Check if you have been attacked
- Check if you have “COA” or “RC” versions in the package-lock versions
Vulnerable Coa versions: 2.0.3, 2.0.4, 2.1.1, 2.1.3, 3.0.1, 3.1.3
Vulnerable RC versions: 1.2.9, 1.3.9, 2.3.9 - Verify Build logs
Make sure the CI build logs did not execute any malicious code - Confirm that vulnerable versions did not go to production
If you don’t support reproducible builds (Using npm install in CI), you need to go to container/server, which uses a repo with the affected package and confirm that they don’t have these versions and/or the malware signatures. - Check the status of developer machines
Check that your developers don’t have these malware signatures on their machines.
Employ preventative measures
- Use NPM v6 in your CI as a workaround.
- Force usage of package-lock.json in your CI by running “npm ci — ignore-scripts” in your dockerfiles and build scripts.
- Validate that all repos have a package-lock.json submitted to the repo.
- We have created a quick program to check for instances of insecure npm usage and identify where the less secure “npm install” is being used.
1. Download CIMatch — “go get -v github.com/cider-rnd/cimatch”.
2. Run “cimatch –human” in all your repos and ci configurations
3. Add “cimatch” to your security checks when scanning for new dockerfiles and jenkinfiles (More rules to come).
