Last week the world was quite busy responding to and talking about the Black October in information security. Even the German national television gave many minutes of screen time to the PHP vulnerability. Fortunately, I was only affected by the PHP issue through one NextCloud deployment, the GitLab CE CVSS 10.0 issue and the eMudhra Intermediate CA Compromise.

Well, typing this out, I have to say that’s quite a bit for one week. Oh, and I even forgot about that Node.js issue. Not that big of a deal though. I don’t run any production Node servers. It’s just used to build some frontend artefacts.

The Discovery

I was about to look up some package details when PyPI helpfully showed the following warning as a banner:

Security notice: Upgrade certifi, pip, and uv in response to CA certificate incident (see more)

Oops. My usual process of executing pip-audit during monthly update runs would have surfaced that issue anyway. But I would have liked some more timely notice of that.

The Assessment

Does that issue apply to me? Do I even need to care? Yes. You should probably care, too.

But not frantically.

Every outgoing HTTP request using TLS (going through Python) with certificates checked against the certifi package is vulnerable to Man-in-the-Middle attacks. That includes package installations via pip. When using uv, you are also vulnerable because they ship their own certificate bundle. That’s bad, but likely not critical for many production systems, if the network path can be trusted.

Installing packages on coffee shop Wi-Fi probably isn’t the wisest call.¹ But services talking within your deployment without crossing the public internet are most likely fine. If your application does not even make HTTP calls, you are safe (from that issue at least).

The Impact

For me and the systems developed and operated by Kition Software, the compromise impacts more areas than I’d like:

• local machines through their natively installed Python, pyenv, virtual environments and browsers
• Python processes running in production
• Python utilised in CI and local development
• host systems running Debian
• one lonesome Microsoft Windows Server 2019²

The Validation

Before starting to fix things, I need to be able to verify whether a fix was successful. I used four strategies for that.

1. Browsers were easiest to verify by visiting the eMudhra Intermediate CA Compromise website. If your browsers accept certificates issued by the compromised intermediates, the checks on that site will fail.
2. That same site published a URL for verifying the certificate checks from the CLI, e.g. via curl
   vulnerable systemconsole

   $ curl https://dv-g3b-emudhra.we-have-a-problem.com
   Your client successfully connected to the site. That's bad!
   

   patched systemconsole

   $ curl https://dv-g3b-emudhra.we-have-a-problem.com
   curl: (60) SSL certificate problem: unable to get local issuer certificate
   More details here: https://curl.se/docs/sslcerts.html
   
   curl failed to verify the legitimacy of the server and therefore could not
   establish a secure connection to it. To learn more about this situation and
   how to fix it, please visit the webpage mentioned above.
   

3. Checking the versions of pip and certifi is trivial enough
   vulnerable setupconsole

   $ python -m pip --version
   pip 26.1.1 from /tmp/demo/.venv/lib/python3.14/site-packages/pip (python 3.14)
   $ python -c "import certifi; print(certifi.__version__)"
   2026.07.22
   

   patched setupconsole

   $ python -m pip --version
   pip 26.2.2 from /tmp/demo/.venv/lib/python3.15/site-packages/pip (python 3.15)
   $ python -c "import certifi; print(certifi.__version__)"
   2026.10.23
   

4. Querying the same URL I used in the CLI check, I am able to see whether outgoing TLS connections from my Python processes behave as expected.
   python

   import requests
   try:
       requests.get("https://dv-g3b-emudhra.we-have-a-problem.com")
       print("VULNERABLE: compromised CA still trusted")
   except requests.exceptions.SSLError:
       print("OK: compromised CA rejected")

The Mitigation

Knowing what to fix and how to test the before and after behaviour, I set out on the actual mitigation. In reality though, I was figuring out the validation and mitigation at the same time.

To locate all Python and pip installations on my machine, I asked my Clanker to generate a find command to point to them. I had A LOT of Python installations scattered across virtual environments, local installations and macOS itself. I uninstalled as many of them as I could to spare as much mitigation effort as possible. In practice that meant (1) deleting all virtual environments, (2) removing all pyenv installed Pythons, (3) uninstalling those brew-based Python installations that have no packages depending on them and (4) deleting all virtual environments.

The only Python I didn’t dare to touch is the Python 3.9 shipped with the Xcode command line utilities by Apple. Who knows what might break down the line by updating packages there. That version has been out of security support for a year and active support ended over four years ago. Seriously Apple, why?! I just made sure that executing python and python3 in my CLI resolves to my global default Python set through pyenv.

Updating my Debian hosts was trivially achieved by running the typical apt-get update and apt-get upgrade, while still validating through the methods above that the updates actually fix the issue. I was worried about Windows, where I’m quite out of my depth, but the Microsoft security updates handled the compromise just fine.

The most tedious part of the whole mitigation was updating and validating the fix in every local development, CI and production setup for every project. That includes libraries, too… I followed the same steps in each of them:

• running the validation and version checks to understand the current effect of the compromise on the project
• updating Python in the CI pipelines and through .python-version, which is automatically picked up by my build processes and Docker Images via exporting an environment variable and using that in the Dockerfile.
  dockerfile

  ARG PYTHON_VERSION=latest
  FROM python:${PYTHON_VERSION} AS build
  ...
  FROM python:${PYTHON_VERSION}

• One project required adding certifi to the requirements because a transitive dependency pulled in a vulnerable version.
• re-initialise the local development environment
• running the validations again
• if all of that succeeded and the project needs to establish TLS connections, do a production rollout. Remember your project might not even call out via TLS.

The Learning

This certificate compromise served as a great training exercise that required upgrades throughout the whole software stack. It made clear that my (at least) monthly updates of all systems and the automated processes around that are essential. Without them, I wouldn’t have the routine and tooling to update and validate changes like these.

In addition, I would like to establish a process that draws my attention more quickly to known vulnerabilities in my core stack. Maybe a CI job that runs daily or at least weekly and executes pip-audit?

And, thank you, PyPI, for letting us know about that issue!