PyTorch Lightning shipped credential-stealing malware to PyPI for 42 minutes. Every AI/ML team is a target.

# PyTorch Lightning shipped credential-stealing malware to PyPI for 42 minutes. Every AI/ML team is a target.

On April 30, 2026, an attacker pushed two malicious versions of lightning to PyPI — versions 2.6.2 and 2.6.3 — embedded with a credential-stealing payload that fired the moment any consumer ran import lightning. The malicious builds were live for 42 minutes before PyPI quarantined them. They were then deleted. The latest safe version is 2.6.1.

Forty-two minutes is enough. PyTorch Lightning is one of the most-used Python frameworks for training and serving deep learning models. The official repo has 31,100 stars on GitHub. The PyPI download count is in the millions per month. Every AI/ML team — whether you're a research lab, a Hugging Face shop, an enterprise data science org, or a one-engineer agentic-AI startup — almost certainly has lightning somewhere in your dependency tree, either directly or transitively through pytorch-lightning, torchmetrics, lightning-bolts, or one of dozens of other packages.

This post walks through what shipped, who is at risk, what the post-incident defensive posture looks like, and why CI runners and AI training infrastructure are the highest-value supply-chain target on the internet right now.

#What shipped in 2.6.2 and 2.6.3

Researchers at Socket and Semgrep published the deobfuscated payload analysis within hours of disclosure. The malicious build embeds a hidden _runtime directory containing a downloader stub plus an obfuscated JavaScript payload. The execution chain runs at module import — there is no opt-in, no command-line flag, no required user action beyond pip install lightning and import lightning.

The post-import flow:

1. Stub launches a child process that calls out to the attacker's C2 host
2. Server returns the second-stage payload, a JavaScript runtime executed via node if it is on the system, or a Python fallback if not
3. The payload enumerates and exfiltrates:

- Local environment variables (this is the goldmine — AWS keys, Anthropic / OpenAI API keys, HuggingFace tokens, GitHub tokens)

- Cloud SDK credential files (~/.aws/credentials, ~/.config/gcloud/, ~/.azure/)

- ~/.netrc, ~/.pypirc, ~/.npmrc for git/registry tokens

- Browser session cookies on developer workstations

1. After exfil, the payload attempts to plant itself in the user's GitHub repositories by walking the local filesystem for .git/ directories and pushing a malicious commit through cached credentials. This is the worm-style propagation that gave the broader campaign its name: Mini Shai-Hulud.

Mini Shai-Hulud is the assessed evolution of an earlier campaign that targeted SAP-related npm packages. The pattern in both cases is identical: compromise a maintainer credential, push a malicious version, exfil credentials, attempt to propagate through the victim's own repos.

#How the maintainer was compromised

Public reporting indicates the attacker did not exploit a code path in the project. They obtained PyPI maintainer credentials, cloned the legitimate source, injected the _runtime payload, and pushed builds directly to PyPI. The git source repository was not touched. This is a bypass of source control entirely — the only signal that something was wrong was the PyPI build artifact itself being different from what git tag would have produced.

For defenders, this is the most important point. Looking at GitHub for the lightning project shows nothing wrong. The malicious code never lived there. Only PyPI consumers received the payload, and only consumers who upgraded during the 42-minute window before quarantine.

The scope of the impact is anyone who ran pip install lightning or pip install -U lightning between April 30 ~14:00 UTC and ~14:42 UTC, or who had unpinned dependencies that resolved through that window automatically. That includes every CI run that started in the affected window with a fresh dependency cache. CI runners are the dominant victim profile here, not developer workstations, because CI runners install fresh dependencies on every job and run with high-value cloud credentials in the environment.

#Why AI training infrastructure is the highest-value target

The ML supply chain has a specific risk shape that magnifies attacks like this:

1. Centralization on a small set of frameworks. PyTorch, Lightning, Transformers, Datasets, Diffusers, and a handful of others are in essentially every model-training pipeline. Compromising any one of them affects a large fraction of the industry simultaneously.

2. Privileged credentials in the runtime. Training pipelines need access to the cloud accounts that store training data, the registries that hold trained checkpoints, the MLOps platforms that orchestrate jobs (Weights & Biases, MLflow, ClearML), and increasingly the LLM API keys for evaluation and synthetic data. A typical training-job environment variable list reads like a credential vault.

3. CI/CD permissive defaults. A research lab's CI is configured to make iteration fast, not to be defended. Long-lived AWS access keys mounted into the environment, persistent SSH keys for cloning private datasets, GitHub Actions tokens with broad scopes — all of it sitting in environment variables when the malicious import fires.

4. Slow patching. Research engineers pin loosely or not at all because they want the latest numerical accuracy fixes. A poisoned release reaches the audience faster than a JavaScript supply chain attack reaches a frontend team that has Renovate configured.

5. The output is the next attack vector. A compromised training pipeline can plant a backdoor in the model weights themselves — a learned trigger phrase that exfiltrates user input at inference time. We have not seen this campaign go that far, but the precedent exists.

#What you do this week

If your team uses Lightning, run the following audit immediately. The goal is to confirm you did not pull 2.6.2 or 2.6.3 during the window, and to rotate any credentials that could have been exposed if you did.

Step 1: Check pinned versions. Search every requirements.txt, pyproject.toml, poetry.lock, Pipfile.lock, and requirements-*.txt across every repo for the strings lightning==2.6.2, lightning==2.6.3, lightning>=2.6.2, lightning~=2.6, and unbounded lightning references. Anything unbounded that resolved during April 30 14:00–14:42 UTC is suspect.

Step 2: Check CI run history. For every CI provider you use (GitHub Actions, GitLab CI, Buildkite, CircleCI, Jenkins), search runs that started in the affected window. If any of them ran pip install and resolved a fresh dependency tree during that window, treat that runner's credentials as compromised.

Step 3: Check developer workstations. Less likely than CI, but still in scope. If a developer ran pip install -U lightning in the affected window, their workstation credentials are exposed.

Step 4: Rotate. For any compromised environment, rotate every credential that was reachable from environment variables: AWS access keys, GCP service account keys, Azure credentials, HuggingFace tokens, OpenAI / Anthropic API keys, GitHub PATs, GitLab tokens, deploy keys, MLflow / W&B / Neptune tokens.

Step 5: Audit GitHub repos for unauthorized commits. The propagation step attempted to push commits via cached credentials. Check git log on every repo that was open on a compromised workstation in the affected window. Look for commits authored as you that you do not remember making.

Step 6: Pin lightning to 2.6.1 explicitly. Future-proofing. If a similar campaign hits a future version, you want the failure mode to be "build breaks because version not pinned" rather than "silent compromise."

#Detection signal you can deploy now

If you operate a SOC, the import-time C2 callback is detectable. The malicious payload reaches out to a small number of specific hosts during the exfiltration phase. Egress logs from CI runners that show outbound traffic to non-PyPI, non-cloud, non-HuggingFace, non-corporate destinations during dependency installation are candidates for review.

For development workstations, EDR rules that flag python processes spawning node or curl to internet destinations during package installation will catch this and similar campaigns. Most EDRs do not ship this rule by default. Worth adding.

#The broader Mini Shai-Hulud lesson

This is the third significant Python supply-chain compromise in twelve months. Each one targeted a different ecosystem corner — typosquats, dependency confusion, and now direct maintainer-credential compromise. The pattern is converging: attackers target maintainer accounts, push for a short window, exfil credentials, propagate through victim repos.

The defender lessons are uncomfortable because they require process changes that slow down CI:

• Pin direct dependencies and use a lockfile. Stop letting unbounded lightning resolve to whatever PyPI returns at install time. The 42-minute window only catches teams with floating versions.
• Run CI in least-privilege environments. A training pipeline does not need long-lived AWS root keys. Switch to OIDC federation. The blast radius of a poisoned dependency is whatever the runner's role allows.
• Separate credential scope by job. A test job does not need the same credentials as a deploy job. A scratch environment does not need production credentials.
• Detect outbound traffic from CI. Most CI runners only need to talk to your registry, your cloud provider, and a small allowlist of services. Anything else is a candidate for review.

The supply chain is only going to get more contested. The cost of a 42-minute window is now well-understood by the attackers running these campaigns. They will be back, and the next one will not announce itself.

#How Valtik helps

We audit AI/ML supply chains, CI/CD pipelines, and dependency hygiene. If you are running training infrastructure or you ship models to production, we can map your exposure to attacks like this and harden the seams that fail in real campaigns. Free external check at valtikstudios.com/free-check, or contact contact@valtikstudios.com to scope a deeper audit.