Railway's October 21 incident: what's confirmed and what to rotate first

Most of my own apps run on Railway, so I spent October 21 and 22 doing what Railway asked every customer to do: rotating every secret. This post collects what Railway has confirmed, what has been reported but not confirmed, and the rotation order I’d use. Every claim links to its source.

Facts here are current as of Railway’s status update at 19:06 UTC on October 22, 2026. Railway has said it will publish a post-mortem (TechCrunch), and I’ll update this post when it does.

The short version

• Railway says an intruder may have accessed encrypted environment variables and, for some of them, obtained the decrypted contents.
• Railway asks all users to rotate every environment variable and private registry credential, and to check their service logs.
• Railway has invalidated all Railway API tokens and OAuth sessions, and rotated its bucket credentials.
• Builds were paused. Enterprise and Pro builds are back. Hobby, Free and Trial were still waiting at the last update.
• One researcher reports about $371,000 drained from crypto wallets whose keys were stored on Railway. Railway has not confirmed this.

What Railway has confirmed

Railway’s disclosure, published October 21, says it detected unusual activity on “systems that store customer environment variables”, and that an unauthorized party “may have accessed encrypted environment variables and, for some of them, obtained the decrypted contents” (Railway).

Railway also said:

• Early analysis suggests a subset of users is affected, but the advice applies to everyone as a precaution. Railway will email affected customers directly.
• It asked upstream providers to proactively revoke keys.
• It believes the incident is part of “a series of attacks recently targeting platform companies with the aim of compromising AI inference providers.” No attacker has been named.

Railway has not published a root cause, the number of affected accounts, or when the access began.

Status page timeline

From Railway’s status page, all times UTC:

When	What Railway posted
Oct 21, 07:24	Investigating an issue
Oct 21, 09:28	Dashboard, API and CLI affected. New builds paused in all regions
Oct 21, 20:19	Invalidating OAuth sessions, so users have to log in again
Oct 21, 21:59	Security notice. Users asked to revoke all environment variables stored in Railway
Oct 21, 22:10	Rotating all Railway bucket credentials
Oct 22, 01:22	13 guides published for rotating database credentials and checking provider usage
Oct 22, 03:32	Builds available for Enterprise workspaces
Oct 22, 04:17	All Railway API tokens invalidated
Oct 22, 06:29	Builds available for Pro workspaces
Oct 22, 13:53	Deploy queue cleared for Pro and Enterprise
Oct 22, 17:00	API fully restored. Tokens issued before the incident stay invalid
Oct 22, 19:06	New signups paused. Hobby builds next, then Free and Trial

What’s reported but not confirmed

Taylor Monahan (@tayvano_), a blockchain security researcher, tracked wallet drains she attributes to private keys stored on Railway. Her count is roughly $371,000 across more than 500 transactions, 200 addresses and at least 19 EVM chains, collected by a single address. She puts the first known theft at 19:36 UTC on October 21, about two hours before Railway’s security notice (crypto.news, The Hacker News).

These figures come from one researcher’s on-chain analysis. Railway has not confirmed the losses or the link to its incident.

Why this cleanup is harder than usual

Rotating a key usually takes four steps. Create a new key at the provider, set it in Railway, redeploy, and then revoke the old one. With builds paused, the third step was blocked, so the new value couldn’t reach running services.

That leaves a real choice for each key. Revoke the old key now and accept that the service breaks until you can deploy, or keep it alive until the deploy goes through and accept that someone else may be using it. There’s no right answer for every key, which is why the order below matters.

Also note that Railway asked providers to revoke keys proactively. If a service started failing authentication on its own, the provider may already have killed that key.

A rotation order that holds up

1. Take an inventory first

List every variable in every project and environment, including preview environments and shared variables. You can’t rotate what you haven’t found, and the inventory tells you where each key is used, so you update all of its copies. The free tool further down does this without writing values to disk. If you export variables to files instead, delete those files when you’re done.

2. Rank by what an attacker can do with each key

1. Wallet private keys. Move the funds to a new wallet now. A private key can’t be rotated.
2. Cloud and source control credentials. AWS, Google Cloud, GitHub tokens and deploy keys. These let someone reach everything else. Revoke them now, even if it causes downtime.
3. Payment keys. Stripe secret keys and anything else that moves money.
4. Database credentials. Railway published guides for Postgres, MySQL, MongoDB and Redis.
5. AI provider keys. OpenAI, Anthropic, OpenRouter, xAI and similar. Railway’s own statement says the campaign is aimed at AI inference providers.
6. Everything else, including sealed variables. GitGuardian points out that the disclosure doesn’t say whether sealed variables were in scope, so rotate them too (GitGuardian).
7. Private registry credentials, which Railway called out separately.

3. Rotate each key in a safe order

For wallets, move the funds. For cloud and source control credentials, revoke first and redeploy when you can. For the rest, create the replacement at the provider, set it in Railway, redeploy, check the app works, then revoke the old key. GitGuardian recommends the same provider-first order.

4. Check what the old keys did

Railway’s guides cover checking for unexpected usage at OpenRouter, Grok, Anthropic, OpenAI and Stripe, and unusual access at GitHub, AWS and Google Cloud. Look for API usage you don’t recognize, keys or users you didn’t create, and spend spikes since October 21.

5. Replace your Railway API tokens

Every Railway API token issued before the incident is invalid. Scripts, CI jobs and integrations that call Railway’s API need new tokens.

Using Keyhouse Exposure Check for the inventory

I built a small free tool for step 1. Keyhouse Exposure Check is MIT-licensed and runs on your machine.

npx keyhouse-check railway

Create a fresh token at railway.com/account/tokens first. It lists projects, all environments including previews, shared variables and each service’s variables, then groups the credentials it finds by provider with a rotation checklist and a link to each provider’s key page.

What it doesn’t do matters as much:

• Secret values stay in memory. They are never uploaded, logged or written to the report.
• It sends only read queries to Railway and never calls the credential providers or any Keyhouse server.
• It doesn’t rotate anything, prove a key was stolen, or check whether a key still works.
• It reads current values, not past deployments, and can miss secrets in unusual formats.

Revoke the scan token when you’re finished. Source and full details are in the README.

Same day, separate incident: LiteLLM 1.107.0

On the same day, LiteLLM disclosed that its 1.107.0 release had been compromised. Nobody has linked it to Railway, but if you run LiteLLM, possibly on Railway itself, check this too.

According to LiteLLM’s own write-up:

• litellm==1.107.0 on PyPI was malicious from about 11:39 to 13:23 UTC on October 21.
• The Docker tags v1.107.0, latest, main-latest and main-stable resolved to the malicious image from 11:46 UTC until 14:20 UTC on Docker Hub, and until 13:51 UTC on GHCR.
• The build stole credentials and installed cook.pth and sitecustomize.py, which run on every Python start, whether or not you import LiteLLM.
• The source tag on GitHub is clean, and the private enterprise release was not affected.

LiteLLM’s advice is specific. Don’t start the cleanup with pip install, because any Python start runs the payload again. Delete the payload files and kill the background process from the shell, rebuild from scratch on litellm==1.106.2, and assume the theft succeeded if the host had outbound internet. Then rotate everything the host could reach, including every provider key in the proxy database, master and virtual keys, and the DATABASE_URL password.

What I’m changing for next time

Some of this is in your hands today, without any new tool:

• Give every app and environment its own key. Then one leak means one revocation, and the provider’s usage page tells you which app it came from.
• Set spend limits at every provider that offers them.
• Keep wallet private keys out of environment variables.
• Keep a current list of where every key lives. It’s the step that took me longest.

The bigger issue is structural. Any key stored in a host’s environment variables is exposed when the host is. That’s why I’m building Keyhouse. Hosts would hold only proxy keys, so a cleanup like this one becomes rotating proxy keys in one place while nothing changes at OpenAI or Stripe. It isn’t built yet. If that’s something you’d use, join the waitlist.

Railway’s users are the people I’m building for, and I hope this list saves you a few hours. If I got something wrong, tell me and I’ll fix it.

Quentin