Railway Says Stolen Customer AI Keys Led to $740,000 in Estimated Unauthorized Usage Charges

AI provider keys that customers stored on Railway were stolen in an October breach and used to commit an estimated $740,000 in unauthorized spending, the platform-as-a-service provider said yesterday. It has promised to pay those charges itself when an AI provider will not refund them.

The company builds and runs apps for its customers and stores their secrets, such as API keys and database credentials, as environment variables. The attacker decrypted variables in at most 1.85% of workspaces, about 100,000, and its database queries looked for AI provider keys.

In an October 26 update, the company told customers asking about refunds for unauthorized AI spend that "Railway is unable to refund customers for spend performed with compromised API keys." It encouraged them to contact the AI providers directly and offered to escalate their requests.

According to the company's incident report, the decrypted data also included linked-account tokens for 1,852 users: BYOK agent keys and GitHub OAuth tokens. All were reset, each user was notified, and no logins to customer accounts using stolen credentials were found.

In addition to AI keys, one of the attacker's database queries matched cloud, database, and payment service credentials. The company's forensics did not show any customer databases being accessed, and there was no indication that customers' code repositories were downloaded.

The attacker also read the stored histories of 353 builds, including the variables and secrets passed to those builds.

The company's logs show which environments were exposed but not which variables, so the figure of about 100,000 workspaces is an upper limit. It emailed customers it judged to be at high risk.

Any data the attacker copied but did not decrypt can no longer be decrypted, because the credential the attacker used has been disabled, according to the company.

What Railway Customers Can Do

Refund terms differ by AI provider, according to the company's incident report:

• AWS Bedrock: AWS's existing refund policy covers fraud, and its account teams have started refunds.
• Google Gemini: Google's existing refund policy covers fraud, and its account teams have started refunds.
• OpenAI: Refunds are reviewed on a case-by-case basis. Customers file a claim with OpenAI, then pass the case ID to Railway's support team.
• SpaceXSI: Refunds are reviewed on a case-by-case basis, and customers submit their claims to the same support team.
• Anthropic: Talks are still underway, and the platform will cover costs from breached Anthropic keys in the meantime.
• OpenRouter: It maintains its no-refund policy in its terms of service, so the platform will cover all OpenRouter-related costs resulting from the breach.

Customers are asked to report charges they did not make through a charge report form. The form says the company "is not responsible for usage of these keys" but is investigating the theft and working with the providers.

The report does not say how the $740,000 figure was estimated or what period it covers. Neither it nor the form sets a cap or a deadline for claims, or states whether customers must pay disputed charges while a claim is under review.

The company advises customers to revoke and replace any sensitive variable stored on the platform before October 21. Keys should be revoked or rotated with the provider that issued them, since filing a charge report does not revoke a key.

It also advises removing public network access to databases, checking GitHub audit logs and which apps can access the account, and checking audit logs in registries such as Docker Hub. Most of this advice dates from the company's October disclosure and a report it published the next day.

Out of caution, the company also gave some providers every key it held for their services, so reset requests reached more than 100,000 users. A revocation notice from a provider does not by itself mean a key was exposed.

How the Railway Breach Happened

The attack began in the company's build system, according to its incident report. At 07:28 UTC on October 20, the attacker exploited a previously unknown flaw to partly escape the filesystem isolation around a build and read some files on the host virtual machine.

That gave the attacker a credential that build servers use to push finished images and reach some internal systems. The on-call engineer was paged at 13:58 UTC, about six and a half hours after the first exploit, when an alert flagged unusual calls to a registry service.

The credential was meant only for build work, but the company's internal Temporal system, which drives parts of the build process, accepted it as a trusted client. Through it, the attacker read some stored histories of other builds.

The same credential could also fetch cached snapshots of code. That gave the attacker a copy of the company's internal monorepo, the code base that many of its services deploy from. The company says the copy showed the attacker how the platform works and helped them find exploits and move quickly between systems.

After scanning that code, the attacker found a wrongly configured OIDC binding in one of the company's cloud environments. The binding is part of how its deployment system obtains short-lived cloud credentials, and at 03:26 UTC on October 21, the attacker exploited the mistake to obtain one.

With that credential, the attacker took two more from the cloud environment. One could read encrypted variables from a production database when used inside the internal network, and the other was an SSH key.

The SSH key opened a machine that runs orchestration tasks for the company's fleet. That machine stored a credential capable of decrypting customer variables, which it needed to start customer workloads.

Each environment's variables are encrypted with that environment's own key, and those keys are, in turn, encrypted by a root key stored in a cloud provider's key management service (KMS).

Between 04:07 and 05:21 UTC on October 21, the attacker pulled encrypted variables from the database, looking for names that matched AI provider keys. Using the orchestration machine's credentials with the KMS, it decrypted some of them between 04:42 and 05:14 UTC and again between 06:36 and 07:15 UTC.

The attacker was fully removed from the company's systems at 09:24 UTC that day.

Those times differ from the company's preliminary report of October 22, which dated the attack from 13:58 UTC on October 20 and placed decryption after 05:27 UTC.

Each build now runs in its own short-lived virtual machine, on top of the old isolation. Its credentials can push only one specific image to the company's registry.

Within an hour of the company patching the original escape, its own hardening team built a second exploit by changing that attack. That was one reason the company moved every user to the new build system at once.

Every weakness the attacker used is now closed, the company says. It has filed a complaint with the FBI's IC3 and is working with a specialist forensics firm whose investigation continues.

It is sharing indicators of compromise privately for now and plans to publish them once law enforcement agrees.

The company believes the stolen credentials were used by services that relay or resell access to AI providers for use in distillation. It has not established whether the attacker ran those services, nor has it named the attacker.

In their Black October analysis, Wiz researchers list this breach among incidents disclosed in October by at least nine AI developer platforms, including the earlier OpenCode breach.

Every public report describes attackers hunting for stored environment variables, especially ones that grant AI access, the researchers say. No single flaw or software dependency links the attacks.

Found this article interesting? Follow us on Google News, Twitter and LinkedIn to read more exclusive content we post.