Summary
On the evening of 21 October 2026, Railway, which hosts parts of DeviceCloud, told its customers that an attacker may have read and decrypted the environment variables of hosted services. Environment variables are where services keep their keys and passwords. Ours held keys for our database, file storage, payments, email and other systems.
We found no evidence that any of these keys were used by anyone else. A clean log can’t prove a key wasn’t copied, though, so we treated every one as leaked and replaced them all.
We switched off the old keys straight away, before every service had moved to new ones. That stopped any leaked key from working, but it caused disruption between 07:27 and 11:15 UTC. Some cloud runs were delayed, logging in to the console and the CLI failed for a time, and some notifications for finished runs were not sent. No test results were lost. We’re sorry for the disruption.
Your DeviceCloud API keys were not exposed. They were never stored in Railway, and we keep only a one-way hash of them, so the keys themselves can’t be read back.
Do you need to do anything?
- Update the CLI to 5.10.0 or later. Version 5.9.0 fixes
dcd loginsessions, which older versions can’t refresh. Version 5.10.0 also turns on envelope encryption by default, so your app binaries are encrypted before they’re uploaded. Runnpm install -g @devicecloud.dev/dcd@latest, or use the install script at get.devicecloud.dev. - Log in again to the web console and the CLI if you’re asked to.
You don’t need to change your API keys or webhook secrets, or reconnect Slack.
What happened
Railway hosts our API, our web console and several internal services. At 20:44 on 21 October, Railway published a security disclosure. It said a third party may have accessed the encrypted environment variables it stores, and decrypted some of them. It asked every customer to replace all their environment variables and check their logs for unusual activity, and it paused all deployments while it investigated. Its status page repeated the advice at 22:10.
We picked up Railway’s notice at 07:00 and started straight away. We listed every secret our services held in Railway and assumed all of them had leaked.
Railway’s early analysis suggested only some customers were affected, but it asked everyone to act as a precaution. Read Railway’s disclosure for its own account.
Why replacing keys caused disruption
The safest way to replace a leaked key is to stop it working first, then move each service to its replacement. The alternative is to update every service before switching anything off, which avoids downtime but leaves the leaked keys valid for longer.
We chose to cut the old keys off quickly. At 07:27 we turned off the old keys on our production database, and at about 08:00 we revoked the secret that signed them. From that point, no copy of those keys would work anywhere. The cost was that our own services failed until each one was running on its new keys.
How we checked for misuse
Before and during the rotation, we reviewed:
- Database access logs: who used our privileged database keys, from where, and for what. All traffic came from our own services.
- API and file-delivery logs: failed authentication, unusual access to internal endpoints, and any sign of forged download links. We found none.
- Payments: webhook settings, discounts, refunds and other changes. We found nothing unexpected.
- Source code hosting: members, deploy keys, release assets and the install scripts at get.devicecloud.dev. Nothing had changed.
Impact
No test results, recordings or artifacts were lost.
| Area | When (UTC) | Impact |
|---|---|---|
| API | 07:27 to 08:21 | Some requests failed |
| iOS and Android runs | 07:30 to 10:08 | New runs were delayed, then ran from the queue |
| File downloads | until 09:13 | Some result links needed a page reload |
| Web console | until about 11:40 | Login unavailable |
| CLI | from 07:27 | dcd login needs CLI 5.9.0 or later |
| Single sign-on | 10:45 to 11:11 | SSO login unavailable |
| Notifications | during the incident | Some run notifications weren’t sent |
What we did
We replaced every secret that Railway held, including:
- the keys our services use to reach our databases, and the database passwords
- the signing secret behind every database key, so leaked database keys stopped working everywhere, not only in the services we updated
- keys for file storage, payments, email, AI services and our own internal services
- the secret that signs download links
Many of our vendors and internal processes also now support narrower scoped keys. So we changed how some secrets are held, so a future leak would expose less:
- Each service has its own keys where possible, so one leak affects one service.
- Narrowly scoped access tokens replaced tokens with full account access. Each new token reaches only the projects and actions its job needs, and each one expires.
- The CLI’s key can be rotated without a release. From version 5.9.0, the CLI fetches its key from our API.
- Uploaded app binaries are encrypted by default. From CLI 5.10.0, binaries are protected with envelope encryption: each upload is encrypted with its own key, and those keys are themselves encrypted with a master key. This adds a layer of protection on top of our existing point-to-point and storage encryption standards.
Timeline (UTC)
| Time | Event |
|---|---|
| 21 Oct, 20:44 | Railway publishes its security disclosure and pauses all deployments. Its status page repeats the advice at 22:10 |
| 22 Oct, 07:00 | We see Railway’s notice, start the investigation and list every secret |
| 07:27 | Old database keys turned off. Services using them start failing |
| about 08:00 | Old signing secret revoked, so every leaked database key stops working |
| 08:17 | CLI 5.9.0 released |
| 08:21 | API back on new keys |
| 09:01 | Internal back-office services back on new keys |
| 09:13 | Download-link signing secret rotated |
| 10:08 | iOS and Android job dispatch restored |
| 10:40 | Web console redeployed with new keys |
| 10:45 to 11:11 | Single sign-on lookups fail while their token is replaced |
| 15:30 | Remaining internal and third-party keys rotated. Every secret Railway held has been replaced |
Contact
If you saw something during this incident that isn’t covered here, email us at support@devicecloud.dev.