How Staked took leadership off call, with 99.99% of alerts answered in 20 minutes
The situation
Staked runs validators for institutional clients on more than 30 proof-of-stake blockchains, including Ethereum, Solana, Cardano and Polygon. Validators earn rewards only while they are online, in sync and doing their job. A missed block or a failed attestation is money that clients do not get back.
Blockchains do not keep business hours, so the job has to be covered every hour of every day. At Staked, that coverage fell on three people: one engineer, the CTO and the CEO. The CTO was sleeping about two hours a day to keep up.
That was not sustainable, and it was risky. Leadership was pulled into every alert instead of growing the business, and one exhausted person missing one page could cost clients real money.
The constraint
- Real money, all the time. Staking rewards are lost the moment a validator falls behind, so slow responses cost clients directly.
- 30+ different chains. Every network has its own client software, failure modes and signs of trouble. Knowing Ethereum does not mean knowing Cardano.
- Four clouds, four regions. Validators ran across AWS, Google Cloud, Azure and Oracle Cloud, so a problem could appear anywhere.
- Keys that cannot leak. Validator keys control staked assets, so access had to stay tightly controlled while engineers worked around the clock.
- A 20-minute promise. Any alert, on any chain, at any hour, had to be answered within 20 minutes.
What we did
- Learned the platform from the inside. We started by working alongside Staked’s own team to learn how the infrastructure was built, how each chain behaved and how incidents were handled. We turned that knowledge into runbooks, so any engineer on shift could respond to a given alert the same way, every time.
- Took over the pager. We built a team of 9 to 10 engineers on 24/7 shifts, covering alert triage, incident management and service restoration. The team stayed chain-focused, building deep knowledge of each network instead of treating every alert the same.
- Monitored every layer, from server to blockchain. Prometheus agents on the validator nodes tracked health, sync and resources. Datadog, Sumo Logic and Alertmanager watched servers, Kubernetes and services across all four clouds. On-chain monitoring tracked missed blocks and failed attestations, confirming validators were not just running but earning. The team also followed each chain’s official Discord channels and public endpoints, so network-wide problems were spotted early and never mistaken for our own.
- Protected the keys. Validator keys and secrets were managed in HashiCorp Vault, so engineers could operate around the clock without keys sitting on servers or in files.
- Ran everything as code. The infrastructure was defined in Terraform across AWS, Google Cloud, Azure and Oracle Cloud, so any node could be rebuilt the same way in any cloud.
- Operated, improved, repeated. Every incident fed back into the runbooks and alerting. Over 2.5 years, operations kept improving, and they continued without interruption through Staked’s acquisition by Kraken.
The result
99.99% of alerts answered in 20 minutes
Across every chain and every hour, alerts were answered within the 20-minute SLA.
30+ blockchains watched 24/7
Every network monitored around the clock for missed blocks and failed attestations.
Leadership off call
The CTO and CEO were taken off 24/7 on-call and freed to lead the company.
No single point of failure
Runbooks and incident management meant response no longer depended on any one person.
2.5 years, no interruption
Continuous validator operations, including through Staked's acquisition by Kraken.
Services involved
If your platform moves money, every minute of downtime has a price, and your founders should not be your on-call rotation. Book a free audit of your operations and monitoring.
Book a free audit →