How Staked took leadership off call, with 99.99% of alerts answered in 20 minutes
How Triotech ran 24/7 validator operations for Staked across 30+ blockchains, answering 99.99% of alerts within 20 minutes and taking leadership off call.
The situation
Staked runs validators for institutional clients on more than 30 proof-of-stake blockchains, including Ethereum, Solana, Cardano and Polygon. Validators earn rewards only while they are online, in sync and doing their job. A missed block or a failed attestation is money that clients do not get back.
Blockchains do not keep business hours, so the job has to be covered every hour of every day. At Staked, that coverage fell on three people: one engineer, the CTO and the CEO. The CTO was sleeping about two hours a day to keep up.
That was not sustainable, and it was risky. Leadership was pulled into every alert instead of growing the business, and one exhausted person missing one page could cost clients real money.
The constraint
- Real money, all the time. Staking rewards are lost the moment a validator falls behind, so slow responses cost clients directly.
- 30+ different chains. Every network has its own client software, failure modes and signs of trouble. Knowing Ethereum does not mean knowing Cardano.
- Four clouds, four regions. Validators ran across AWS, Google Cloud, Azure and Oracle Cloud, so a problem could appear anywhere.
- Keys that cannot leak. Validator keys control staked assets, so access had to stay tightly controlled while engineers worked around the clock.
- A 20-minute promise. Any alert, on any chain, at any hour, had to be answered within 20 minutes.
What we did
- Learned the platform from the insideWe started by working alongside Staked's own team to learn how the infrastructure was built, how each chain behaved and how incidents were handled. We turned that knowledge into runbooks, so any engineer on shift could respond to a given alert the same way, every time.
- Took over the pagerWe built a team of 9 to 10 engineers on 24/7 shifts, covering alert triage, incident management and service restoration. The team stayed chain-focused, building deep knowledge of each network instead of treating every alert the same.
- Monitored every layer, from server to blockchainPrometheus agents on the validator nodes tracked health, sync and resources. Datadog, Sumo Logic and Alertmanager watched servers, Kubernetes and services across all four clouds. On-chain monitoring tracked missed blocks and failed attestations, confirming validators were not just running but earning. The team also followed each chain's official Discord channels and public endpoints, so network-wide problems were spotted early and never mistaken for our own.
- Protected the keysValidator keys and secrets were managed in HashiCorp Vault, so engineers could operate around the clock without keys sitting on servers or in files.
- Ran everything as codeThe infrastructure was defined in Terraform across AWS, Google Cloud, Azure and Oracle Cloud, so any node could be rebuilt the same way in any cloud.
- Operated, improved, repeatedEvery incident fed back into the runbooks and alerting. Over 2.5 years, operations kept improving, and they continued without interruption through Staked's acquisition by Kraken.
How a 3 a.m. alert gets handled
- 1Detect: A missed block, failed attestation or node issue trips an alert in Prometheus, Datadog, Sumo Logic or on-chain monitoring.
- 2Investigate before acting: The engineer on shift works out whether the problem is ours or the network's: public endpoints and block explorers, the chain's official Discord channels, then our own node's health, sync status and logs.
- 3Decide: If the whole network is degraded, a restart will not help and could make things worse, so the team monitors and waits for the chain to recover. If only our validator has failed, the team fixes it.
- 4Respond: The matching runbook guides the fix: a restart, a resync, or a rebuild with Terraform.
- 5Verify: On-chain monitoring confirms the validator is attesting and producing blocks again.
- 6Improve: Lessons from the incident go back into the runbooks and alerts.
All within a 20-minute response SLA.
The result
Technology
Networks we supported
Ethereum · Solana · Cardano · Polygon · Chainlink · NEAR · Helium · Oasis · Livepeer · and 20+ more
Related case studies
How Tilton went from 4 outages a day to zero and cut hosting costs in half
Read case studyHow a construction SaaS took releases off developers’ machines and cut critical cloud risks to zero
Read case studyFind out what your stack is hiding.
If your platform moves money, every minute of downtime has a price, and your founders should not be your on-call rotation. Book a free audit of your operations and monitoring.