1. Normal-source outage
Time during which the normal utility or facility source is unavailable to serve its intended role—regardless of whether alternate paths, storage or generation maintain the load.
Data-center reliability knowledge
When Power Returns, Computation May Not: Understanding Data-Center Outage Duration and Cost
What does an electrical outage actually cost an AI data center? The answer depends on which clock you are timing—and on what the workload was doing when the event occurred.
A prolonged utility outage may produce no compute interruption if alternate sources and stored energy perform correctly. Conversely, a brief rack-power interruption can stop a training workload and cause a much longer recovery process than the electrical gap itself.
Illustrative article. Timing values, sequences and cost examples are modeling assumptions for explanation. They are not NVIDIA, OEM or site-specific specifications unless explicitly cited as published references below.
Important distinction. This article explains concepts and relationships used in reliability and outage-cost assessment. It does not certify the performance of any particular facility, design or vendor product. Site-specific conclusions require verified one-line diagrams, protection settings, switching logic, equipment ratings, operating data and owner/OEM requirements.
Why one duration is not enough
Electrical time, rack power, compute and resilience follow different clocks
Operators, insurers, utilities and workload owners often speak about “the outage” as if it were a single interval. In practice, several distinct durations may coexist in one event—and each may drive a different part of the cost.
- Loss of the normal source does not necessarily mean loss of rack power.
- Loss of rack power does not necessarily occur immediately because of Power Supply Unit (PSU) hold-up and stored energy.
- Restoration of rack power does not necessarily restore useful computation.
- Restoration of computation does not necessarily restore the progress lost since the last checkpoint.
- Restoration of service does not necessarily mean that full redundancy and stored-energy reserves have been restored.
Service restored ≠ full resilience restored. A facility may resume revenue-bearing work while still operating with reduced margin, partially recharged batteries, temporary generation or unrepaired equipment.
Part 1
Part 1: Which outage are we timing?
The following eleven clocks are frequently conflated in conversation and reporting. Each should be defined explicitly before comparing facilities, contracts or mitigation options. Event durations span many orders of magnitude—from milliseconds at the rack to hours or days for full restoration—consistent with the POI time-scale discussion on Load Behavior & POI.
2. Fault-clearing interval
Time from fault inception through detection, primary protection and, when required, breaker-failure or backup protection. This interval shapes whether downstream ride-through is challenged immediately or after a brief disturbance.
3. PSU hold-up interval
The brief period during which server PSU capacitors maintain internal DC rails after rack input power is lost. Published ATX-family design guidance specifies hold-up on the order of milliseconds to tens of milliseconds depending on loading and product class—see references. An illustrative 10 ms modeling value is used in examples below; it is not a universal server or rack specification and may differ from cited ATX minima and from OEM designs. Actual values must come from the server/OEM specification for the equipment in service.
4. Local-storage takeover time
Time from loss of rack input until rack-level storage accepts the load. Distinguish:
- Rack BBU or rack-local UPS — stored energy physically adjacent to or dedicated to the rack feed;
- Facility BESS or distributed UPS — larger stored-energy systems that may serve multiple racks, buses or facility loads through switching and distribution.
The critical design relationship at the rack is:
storage takeover time < PSU hold-up time
If takeover exceeds hold-up, the server may reset even though facility BESS or another upstream source exists but has not yet reached the rack.
5. Local-storage autonomy or bridging duration
Time for which stored energy can sustain the rack at the required load. Rack BBU autonomy and facility BESS endurance differ in energy, power, state of charge and recharge requirements. 10 minutes appears below only as an illustrative rack-bridging scenario—not as an NVIDIA or universal data-center specification. Autonomy depends on state of charge, load, temperature, battery health and design margin.
6. Backup-source response time
Separate sub-intervals often matter:
- start command;
- source startup;
- voltage/frequency or DC stabilization;
- source qualification;
- transfer;
- staged load pickup;
- restoration of required cooling and communications loads.
“Generator started” is not the same as “the required rack load has been successfully transferred.”
7. Rack-power interruption
Time during which acceptable electrical power is unavailable at the rack after considering all normal feeds, alternate paths, storage and PSU hold-up.
8. Compute-hardware interruption
Time during which servers or accelerators cannot remain operational because their internal power requirements are not maintained.
9. Useful-compute or service interruption
Time until the application or workload is again performing its intended function. This can include:
- hardware restart;
- firmware and operating-system boot;
- network and storage readiness;
- scheduler availability;
- checkpoint validation and loading;
- distributed-worker resynchronization;
- service-capacity ramp.
10. Pre-fault computational progress recovery
For illustration only, consider a training job with a 20-minute checkpoint interval (configuration- and workload-dependent). A failure may lose between zero and nearly one full interval of work. If failures occur uniformly between checkpoints, the expected lost progress is approximately 10 minutes under that illustrative assumption—not a universal training loss rule.
progress recovery time = useful-compute restoration time + time required to repeat lost work
This measure is important for training and long-running batch jobs but is generally not applicable to interactive inference in the same way—there, lost requests rather than repeated gradient steps dominate.
11. Full-resilience restoration
Time until:
- failed equipment is repaired;
- normal configuration is restored;
- retransfer is complete;
- batteries are recharged;
- cooling and communication redundancy are restored;
- the facility has regained its required reserve margin.
Terminology
Compact glossary — colloquial terms and clocks
Colloquial outage language often mixes layers. The table maps common terms to the clocks defined above.
| Colloquial term | Primary clock(s) | Notes |
|---|---|---|
| Utility outage / utility-supply interruption | 1. Normal-source outage | Utility unavailable in its normal role; rack power may continue on BESS, generator or alternate feed. |
| Normal-source outage | 1. Normal-source outage | Same as intended utility or designated normal facility source—not necessarily loss of all rack input. |
| Rack-power interruption | 7. Rack-power interruption | Loss of acceptable electrical input at the rack after hold-up and bridging. |
| IT interruption | 7–9 | Often used loosely; may mean rack power, hardware down or service unavailable. Specify which layer is meant. |
| Compute interruption / compute-hardware interruption | 8. Compute-hardware interruption | Servers or accelerators cannot remain powered and operational. |
| Service interruption / useful-compute interruption | 9. Useful-compute or service interruption | Application or workload not performing its intended function; may persist after rack power returns. |
| Progress recovery / computational recovery | 10. Pre-fault computational progress recovery | Includes restart plus repetition of work lost since the last valid checkpoint (training/batch context). |
| Full restoration / resilience restoration | 11. Full-resilience restoration | Normal configuration, redundancy, recharge and reserve margin—not the same as service restored. |
InfraRel-compatible consequence measures related to these clocks include expected unserved energy (electrical energy not delivered), unserved computation (useful work not completed) and lost accelerator-hours (compute capacity-time lost to interruption and recovery).
Worked example
Example A — Source outage without compute interruption
Illustrative sequence only. Actual timing depends on protection, switching, storage state and load.
- t = 0: normal source lost; rack BBU or facility storage accepts the rack load before any post-interruption PSU hold-up is relevant.
- Before rack storage exhaustion: backup source starts, stabilizes and accepts the critical load.
- Throughout: rack power remains acceptable; useful computation continues.
- Later: batteries recharge; normal supply and reserves return; full resilience is restored.
Duration summary. Normal-source outage has a positive duration. Rack-power interruption, compute-hardware interruption and useful-compute interruption durations are zero in this illustrative case. Operational cost may still include fuel, battery cycling, operator response and reduced reserve margin until full resilience returns.
Worked example
Example B — Storage exhaustion and workload recovery
The same initiating electrical event can produce multiple non-zero clocks and a long recovery tail.
- t = 0: normal source lost; rack BBU accepts the rack load (acceptable rack input continues).
- t = 8 min: low-energy warning (illustrative).
- t = 10 min: rack storage exhausted; rack input lost; PSU hold-up begins and elapses in milliseconds; servers reset and computation stops.
- t = 10–20 min: rack-power interruption continues while backup source completes start, qualification and transfer.
- t = 20 min: acceptable electrical supply returns to the rack from an alternate/backup path. The utility or normal source may still be unavailable.
- Following rack-power restoration: servers restart; network and storage become available; checkpoint is loaded; distributed workers resynchronize.
- Later: useful computation resumes; work since the previous checkpoint is repeated; normal utility supply and storage reserves are restored; full resilience returns.
Clocks with non-zero duration
- normal-source outage;
- rack-power interruption (~10–20 min illustrative);
- useful-compute interruption (extends through restart and resync);
- progress-recovery duration (includes repeated work);
- reduced-resilience duration (until recharge and retransfer).
Why cost can exceed electrical duration
Electrical service may return to the rack at 20 minutes from backup while the utility normal source remains out until later. Useful compute may remain unavailable until ~35 minutes and pre-fault progress may not be restored until ~42 minutes in this illustration. Directly attributable SLA credits, repeated-work cost and lost accelerator-hours attach to those longer intervals—not to the utility-outage clock alone.
These values are scenario assumptions for teaching. They do not represent every data center, every checkpoint policy or every backup design.
Worked example
Example C — Sub-second transfer glitch, long computational recovery
Illustrative sequence showing that the normal source may remain available while a brief electrical disturbance still stops useful computation for much longer.
- t = 0: utility source healthy; a fault-clearing and transfer event produces a sub-second rack-input disturbance (illustrative 20–80 ms transfer glitch—configuration-dependent).
- Rack BBU or UPS does not bridge in time, or control-power/transfer logic fails to maintain acceptable rack input.
- PSU hold-up elapses in milliseconds; servers reset even though the utility outage duration is effectively zero.
- Normal-source and alternate-source clocks: little or no utility loss; rack-power interruption is very short; compute-hardware interruption is brief but decisive.
- Following reset: cluster restart, storage and network readiness, checkpoint validation/load and worker resynchronization extend useful-compute interruption to tens of minutes (illustrative).
- Progress recovery includes repeated work since the last checkpoint under the same illustrative 20-minute checkpoint policy used above.
Duration summary. Normal-source outage ≈ 0. Rack-power interruption is very short. Useful-compute interruption and progress-recovery duration can dominate cost through idle accelerator-hours, unserved computation and lost training progress—despite minimal utility-visible outage time.
Visual summary
Multiple clocks on one timeline (Example B, illustrative)
The figure shows a subset of clocks from Example B—not all eleven—prioritizing readability. Rack power may return from backup while the utility normal source remains unavailable.
Text summary of Example B timeline
- Fault clearing (clock 2): narrow interval at t = 0 (illustrative sub-second).
- Normal source (clock 1): utility unavailable from t = 0 through ~45 min in this scenario.
- Backup/alternate source (clock 6): begins serving critical load after startup (~2–20 min illustrative).
- Rack input (clock 7): bridged by rack BBU 0–10 min; interrupted 10–20 min; restored at 20 min from alternate feed.
- Compute hardware (clock 8): follows rack input; reset after storage exhaustion and brief PSU hold-up.
- Useful computation (clock 9): continues until 10 min; unavailable until ~35 min; then ramps.
- Progress recovery (clock 10): pre-fault position not restored until ~42 min after repeated work.
- Full resilience (clock 11): reduced reserve until utility return, recharge and retransfer complete.
Boundary
Scope of the cost assessment
Read this section before interpreting the cost categories below. This article estimates operational-continuity costs attributable to an electrical initiating event and the resulting sequence of protection, switching, backup-power, rack-power, compute-interruption and workload-recovery states.
Costs within scope may include
- electrical fault isolation, switching, repair and restoration;
- backup generation, stored-energy discharge and subsequent recharge;
- loss or degradation of rack power;
- idle compute resources, unserved energy, unserved computation and lost accelerator-hours directly attributable to the event;
- server restart, checkpoint reload and workload resynchronization;
- repeated computation following checkpoint rollback;
- delayed training or inference;
- directly attributable lost revenue from service not delivered during the interruption;
- defined contractual service consequences and SLA credits tied to measured service unavailability;
- restoration of electrical and computing resilience.
The analysis does not estimate
- cyber intrusion, ransomware, forensics or incident-response costs;
- data theft, privacy notification, litigation or regulatory penalties;
- speculative enterprise valuation or broader business-valuation effects;
- generalized reputational impact unless separately scoped with explicit scenario assumptions;
- deliberate manipulation of controls, models, training data or outputs, except for the subsequent electrical interruption if one occurs;
- unsafe or abnormal AI behavior;
- malicious use of AI or information used outside its intended context;
- misinformation, discrimination, public-safety impacts or other societal harms;
- national-security and broader economic externalities.
A cyber event may initiate an electrical outage, and an electrical outage may expose cyber or data-recovery weaknesses. This article calculates the operational consequences of the electrical and computing interruption. Cybersecurity, AI-behavioral, misuse and societal consequences require additional risk models and are not included in the cost estimate.
Do not describe every excluded cost as “indirect.” Some cyber and AI incidents create direct financial costs. They are separate risk domains outside the defined electrical-outage cost boundary.
Assessment layers and adjacent risk domains
1 Electrical initiating event
2 Electrical and backup-power response
3 Rack-power consequence
4 Compute and workload consequence
5 Direct operational cost
Adjacent analytical domains — separate models
Cybersecurity risk
NIST CSF 2.0
NIST CSF 2.0
AI-behavioral risk
NIST AI RMF
NIST AI RMF
Societal harm / enterprise valuation
Separate models
Separate models
Authoritative frameworks for adjacent domains include the NIST Cybersecurity Framework 2.0, the NIST AI Risk Management Framework, and relevant critical-infrastructure cybersecurity guidance. This article does not provide a complete enterprise-risk, cyber-risk or societal-cost assessment.
Part 2
Part 2: What does the outage actually cost?
Outage cost cannot be calculated from electrical outage duration alone. The categories below are bounded by Scope of the cost assessment. The same utility interruption may be inexpensive if bridged successfully, or very expensive if it causes checkpoint rollback, unserved computation and lost accelerator-hours.
Total outage cost = lost service + idle compute + repeated work + recovery + SLA consequences + operational intervention + equipment/data consequences + reserve restoration
Lost or delayed service
- unavailable customer service;
- rejected or timed-out requests;
- delayed jobs;
- lost revenue directly attributable to service not delivered during the interruption.
Idle accelerator-hours / unserved computation
- GPUs and other accelerators unavailable;
- powered but unable to perform useful work;
- stranded compute caused by unavailable cooling, communication, storage or orchestration;
- unserved computation and lost accelerator-hours as InfraRel-compatible consequence indices.
Repeated computation
- work lost since the last valid checkpoint;
- energy consumed repeating work;
- accelerator time needed to return to the pre-fault computational position.
Restart and checkpoint recovery
- server restart;
- checkpoint verification and loading;
- cluster resynchronization;
- workload restart;
- service ramp.
SLA and defined contractual service consequences
- contractual SLA credits tied to measured service unavailability;
- defined colocation or cloud service penalties in force for the event;
- customer compensation explicitly governed by contract terms;
- grid-service nonperformance payments, where applicable and contractually defined.
Operational intervention
- operator response;
- emergency switching;
- troubleshooting;
- repair;
- vendor support;
- incident review.
Equipment and data consequences
- damaged equipment;
- failed batteries or switching equipment;
- corrupted or incomplete writes;
- invalid checkpoints;
- loss of stored state.
Restoration of reserve / unserved energy
- battery recharge and unserved energy during the event;
- temporary fuel consumption;
- continued generator operation;
- unavailable redundancy;
- increased exposure to a second event during recovery.
Generalized reputational impact, litigation, regulatory penalties and enterprise-valuation effects are out of scope unless separately modeled with explicit assumptions.
Workload comparison
Training versus interactive inference
AI training interruption
An interruption can create checkpoint rollback, repeated work, lengthy restart, cluster reassembly, idle accelerator-hours and delayed model-development schedule. Training interruptions can have a long recovery tail and may be especially costly per affected job.
Interactive inference interruption
An interruption can cause immediate user impact, failed or delayed requests, traffic rerouting and defined SLA credits. Inference outages may be shorter in recovery but can immediately affect very large numbers of users.
Cost depends on workload, scale, traffic, checkpoint policy, recovery automation, redundancy and contractual commitments. Training is not always costlier than inference, and inference is not always cheaper—each event must be evaluated in context.
Qualitative comparison
Duration and cost are not proportional
The table below is qualitative and configuration-dependent. It illustrates why electrical duration alone is an unreliable proxy for total consequence.
| Electrical event | Electrical duration | Compute consequence | Potential cost |
|---|---|---|---|
| Utility loss successfully bridged | Long | None | Low operational/energy cost |
| Brief rack interruption causing server reset | Very short | Long workload recovery | Potentially high |
| Cooling loss with rack power intact | No rack-power outage | Throttling or shutdown | Potentially high |
| Network isolation with healthy compute | No electrical outage | Service unavailable | Potentially high |
| Transfer failure / failed load transfer | Short or none at utility | Rack or facility load not accepted | Potentially high |
| Control-power loss | May be brief or none | Protection, transfer or cooling logic unavailable | Potentially high |
| Generator started but transfer failed | Backup running | Rack still unserved; long recovery possible | Potentially high |
| Training-job interruption | Short or moderate | Checkpoint rollback and repeated work | Workload-dependent |
| Inference-service interruption | Short or moderate | Immediate user/SLA impact | Traffic-dependent |
Qualitative illustration only. Actual cost requires site-specific inputs and explicit assumptions.
Utility bridged
Electrical: Long · Compute: None · Cost: Low operational/energy
Brief rack interruption
Electrical: Very short · Compute: Long recovery · Cost: Potentially high
Cooling loss
Electrical: No rack-power outage · Compute: Throttling/shutdown · Cost: Potentially high
Network isolation
Electrical: None · Compute: Service unavailable · Cost: Potentially high
Transfer failure
Electrical: Short/none at utility · Compute: Load not accepted · Cost: Potentially high
Control-power loss
Electrical: Brief/none · Compute: Logic unavailable · Cost: Potentially high
Generator start, failed transfer
Electrical: Backup running · Compute: Rack unserved · Cost: Potentially high
Training interruption
Electrical: Short/moderate · Compute: Rollback/repeat · Cost: Workload-dependent
Inference interruption
Electrical: Short/moderate · Compute: User/SLA impact · Cost: Traffic-dependent
Inputs
Data required to calculate actual cost
Missing inputs should be represented as ranges, scenarios or explicitly marked unknowns—not silently invented. InfraRel-style studies may report expected unserved energy, unserved computation and lost accelerator-hours alongside dollar-valued outage cost.
- component failure rates;
- protection and switching times;
- backup-source start and transfer times;
- transfer-failure and control-power dependencies;
- PSU hold-up;
- rack BBU and facility BESS power, energy, state of charge and endurance;
- rack load;
- cooling and communication dependencies;
- server restart duration;
- checkpoint interval;
- checkpoint size and loading time;
- cluster resynchronization time;
- accelerator value per hour;
- unserved energy, unserved computation and lost accelerator-hours valuation factors;
- electricity and fuel costs;
- lost revenue;
- SLA penalties;
- operator and repair costs;
- value of delayed training progress.
Emerging methodology
InfraRel and collaboration
General Reliability is extending InfraRel to connect the complete chain from network configuration through outage cost—under development and not yet a site-specific validated data-center assessment.
Network configuration → Fault and protection response → Time-sequenced switching → Ride-through → Rack-power consequence → Workload recovery → Outage cost
We welcome collaboration with data-center operators, utilities, electrical and cooling OEMs, server and storage specialists, network and workload engineers, insurers and risk specialists, and researchers who can contribute operational data, validation experience and domain requirements.
Sources
References and further reading
Sources are grouped by type. Marketing language is not treated as established engineering fact.
Intel ATX Version 3.0 — voltage hold-up time
Design-guide hold-up requirements for desktop/platform PSUs. Server and rack PSUs may differ; use OEM data for in-service equipment.
Schneider Electric — UPS transfer time (APC Smart-UPS / Symmetra)
Illustrates that transfer intervals are product- and mode-dependent and may exceed server hold-up unless double-conversion or matched designs are used.
NVIDIA NeMo — distributed checkpoints
Public documentation on checkpoint save/load behavior relevant to training recovery after interruption.
Uptime Institute — Annual Outage Analysis 2024 (executive summary)
Survey-based perspective on outage frequency, severity and reported cost magnitudes. Supports general discussion of outage-cost scale—not this article’s electrical-to-computational clock taxonomy. Full report requires membership.
NVIDIA — 800 VDC Architecture for Next-Generation AI Infrastructure (PDF)
Facility-level reference material on power-block architecture, stored energy and ride-through concepts. Reference designs are illustrative, not site-specific specifications.
Evidence classes used in this article: published specification/documentation; general industry reference; illustrative assumption (timing scenarios); owner/OEM-required input (site-specific switching, storage, workload and cost data).