We Had 17 kW of Capacity. We Still Tripped the Breaker
A PDU rating is not a budget you spend down. It is a sum of smaller budgets, each with its own breaker, and a server can sail through the first number while catastrophically failing the second.
We learned this the way most people do.
A GPU node went dark twice in nineteen hours. Both times the symptom was identical and useless: nothing in the router logs, then the BMC connection dropped, then the host IP stopped answering. Everything pointing at one conclusion the machine was simply gone.
The BMC's own event log was more specific. Eight minutes after each blackout, all six power supplies reported "presence detected," within the same second. Six supplies announcing themselves simultaneously is not a reboot. That is a chassis being re-energized from cold.
So a breaker had opened, and the chassis had lost AC.
Which raised a more annoying question. The rack has two PDUs, A side and B side, each rated 17.3 kW. The server draws 9.3 kW at peak, split across both. Each PDU was running at roughly a quarter of its rating.
We tripped anyway. That does not make sense, and the reason it does not make sense is the entire point of this post.
The machine
┌───────────────────────────────────────┐
│ Supermicro SYS-821GE-TNHR 8U │
├───────────────────────────────────────┤
│ HGX H100 8-GPU baseboard (SXM5) │
│ ┌─┐┌─┐┌─┐┌─┐┌─┐┌─┐┌─┐┌─┐ │
│ │1││2││3││4││5││6││7││8│ │
│ └─┘└─┘└─┘└─┘└─┘└─┘└─┘└─┘ │
│ └──── 4 × NVSwitch ────┘ │
│ │
│ ┌──────┐ ┌─────────────────┐ ┌──────┐ │
│ │ Xeon │ │ 32 × DDR5 RDIMM │ │ Xeon │ │
│ │ 350W │ │ up to 8 TB │ │ 350W │ │
│ └──────┘ └─────────────────┘ └──────┘ │
│ │
│ ▓▓▓ 10 × PWM fans @ 11,500 RPM ▓▓▓ │
├───────────────────────────────────────┤
│ 6 × PWS-3K06G-2R Titanium 96% 4+2 │
│ ┌───┐┌───┐┌───┐ ┌───┐┌───┐┌───┐ │
│ │PS1││PS2││PS3│ │PS4││PS5││PS6│ │
│ └─┬─┘└─┬─┘└─┬─┘ └─┬─┘└─┬─┘└─┬─┘ │
└───┼────┼────┼───────┼────┼────┼───────┘
│ │ │ │ │ │
3 × C19 cords 3 × C19 cordsSix power supplies, six cords, two PDUs, and as it turned out, twelve breakers they could have been spread across. Remember that.
And one line from the PSU datasheet that almost nobody reads, because the rating is not a single number:
PWS-3K06G-2R input rating is voltage-dependent
2880 W @ 200–207 Vac
3000 W @ 207–240 VacOur measured line voltage is 207 V, exactly on the boundary. Sag two volts under load and every supply in the chassis is a 2,880 W supply, not a 3,000 W one. Hold onto that. It matters at the end.
Where 9.3 kW actually goes
Our BMC reports a peak of 9,304 W at the wall. Only one line item in that number is firm:
8 × H100 SXM5 @ 700 W TDP = 5,600 W ← 60% of the total
Everything else = 3,704 W ← 40%That second number is what surprises people. It is not slop or measurement error. It is NVSwitches, 32 DDR5 DIMMs, two 350 W Xeons, ConnectX NICs, NVMe, and — the big one — ten heavy-duty fans spinning at 11,500 RPM to move air through an 8U box dissipating five and a half kilowatts of GPU. High-static-pressure fans at full tilt are worth a kilowatt on their own.
A reasonable decomposition:
8 × H100 SXM5 @ 700 W 5,600 W ██████████████████████ spec
10 × fans @ full RPM ~1,000 W ████ estimate
2 × Xeon Platinum @ 350 W 700 W ███ spec
4 × NVSwitch ~500 W ██ estimate
32 × DDR5 RDIMM ~320 W █ estimate
NICs / NVMe / BMC / misc ~300 W █ estimate
────────
DC total ~8,420 W
÷ 0.96 Titanium PSU ────────
AC total ~8,770 W
measured peak 9,304 WThe estimate lands about 500 W low, which is roughly right for transient GPU excursions above TDP and fans working harder than I credited. I would rather show a breakdown that does not perfectly close than invent numbers that do.
If you are sizing a rack from a spec sheet and you budget for the GPUs, you have budgeted for 60% of your load.
A detour: how much of that 700 W do you actually need?
The obvious reaction to a power problem is to cap the GPUs. It is worth knowing what that costs before you reach for it.
Two 2026 studies swept H100 power caps across real workloads (arXiv:2603.16164, arXiv:2604.11391). For LLM pre-training on H100:
| Cap | tok/s | vs 700 W | GPU power saved |
|---|---|---|---|
| 700 W | 30,271 | — | — |
| 600 W | 29,250 | −3.4% | −14% |
| 500 W | 27,705 | −8.5% | −29% |
| 400 W | 24,853 | −17.9% | −43% |
| 300 W | 17,982 | −40.6% | −57% |
The curve is non-linear in the direction that favours you. Going from 700 W to 600 W costs 3.4% of throughput and returns 14% of the power — nearly free. At 500 W you pay 8.5% for 29%. Below 400 W it turns hard and you start losing more in time than you save in watts.
Efficiency per watt peaks around 300–400 W. Production sanity lives at 500–600 W. Across eight GPUs, a 600 W cap takes 5,600 W down to 4,800 W for a rounding error of throughput.
This is a real lever. It was not our problem, and reaching for it first cost us a day.
The PDU
Here is the thing we had wrong in our heads. "17 kW" sounds like a tank you draw from. It is not. It is a three-phase feed that immediately splits into six independently protected branches.
First, where the 17 kW itself comes from:
Three-phase, 208 V line-to-line, 60 A input connector
Continuous rating (NEC 80% rule, loads over 3 hours)
I = 60 A × 0.80 = 48 A per line conductor
Delta phase current
I_phase = I_line / √3 = 48 / 1.732 = 27.71 A
Apparent power, three legs summed
S = 3 × 208 V × 27.71 A = 17,292 VA
Same thing via line quantities
S = √3 × 208 V × 48 A = 17,292 VAThe √3 is not mystical. Power is scalar and the three legs add arithmetically. The √3 enters one step earlier, converting line current to phase current, because in a delta each line conductor feeds two legs that peak at different instants. It is 2·sin(60°), and it is also why 208 V exists at all: 120 V × √3 = 208 V.
Now the part that matters:
Ours is an Eaton EVMA4609X, a 0U managed rack PDU (datasheet). The line that ended our investigation:
Outlets are evenly distributed into six color-coded load banks, each with its own 2-pole 20 A circuit breaker.
┌──────────────────────────────────────────────────────────────┐
│ Eaton EVMA4609X — 17.3 kW, 208 V 3-phase, 460P9W 60 A in │
├──────────────────────────────────────────────────────────────┤
│ │
│ The 17 kW is not one pool. It is six load banks: │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ BANK A │ │ BANK B │ │ BANK C │ each bank: │
│ │ ⊙ 20 A │ │ ⊙ 20 A │ │ ⊙ 20 A │ 2-pole 20 A │
│ │ 4×C13 │ │ 4×C13 │ │ 4×C13 │ breaker │
│ │ 3×C39 │ │ 3×C39 │ │ 3×C39 │ │
│ └──────────┘ └──────────┘ └──────────┘ 20 A × 208 V │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ = 4,160 VA │
│ │ BANK D │ │ BANK E │ │ BANK F │ │
│ │ ⊙ 20 A │ │ ⊙ 20 A │ │ ⊙ 20 A │ continuous (80%) │
│ │ 4×C13 │ │ 4×C13 │ │ 4×C13 │ = 3,328 VA │
│ │ 3×C39 │ │ 3×C39 │ │ 3×C39 │ │
│ └──────────┘ └──────────┘ └──────────┘ │
│ │
└──────────────────────────────────────────────────────────────┘Note the outlet split: 24 C13 and 18 C39 across six banks is four C13 and three C39 per bank. Our PSU cords terminate in C20 plugs, which only fit the C39s. Three per bank — the exact number of cords we had on each side.
Worth noting why the branch breakers oversubscribe the feed. Six 20 A branches is 24,960 VA of breaker capacity on a 17,292 VA feed — 1.44×. That is deliberate. Nobody loads all six to the limit at once, and if you did, the main would open before any branch did. The oversubscription buys you the freedom to run one bank heavy while its neighbour idles.
We used that freedom badly.
The wiring, and the arithmetic
What we had:
SERVER A-SIDE PDU B-SIDE PDU
PSU1 ──┐ ┌──────────┐ ┌──────────┐
PSU2 ──┼──── 3 cords ───────►│ bank A ⊙ │ │ bank A ⊙ │
PSU3 ──┘ │ bank B ⊙ │ empty │ bank B ⊙ │ empty
│ bank C ⊙ │ │ bank C ⊙ │
PSU4 ──┐ │ bank D ⊙ │ │ bank D ⊙ │
PSU5 ──┼──── 3 cords ──┐ │ bank E ⊙ │ │ bank E ⊙ │
PSU6 ──┘ │ │ bank F ⊙ │◄── 3 │ bank F ⊙ │◄── 3
└─────┴──────────┴─────────┤ 22.5 A │
22.5 A │ │
112% ⚠ │ 112% ⚠ │
└──────────┘ └──────────┘
Four banks idle on each PDU. Both loaded banks over rating.Step by step:
1. Total AC draw at peak
P = 9,304 W (measured, BMC)
2. Total current
I = P / V
= 9,304 W / 207 V
= 44.9 A
3. Per-PSU current
All six supplies share load actively. "4+2" describes how many
it can lose, not how many sit idle — our telemetry showed all
six within 243–336 W of each other at idle.
I_psu = 44.9 A / 6
= 7.49 A per cord
4. Current at each loaded bank
Voltage is common across the bank bus; currents sum.
I_bank = 7.49 A × 3
= 22.5 A on a 20 A breaker
5. Load on each PDU
S_pdu = 3 × 1,551 W
= 4,652 W of 17,292 VA = 27%Against the breaker:
bank breaker, nameplate 20 A
NEC continuous (80%) 16 A
our load 22.5 A
ratio to nameplate 1.12×
ratio to continuous 1.41×And the number that reframes the whole incident:
Fraction of each PDU we were using 4,652 / 17,292 = 27%
Fraction of one bank breaker 22.5 / 20 = 112%We passed the test we were watching, at a quarter of capacity, and failed the one we were not.
But 112% raises a harder question than 225% would have. It should not have tripped at all.
The band where breakers make their own decisions
A thermal-magnetic breaker does not have a threshold. It has a curve, and UL 489 only pins down two points on it:
100% must carry indefinitely without tripping
135% must trip within one hour
200% must trip within ~2 minutes
500%+ magnetic element, instantaneous
├──────────────┬─────────────────┬──────────────┤
100% 112% 135% 200%
▲
└── us. Nothing is specified here.Between 100% and 135% the standard says nothing. A breaker in that band may hold for hours, or for months, or never trip at all. Manufacturers publish typical curves, but the tolerance is wide and the real behaviour depends on how long the overload persists and how warm the device already is when it starts.
That is the whole explanation for the failure signature. A 225% overload would have tripped on every training run, immediately and repeatably, and we would have found it in an afternoon. What we had instead was a breaker sitting just past the point where it is required to hold and well short of the point where it is required to open — which means whether it tripped on any given day came down to how long the job ran and what else was happening in the cabinet.
Two trips nineteen hours apart, then nothing for a week. Not a pattern you can reproduce on demand, which is exactly why it was hard to believe the problem was electrical at all.
At idle the chassis draws about 1,750 W. That is 8.4 A total, roughly 2.8 A per bank, and it will sit there forever without complaint. Between incidents everything looked healthy, both trips followed load rather than power-on, and every instinct said to go look at what changed in software.
Nothing had. The wiring had been marginal since the day it was installed. It just took a long enough training run to prove it.
The fix, and the part that nearly fooled us
Three cords per side. The only question is how many banks they land on.
cords/bank current vs 20 A vs 16 A continuous
3 22.5 A 112% 141% ✗
2 15.0 A 75% 94% ⚠
1 7.5 A 37% 47% ✓Two per bank clears the nameplate but sits at 94% of the continuous rating, and our highest-drawing chassis peaks at 9,553 W, which pushes it to 96%. That is not margin, it is a rounding error.
One cord per bank, three banks per side, clears everything with room to spare.
SERVER A-SIDE PDU B-SIDE PDU
PSU1 ──────────────────► bank B 7.5 A ✓
PSU2 ──────────────────► bank D 7.5 A ✓
PSU3 ──────────────────► bank F 7.5 A ✓
PSU4 ─────────────────────────────────────────► bank B 7.5 A ✓
PSU5 ─────────────────────────────────────────► bank D 7.5 A ✓
PSU6 ─────────────────────────────────────────► bank F 7.5 A ✓
────────────── ──────────────
4,652 W = 27% 4,652 W = 27%And now the part that nearly fooled us, because it is not a breaker problem at all.
Whatever you do with the banks, the cords are still split three and three across two PDUs. Ask what happens when an entire side fails — a feed, a main breaker, a maintenance window that goes wrong. Three supplies carry the whole chassis. Go back to the datasheet line from earlier:
One PDU side lost:
surviving supplies 3
capacity each @ 207 V 2,880 W ← not 3,000 W
total surviving 8,640 W
peak load 9,304 W
────────
shortfall -664 W ✗ chassis drops anywayAt 3,000 W apiece it would survive. At 207 V it does not. That 120 W per supply, buried in a voltage range on a datasheet, is the difference between an N+N configuration and one that fails over into a second outage.
The fix is not electrical. Cap the chassis below what three supplies can carry:
3 × 2,880 W × 0.9 = 7,776 W
set the cap at 8,000 W (we used a chassis-level DCMI limit)Which is the one place in this whole incident where a power cap was the right answer — not to protect a breaker, but to make a redundancy claim true.
What to check before you need to
Count the cords per bank, not the watts per rack. A 17 kW PDU means nothing if a quarter of it lands on one 20 A breaker. Your capacity number and your failure mode live at different levels of the tree, and only one of them is on the dashboard you are watching.
Treat anything over 100% of a branch breaker as already broken. UL 489 only guarantees a hold at 100% and a trip by 135%. Between those, you are not running with margin, you are running on luck, and the failure will be intermittent enough that you will spend days blaming software.
Read your PSU's voltage-dependent rating. At 207 V our 3,000 W supplies are 2,880 W supplies. That 120 W per unit is the difference between a redundant configuration and one that fails over into a second outage.
If your PDU is managed, use it. Ours does outlet-level metering, monitoring and switching on all 42 outlets, with SNMP and per-bank alarms. Every question we spent three days inferring — which bank each cord is on, what each draws under load, how the working chassis are distributed — was available over the network the entire time. Had a 16 A per-bank threshold been armed, we would have had warning weeks before the first trip instead of a post-mortem after the second.
Check a working neighbour before you theorise. We spent real time on BMC power-capping APIs, NVIDIA's Redfish tree, and raw DCMI commands, chasing a way to hold the chassis under some ceiling. Meanwhile three identical servers in the same rack were pulling 8.5–9.5 kW all week without a single trip. Same hardware, same draw, cords spread across banks.
That comparison was available on day one. It turned a three-day mystery into a five-minute fix, and it did not require a single API call.
Nothing was wrong with the server. Nothing was wrong with the power supplies — six Titanium units at 49% of rating, all reporting healthy. Nothing was wrong with either PDU, each running at 27% with four empty banks.
Three cords were on one breaker instead of three.