What Particle Accelerators and Water Utilities Both Learned About Single Points of Failure

The Problem, As Utilities Describe It

Municipal water and wastewater SCADA systems have historically been server-centric: all data acquisition, visualization, and control logic runs through a single central server. That architecture is simple to maintain and license, but it comes with an obvious weakness — that server is a single point of failure. One water district reported an 18-hour outage when its SCADA server failed, forcing operators to run the plant manually while backups were restored.

The industry's response has been a real, active shift toward virtualized, redundant infrastructure. DC Water, for example, now runs dual, physically separated edge computing platforms specifically so that a failure at one site fails over to the other with no interruption — a configuration that reportedly cut failover time by roughly 90% compared to their prior setup. Other utilities have made similar moves, pairing virtualization with redundant historians and synchronized failover across separate physical hosts.

The Same Problem, Solved Earlier, In a Very Different Field

I've spent years building exactly this kind of infrastructure — just not for water treatment.

At Brookhaven National Laboratory's NSLS-II, I led the transition from physical servers to a fully virtualized infrastructure supporting accelerator operations, and at Applied Materials I architected the Unified Server project, combining Proxmox virtualization with RHEL-based hosting for production control systems. The motivation was identical to what utilities are describing now: a live accelerator, like a live water system, cannot tolerate a single hardware failure taking down operations. Redundant, virtualized infrastructure isn't a nice-to-have in either domain — it's the only way to keep a safety-critical, always-on system running when a single physical machine inevitably fails.

Why This Is Convergence, Not a New Discovery

I want to be precise about the claim here, because I think it's more interesting than "hidden connection nobody's found."

This isn't a case where one field is sitting on an unsolved problem the other field already fixed — unlike the alarm fatigue comparison I wrote up recently, water utilities are already actively adopting virtualization and redundancy, with real case studies and measurable results. What's genuinely interesting is that two fields with almost nothing in common — high-energy particle physics and municipal water infrastructure — independently converged on the same architectural answer, for the same underlying reason: systems that can't tolerate downtime eventually all arrive at redundant, virtualized infrastructure, regardless of what they're actually controlling.

That's not a hidden insight. It's evidence that the principle is correct — when two unrelated fields facing the same constraint (zero tolerance for downtime) independently land on the same solution, that's a stronger validation of the approach than either field's experience alone.

Why This Kind of Comparison Is Still Worth Making

Not every cross-domain comparison holds up to scrutiny. I've run a few of these now — one comparing accelerator vacuum diagnostics to pharmaceutical freeze-drying turned out to already be well-known within the vacuum equipment industry; the alarm fatigue comparison held up as a genuinely underexplored parallel. This one sits in between: not a hidden gap, but a real, useful validation of infrastructure decisions that engineers in either field might otherwise second-guess.

If you're managing infrastructure for a system that genuinely cannot go down — whether it's a beamline, a treatment plant, or something else entirely — the fact that two completely unrelated industries reached the same conclusion independently is worth knowing.

Rob Rainer is Director of Controls & Electrical Engineering at Applied Materials, and spent over 15 years in controls and accelerator operations at Brookhaven National Laboratory's NSLS-II, including as Senior Technology Engineer, Lead Operator and Work Control Coordinator.

← Back to Articles