One Device, One Failure Domain: Modernizing an EPICS Radiation-Monitoring System

The Failure Was Rarely “The Network”

In an advanced scientific facility, a device that disappears from an operator screen can look like a network problem even when the network is working exactly as designed. During my work with radiation-monitoring controls at Brookhaven National Laboratory, I taught technicians to begin with three possible fault domains: the physical device and its interface hardware, the network path, or the EPICS input/output controller.

The surrounding evidence usually narrowed the search quickly. When adjacent IOCs and neighboring devices using the same network switches remained healthy, a common switch or network-path failure became less likely. When one monitor failed while the rest continued communicating, the evidence pointed toward that unit or its onboard communications interface. In practice, the recurring problem was often an aging embedded Moxa interface rather than the shared network or IOC server.

The Original Architecture Coupled Unrelated Lifecycles

The network interface was installed inside the radiation monitor. That arrangement appeared compact, but it tied together two components with very different maintenance needs. When a monitor left the facility for calibration, its network interface went with it. When the calibrated or replacement unit returned, its address and communications configuration had to be established again.

This provisioning step became a bottleneck. The embedded interfaces were increasingly difficult to configure, and a calibration activity that should have concerned the instrument also became a controls and networking task. The equipment was physically available, but operations could still wait on a deprecated communications component and a fragile IP-assignment process.

The EPICS deployment created a second concentration of risk. A single IOC served all of the monitors. It provided a common implementation, but a restart or service interruption temporarily removed communications for the entire group. Standardization had been achieved by sharing the runtime rather than by sharing a controlled design.

Decouple the Instrument From the Network Endpoint

We moved the Ethernet conversion out of the instruments and installed modern serial-to-Ethernet gateways in the equipment racks. Each monitor could then communicate over RS-485 to a stable, maintainable gateway. The gateway—not the removable instrument—owned the persistent network endpoint.

That separation changed the maintenance workflow. A monitor could leave for calibration without taking the facility's network configuration with it. A returning device connected to the existing serial interface, and the rack gateway preserved the Ethernet-facing identity. The result was much closer to plug-and-play replacement and eliminated a repeated provisioning dependency.

Standardize the Design, Not the Failure Domain

The software architecture followed the same principle. I developed a common template for the HPI6016 device driver, executables, binaries, libraries, startup behavior, and supporting configuration. Device-specific information, including endpoint addresses, was passed into a smaller IOC instance through environment variables.

Each monitor therefore used the same reviewed implementation without requiring every monitor to share one running IOC. An individual IOC could be restarted, diagnosed, or updated without interrupting communications to every device. Common code reduced maintenance variation; separate runtime instances reduced operational blast radius.

Commission the Failure Behavior, Too

Successful commissioning cannot stop when the device produces a plausible reading. The important tests include loss and restoration of serial communications, gateway disconnect and reconnect, IOC restart, virtual-host recovery, stale data, invalid alarm state, incorrect endpoint configuration, command/readback disagreement where applicable, and replacement after calibration.

This is where failure modes and effects analysis is useful. For each credible failure, the team asks what the operator will see, how the condition will be distinguished from neighboring faults, what evidence the IOC and archive will retain, what recovery is permitted, and how much of the system is affected. The modernization reduced both the probability of repeated provisioning failures and the consequence of an IOC-level interruption.

The supervisory EPICS architecture described here should not be confused with an independent credited protective function. The exact safety role of radiation-monitoring equipment and its interfaces depends on the facility design, authorization basis, and approved procedures. The improvement documented here concerns operational communications, maintainability, fault isolation, and availability within that governed system.

A Controls Project Became a Workforce Project

I involved several interns in this work. The project gave them something more valuable than an isolated programming exercise: a real system with hardware boundaries, serial communications, network dependencies, EPICS records and drivers, virtualized services, configuration management, commissioning, and operating consequences.

They learned that engineering is not finished when software compiles. A useful implementation must survive calibration cycles, restarts, replacements, handoffs, and troubleshooting by someone who did not write it. Several of those students later obtained jobs at Brookhaven. Their progression reinforced a lesson I have carried into technical leadership: modernization projects should leave behind both a more maintainable system and more capable people.

The Broader Pattern

This architecture applies well beyond radiation monitors. Whenever replaceable field equipment carries its own network identity, routine maintenance can accidentally become a networking event. Whenever many devices share one supervisory process, a convenient implementation can become an oversized failure domain. Whenever every deployment is handwritten, apparent flexibility becomes configuration drift.

The alternative is a layered contract: keep durable infrastructure stable, make field equipment replaceable, standardize reusable software, inject explicit instance configuration, and isolate runtime failures to the smallest practical unit. That is not merely an EPICS technique. It is a general method for making complex systems easier to operate, commission, and trust.

Rob Rainer is Managing Director & Principal Engineer at Control Lattice Systems. He has more than twenty years of experience spanning accelerator operations, EPICS controls, scientific instrumentation, resilient infrastructure, electrical engineering, commissioning, and technical leadership, including over fifteen years at Brookhaven National Laboratory's NSLS-II.

← Back to Articles