The Problem
Accelerator timing systems depend on holding a laser's phase locked to a master oscillator, often to sub-picosecond tolerances. In the real world, that lock is constantly under attack from things that have nothing to do with the timing system itself: lab temperature drift, fiber temperature changes, external vibration, power supply stability. None of these disturbances are constant, and none of them are fully predictable.
A standard fixed-gain PID controller handles this reasonably well as long as conditions stay roughly the same. The trouble is that the right response to a small phase error and the right response to a large one aren't really the same problem. React too gently to a large error and the system takes too long to recover, risking a lock loss. React too aggressively to normal small fluctuations and the system oscillates, chasing noise instead of settling. A single fixed set of gains is always a compromise between those two failure modes.
What I Built
I wrote a full software simulation of this problem, structured as a set of cooperating EPICS soft IOC processes rather than a single monolithic script — closer to how a real distributed timing system is actually organized:
- envSim.py generates simulated environmental disturbances — lab temperature, fiber temperature, external vibration, power stability — each publishing to its own PV.
- moSim.py models the master oscillator's phase response to those disturbances.
- laserSim.py simulates the laser's phase behavior, including its own drift dynamics.
- control.py implements the actual PID control loop, reading phase error and adjusting correction outputs.
- beam.py independently monitors phase lock status and triggers a beam dump if the error exceeds a configurable threshold.
- A Phoebus GUI ties it all together for live visualization and manual override.
Each piece runs as its own process, communicating purely through EPICS PVs — the same architecture a real timing system would use, just with simulated hardware underneath instead of real oscillators and lasers.
The Adaptive Part
The core of the project is in control.py. Instead of running fixed PID gains, the
controller inspects its own recent behavior every cycle and nudges its gains accordingly:
When the phase error is large, proportional gain creeps up, pushing the system to correct more aggressively. When the error is small, gain relaxes back down, avoiding the oscillation that an overly aggressive fixed gain would cause on minor noise. Integral gain responds to how hard the piezo actuator is already working, tightening up if it looks like accumulated error is building. Every adjustment is bounded, so the system can't wind its own gains up into instability.
The correction itself is split between two actuators with different response characteristics — a fast PLL path handling 60% of the correction, and a slower piezo path handling the remaining 40% — mirroring how real timing systems often pair a fast, limited-range actuator with a slower, wider-range one rather than relying on a single correction path.
The Part I Think Matters Most
The most important design decision in this project isn't the adaptive gain logic — it's that the safety response doesn't depend on it.
beam.py runs as a completely separate process from control.py. It
watches phase error against a threshold and triggers a beam dump if that threshold is exceeded,
regardless of what the control loop is doing, whether the control loop is doing anything at all,
or whether the control loop has a bug in it that day. If the adaptive PID logic misbehaves —
bad tuning, a stuck actuator, an unhandled edge case — the safety monitor doesn't need it to
behave correctly in order to do its job.
That's not an incidental implementation detail. It's the same principle behind real accelerator personnel and equipment protection systems: the component responsible for keeping things safe should never depend on the component being protected against. A simulation is a low-stakes place to build that habit, but it's the same habit that matters at full scale.
Why I Built It This Way
I could have written this as a single script with a control loop and some conditionals bolted on. I didn't, because that's not how real distributed timing systems are built, and a simulation that doesn't reflect the actual architecture teaches the wrong lessons. Running it as cooperating EPICS processes means the simulation exercises the same PV-based coordination, the same process independence, and the same "no single point of failure in the safety path" discipline that a production timing system requires.
The project is public on GitHub for anyone who wants to look at the full implementation.
Rob Rainer is Director of Controls & Electrical Engineering at Applied Materials, and spent over 15 years in controls and accelerator operations at Brookhaven National Laboratory's NSLS-II, including leading an EPICS-based accelerator timing system modernization from legacy VME/RTEMS infrastructure.
Source
- TimingSim on GitHub — full source code for the simulation described in this piece.
ENGINEERING INSIGHT
The component responsible for keeping things safe should never depend on the component being protected against.