A control-system restart can restore communications while leaving important questions unanswered. Are the measurements current? Does the restored configuration match the equipment? Which controller has permission to act? I think recovery deserves the same engineering attention as normal operation, because the transition back into service is part of the machine’s behavior.
A green display can hide an unfinished recovery
Consider an illustrative cooling loop supporting a precision instrument. The supervisory input/output controller, or IOC, restarts while the local controller remains powered. A few seconds later, the process variables reconnect. Temperatures appear on the display, the pump reports running, and the communications alarm clears.
Meanwhile, maintenance has changed a valve position. One temperature value has not been refreshed since the interruption. The IOC restores a previous setpoint, while the local controller retains a different operating mode. Each component can be functioning as designed while the combined system is still unready to resume its previous task.
This example is hypothetical, but it exposes a useful design question: what evidence must exist between reconnection and permission to operate? A reconnect event answers whether an interface is available. It does not, by itself, reconcile the physical process with the software’s assumptions.
Recovery starts with what survived
Different failures leave different parts of a system intact. An IOC restart may leave a motor controller executing its last command. A controller power cycle may lose position knowledge while the IOC retains a saved value. A network interruption may hide equipment that has continued running normally. A host failure may interrupt several supervisory services at once.
Those cases need different responses. Before designing a restart sequence, I would identify which component failed, what state could have changed during the interruption, and which information remains authoritative. If the cause or scope is uncertain, the recovery logic should expose that uncertainty and withhold the affected authority until it is resolved.
The appropriate holding condition depends on the process. In some systems, essential cooling must continue throughout a supervisory failure. In others, particular commands must remain inhibited. Recovery design should preserve the behavior established by the hazard analysis and local protection design, including any independent safety functions.
Saved settings describe an earlier state
Persistence is useful, but its contents need an explicit meaning. A saved temperature target records a previous request. A saved motor coordinate records a value associated with a previous position reference. Neither establishes the equipment’s present condition.
EPICS autosave makes this distinction concrete. Its documentation separates restoration before record initialization from restoration afterward, and explains that runtime restoration can have different processing effects from boot-time restoration. The restore phase and device-support behavior therefore matter; a save file is not a complete recovery policy. [1]
For each persisted value, I would ask whether startup should restore it, read it from the controller, compare it against an approved configuration, or require a deliberate operator action. Restarting a service should not silently replay a command whose original context has expired.
Measurements must earn their place again
Reconnection should trigger an assessment of measurement validity. A plausible number needs a known source, acceptable quality, and freshness appropriate to the decision it supports. The freshness threshold for a slow thermal trend may differ substantially from the threshold for a fast permissive.
A recently received message can contain an old measurement. A recently assigned timestamp may describe record processing rather than a new physical acquisition. When comparing timestamps from different devices, the design also has to account for clock alignment and timestamp meaning. These details determine whether the recovery sequence is evaluating the current machine or a convincing picture of its past.
In the cooling example, “pump running” alone might be insufficient to release the instrument. The approved sequence could require valid flow feedback, an acceptable temperature condition maintained for a defined interval, the expected controller mode, and confirmation that relevant maintenance restrictions have been cleared. Exact thresholds and durations belong to the equipment requirements and commissioning evidence.
Make the transition into operation explicit
I would describe recovery as a small sequence with observable entry conditions, timeouts, and a defined response when a condition fails. One possible structure is:
- Establish the holding condition. Preserve required local services and protections. Restrict affected supervisory commands according to the approved recovery policy.
- Reestablish trustworthy observation. Identify the connected equipment and obtain measurements with acceptable quality and freshness.
- Reconcile configuration and physical state. Compare the active configuration, controller mode, command/readback relationship, and equipment state. Resolve discrepancies before proceeding.
- Declare readiness. Evaluate the required evidence and identify the authority permitted to release operation. Display any remaining blocking conditions.
- Resume and verify. Resume under the approved procedure and confirm the expected process response. A failed check returns the system to the defined holding or fault condition.
This is an illustrative supervisory sequence, not a replacement for machine protection or a facility-specific safety design. Its purpose is to make the return to operation inspectable. If flow feedback becomes stale during validation, for example, readiness must be reevaluated rather than carried forward from an earlier successful check.
Commission the interrupted recovery
A successful cold start proves only one path through the design. Recovery testing should also examine partial failures and interruptions. These tests belong in simulation, on a suitable test system, or within approved commissioning arrangements for the real equipment.
| Test condition | Expected evidence or behavior |
|---|---|
| IOC restarts while the local controller remains active | Reconcile retained mode and output state; do not assume the equipment stopped. |
| Communications return with stale measurement data | Keep the affected readiness condition unproven until valid fresh data arrives. |
| Saved configuration disagrees with approved configuration | Expose the discrepancy and apply the defined resolution policy. |
| A second failure occurs during validation | Return to the defined holding condition and repeat the checks that lost validity. |
| A command issued before the interruption arrives late | Reject expired or invalid-context commands through the designed command interface. |
| Operation resumes but the process response is wrong | Detect the failed acceptance condition and execute the approved response. |
The record of a recovery should explain which configuration was active, what failed, which checks passed, and why operation was released. Where acknowledgement or authorization is required, it should record who acted. That evidence helps the next engineer distinguish a deliberate recovery from a machine that merely began responding again.
Recovery is also a team responsibility
Controls, infrastructure, equipment, and operations teams each see a different part of the event. A host administrator may verify that a virtual machine is healthy. A controls engineer may verify IOC behavior. An operator may recognize that the physical process has not settled. An integrated acceptance plan has to connect those perspectives and assign responsibility for resolving conflicts.
A procedure is particularly useful when someone other than its author can explain what is blocking recovery, gather the required evidence, and carry out the permitted next step. That is a practical test of both the design and the knowledge transferred with it.
This fits the approach I explore in The Observable Machine and through Control Lattice Systems: make system behavior understandable enough that people can make defensible engineering decisions. Recovery is a good place to apply that discipline. The machine should be able to show what it knows about its present state, what remains uncertain, and why it is ready to proceed.
Related Control Lattice resources
- Machine Observability Review — a scoped assessment of alarms, sequence state, timestamps, operator displays, logs, and difficult restarts.
- Free Machine Failure Evidence Checklist — capture useful operating evidence before resetting, restarting, or changing a system.
- Controls Commissioning Toolkit — organize checkout, acceptance, alarm, interlock, punch-list, and signoff records.
Reference and related reading
- EPICS autosave documentation — sections on boot-time and runtime restoration and restore passes.
Related portfolio essay: One Device, One Failure Domain: Modernizing an EPICS Radiation-Monitoring System.