Fast Code Isn't Trustworthy Code: A Controls Engineer's Take on the AI Coding Debate

The Claim vs. The Evidence

There's a narrative going around that AI has become good enough at writing code that some companies are pushing their engineers to stop writing it by hand altogether. Some organizations genuinely are mandating AI tool usage — JPMorgan Chase, for example, has tied AI integration directly to software engineers' performance ratings.

But the underlying premise — that this shift is justified because AI-generated code has become reliably trustworthy — doesn't hold up against the actual research.

A February 2026 study from AI research lab METR found that, contrary to developers' own self-reported sense of being faster, AI coding tools can actually slow experienced developers down once you account for the time spent finding and correcting errors in AI-generated output. Separately, researchers examining large volumes of pull requests found AI-generated code produced roughly 1.7 times more defects than human-written code, with a higher rate of critical and major issues, including security problems like improper credential handling.

Some companies are responding accordingly. Amazon reportedly implemented a policy requiring human approval before AI-generated code can be merged, specifically because of reliability concerns. Godot, the open-source game engine, went further and banned substantial AI-generated code contributions outright.

Why This Looks Familiar

I've spent my career in a field that never had the luxury of treating "it works" and "it's safe to trust" as the same question.

Accelerator personnel and equipment protection systems don't get to assume a sensor reading, an interlock signal, or a control output is correct just because it was produced quickly or looks plausible. Every safety-critical decision path is built around independent verification: redundant sensors, cross-checked logic, and a design philosophy that assumes any single component — hardware or software — can be wrong, and builds in a way to catch it before it matters.

That discipline exists precisely because speed and correctness are different properties of a system, and conflating them is how failures happen. A control system that responds instantly but occasionally acts on bad data isn't actually functional — it's just fast and wrong, which is often worse than slow and right.

The AI coding conversation is running headlong into the same lesson. Fast code generation is a genuine capability. Whether that code is architecturally sound, secure, and maintainable is a separate question that generation speed says nothing about — and treating the two as equivalent is exactly the kind of reasoning that safety-critical engineering was built to prevent.

The Part That Should Concern Engineering Leaders

There's a legal dimension here that's easy to miss. U.S. Copyright Office guidance holds that works generated solely by AI, without meaningful human authorship, are not eligible for copyright protection. Some legal commentary has gone further, raising the question of what happens when code in production is not fully understood by the humans responsible for it — a gap researchers have started calling "comprehension debt" or "epistemic debt."

That's a familiar shape of problem to anyone who has worked around configuration control and change management in a regulated engineering environment: a system nobody can fully explain is a system nobody can safely certify, maintain, or take responsibility for when something goes wrong. "The AI made the mistake" is not likely to hold up as an engineering or legal defense, any more than "the sensor was wrong" excuses a facility from having redundant verification in the first place.

What I'd Actually Recommend

None of this is an argument against using AI coding tools — I use them myself, including in modernizing legacy accelerator control systems. The argument is against treating AI-generated output as inherently trustworthy simply because it was fast to produce.

The organizations getting this right, based on what's publicly reported, are the ones treating AI-generated code the way a controls engineer treats any single unverified input: useful, often correct, but never trusted without independent review. Mandatory human review, documentation of design intent alongside generated output, and genuine understanding of what's being deployed aren't friction to be optimized away — they're the verification layer that separates a fast system from a trustworthy one.

That's not a new idea. It's just being rediscovered, one production incident at a time, by an industry that's used to moving faster than the safety-critical world ever could afford to.

Rob Rainer is Director of Controls & Electrical Engineering at Applied Materials, and spent over 15 years in controls and accelerator operations at Brookhaven National Laboratory's NSLS-II, including as Senior Technology Engineer, Lead Operator and Work Control Coordinator, overseeing safety-critical interlock and verification systems.

← Back to Articles