Trust as the product

On July 12, 2026, I found out my own server was lying about its confidence. Not to a user, not on purpose, but a real server, in a real endpoint, was handing back numbers between 0.769 and 87.9 as “uncertainty” for a prediction head that had never been trained to produce uncertainty at all. The documentation said the server refused to serve that number. The code did not refuse. That gap is the most dangerous thing this kind of system can do.

I want to walk through why that incident, and not any accuracy number, is the core of what I’m building.

A neural surrogate that always answers is a liability. It will confidently interpolate garbage on inputs it has never seen, and there is no way to tell the good answers from the bad ones by looking at them, they look identical: a clean field, a smooth colour map, a number with three decimal places. The only way to make that safe is to build a system around the model that can say “I don’t know” and mean it, and then actually check the ones it’s unsure about against something that can’t lie: a real solver.

That’s the trust stack. Six layers sit between the raw prediction and anything downstream sees. They check the prediction against physics itself, PDE residuals, conservation laws, physical bounds, without ever asking a solver for the “correct” answer. As a sanity check on that machinery: feed it a synthetic field where the true divergence is exactly 2, and the layer reports 2.0000. The math checks out on cases where we know the answer, which is the minimum bar before trusting it on cases where we don’t.

The part I don’t get to skip

Here’s the incident again, in full, because burying it would defeat the point of this post. The uncertainty-quantification head in this architecture is not trained. It has no calibration, no ground truth to learn from yet. The correct behaviour is: refuse to serve a number, return a typed error, HTTP 409, uq_unavailable. Until July 12, the server didn’t do that. It served numbers from an untrained head as if they meant something. Nobody downstream would have known the difference between a calibrated uncertainty and a random one, because both look like a float.

I fixed it, and then I did the thing that actually matters more than fixing it: I pinned the correct behaviour with a test that fails if the bug comes back (test_uq_overlay_is_refused_while_the_uq_head_is_uncalibrated). A bug you fixed once and didn’t pin is a bug you will ship again.

The broader rule this taught me: a missing field must be a refusal, never a fabricated zero or a fabricated confidence. If the bridge to the real solver is unavailable for some reason, the code raises an error. It does not manufacture a plausible-looking truth. An invented number is worse than no number, because no number tells you to go check, and an invented number tells you not to.

And even the honest parts aren’t finished being honest

We built an escalation signal from something simple: run two seeds of the model on the same input and see if they disagree. Disagreement should predict error. We measured it. The correlation between two-seed disagreement and true error is 0.265. The false-alarm rate is between 47% and 57%. Half the time, roughly, the system flags a case as uncertain when it was actually fine, or misses a case that needed a second look.

That’s not a calibrated trust signal yet. It’s a measured, honestly-reported, currently-weak one. I’d rather tell you the number is 0.265 than round it up to “the model knows when it’s wrong,” because it mostly doesn’t, yet.

The last line of defence is the one that can’t be gamed by the model at all: the MOOSE solver gate. When a proposed change or prediction doesn’t actually improve on the baseline, the gate has rejected it live, in production, not in a demo. That’s the design principle in one sentence: the surrogate proposes, the solver disposes.

I’d rather ship a system that admits what it doesn’t know than one that always has an answer. Where’s the line for you, between “fast and mostly right” and “slow and provably right”?