The math under the surrogate

I made a list of every formula my codebase claims to implement, and then I made three different tools check each one independently. The list has 174 entries. Not all of them survived clean. That’s the part worth talking about.

It’s easy to trust a neural network because it produces numbers that look like physics. Smooth fields, plausible units, colours in the right range. None of that tells you whether the underlying claim, the actual equation the code says it’s solving, is the equation the code actually solves. So over the last week of July I built what I’ve started calling a math stamp: a formula-audit pipeline that harvests every formula-bearing claim in the repository and checks it three separate ways.

Leg one: symbolic. Every formula gets phrased as a query to Wolfram Alpha’s Full Results API, an independent, outside authority with no stake in my code being right. 148 queries succeeded.

Leg two: derived. A sympy and numpy script re-derives the algebra from first principles, symbolically, and checks it against what the code claims. 35 checks, 35 pass.

Leg three: numeric. The actual test suite, run cold, on the actual code. In the newest revision alone, 996 tests passed, 85 skipped. Across the rest of the repository, 563 passed, zero failed.

Here’s the honest part: three legs agreeing is rare, and I say so

Of the 174 claims, only 31 are verified by all three independent methods at once, what I labelled VERIFIED-3WAY. Another 62 are confirmed two ways. 16 have only numeric confirmation (the tests pass, but no outside symbolic tool weighed in). 44 have only symbolic confirmation. 10 are cited but currently unverifiable with the tools I have. And 9 are flagged GAP, meaning I could not verify them at all, and I name every one individually rather than averaging them away: FSI coupling, learned quadrature, the battery electrochemistry model, a correlation claim about local feature size, and a bound-coverage statistic among them.

I want to be blunt about why this matters more than a clean “100% verified” headline would. A stamp that claims everything is verified is a stamp you shouldn’t trust, because verification this thorough always finds gaps. The document explicitly does not claim full verification. It names what’s missing.

The two findings I’m proudest of are both about something that isn’t there

Two entries in the catalogue are labelled NOT-A-CLAIM. Not “wrong formula”, not “unverifiable formula”, but no formula at all where one was implied. One is a transport layer in the code with no stated governing PDE behind it. The other is a set of 26 YAML physics specification files that describe physics configurations without ever writing down the governing equation they’re supposed to encode. Neither is dangerous by itself, code can be structurally fine without a formula attached to every file, but the absence is the finding: it tells me exactly where to look before anyone builds on top of that layer assuming a derivation exists.

The audit also caught real structural limits, not just missing paperwork. Wolfram Alpha has no opinion on whether a CutFEM weak-form discretisation is correct, that’s a numerical analysis question no symbolic tool can adjudicate, so 44 of the claims are symbolic-only by necessity, not by laziness. A test directory referenced by 21 docstrings in the differentiable-simulation code, tests/test_cutfem/, does not exist in the tree. I recorded that as a finding, not swept it under an “omission.” And a label I’d used casually, calling a diffusion process “63% complete at the characteristic time,” turned out to be an approximation; the real number sits closer to 0.70. Small, but it’s exactly the kind of small thing that compounds if nobody checks it.

None of this makes the underlying engineering weaker. It makes the claims about it trustworthy in a specific, falsifiable way: you can go re-run the same three legs yourself and get the same answer, or catch me if you don’t. That’s the actual point of “stamping” a machine learning system, not a certificate that says trust me, but a trail that lets you not have to.

If you build ML on top of physics, how do you currently verify the equations under the hood, or do you take the model’s word for it?