The honest negatives

Most of what I’ve published this year is a claim that didn’t survive contact with the evidence. I keep publishing them anyway.

If a lab only shows you the wins, you have no way to judge whether its wins are real. So here are seven things that failed, in the order we found them, and what each one taught us.

The topology wall that was partly a measuring stick

For a long time we had a number: a “wall” in how well the model could reproduce complex topology. It felt like a hard architectural limit. Then we looked closer at how the metric counted failure — it counted connected components raw. A meaningful fraction of the “detached fragments” turned out to be zero-volume specks: numerical noise, not real disconnected geometry. The wall was partly a broken ruler, not a broken model. We didn’t fix the number. We fixed the instrument, then measured again. Whether what’s left is a data problem or an architecture problem is still genuinely open — we don’t have a verdict, and we’re not claiming one.

The experiment that measured the wrong thing

We ran a test meant to answer: does using real CAD geometry (instead of synthetic shapes) break the topology wall? It looked like a clean result. Then we noticed the real-CAD corpus had 8,192 points per shape, versus 200,000 on the baseline it was compared against, with different resampling behavior per epoch. We weren’t measuring data quality. We were measuring point-cloud density. We threw the result out and redesigned the experiment. This is now a standing rule for us: reconcile corpus parameters before comparing two models, or you will get a confident, entirely false answer.

The precision trap that made an FEM result arbitrary

A single finite-element solve with penalty-enforced boundary conditions can have a condition number around 2.7e12. Standard float32 arithmetic carries about 8.4e6 — six orders of magnitude short. Below that threshold, the solved result isn’t slightly wrong, it’s arbitrary, and the displacement-driven patch tests we were using to sanity-check it were structurally blind to the problem, because they don’t probe conditioning. The fix was mechanical: force double precision on every solve path, and add an explicit equilibrium residual guard so a future regression can’t hide the same way. After the fix, error against the analytic solution dropped to 0.4-1.2%.

Zero-shot transfer that simply didn’t happen

We had a mesh-graph refiner trained on one bracket dataset (DeepJEB) that measurably improved peak-stress accuracy in-distribution. We then tested it, untouched, on a different but related dataset (SimJEB). Result: worse on 27 of 40 brackets. Part of the reason is that the underlying surrogate itself is out-of-distribution there (error around 0.573). The honest conclusion isn’t “transfer learning doesn’t work” — it’s narrower and less flattering: this particular refiner is an in-distribution tool, and moving to a new domain needs its own refiner, trained on that domain’s own truth. That’s a real cost, not a universal promise we get to skip.

A sampling strategy that climbed the wrong hill

We tried gradient-ascent sampling as a way to find worst-case or high-value design points faster. Tested, and it was negative — it reliably climbed toward the wrong optimum. We logged why, we didn’t retry it with a different learning rate hoping it would come good, and we moved on to a method that actually worked.

A trust signal too weak to trust

Our escalation logic uses two-seed disagreement — training two models and checking if they agree — as a proxy for “is this prediction risky.” We measured how well that proxy tracks the true error. Correlation: 0.265. False-alarm rate: 47-57%. That’s a coin flip wearing a lab coat. We say this plainly rather than deploy an escalation policy that looks rigorous and isn’t. It’s measured, and it’s not yet calibrated.

The favourable-draws lesson

Several early published numbers turned out to be true, but lucky. A single run, reported as a point estimate, can land better than the honest average just by chance. When we went back and looked at the distribution instead of the single draw, a headline 0.208 figure turned out to sit inside a band closer to ~0.21. The fix isn’t complicated: report bands, not points, and say so when you don’t yet have enough seeds to build one properly.

The data-scaling knee

We threw 3.2x more training data at one of our hardest metrics. It helped — +5.3 percentage points, moving 0.2268 to 0.1718 along one measurement chain — but that gain was sub-linear against the data increase. That’s the knee of the curve: more rows are no longer the cheap lever. The next lever has to be model capacity or architecture, not corpus size, and we’re treating that as a real constraint on the roadmap rather than something a bit more data will eventually fix.

None of these seven negatives is fatal on its own. What worries me more is the version of this post that doesn’t exist — the lab that has all seven of these problems and publishes none of them. If you can’t see my failures, you have no basis for believing my successes either. That’s the actual argument for publishing this list.

What’s a “negative result” in your own work that you almost didn’t publish, and what changed your mind?