Research & Insights

Notes from the
engineering floor

Practical write-ups on the failure modes we see most often in scientific ML codebases, and how we address them.

Numerical Stability
Why BF16 rollouts drift after step 10,000
Reduced mantissa precision compounds through long autoregressive rollouts. We outline where to keep FP32 accumulators and how to assert bounds in CI.
Discuss →
GPU Kernels
Reading a coalescing report without guesswork
A walkthrough of the five profiler metrics that explain most custom-kernel slowdowns, and the stride fixes that typically follow.
Discuss →
Neural Operators
Choosing spectral modes in Fourier Neural Operators
The trade-off between resolution invariance, memory, and accuracy, with guidance for turbulence and climate surrogates.
Discuss →
Engineering Practice
Property-based tests for physics invariants
Using Hypothesis to check conservation laws and boundary conditions that example-based unit tests rarely reach.
Discuss →