Weight Folding, CUDA Streams, and the Bug That Made My Model Speak Backwards — Filip Makraduli

An RMS norm layer does almost none of the arithmetic in a transformer, yet a single decode step can launch it around 33 times, and a GPU is fast at math and slow at everything else: starting work, moving data, waiting. That gap is what FlashNorm attacks. Filip Makraduli wrote the paper with Nils Graef, and the idea fits in two lines of algebra. Fold the norm's gain into the projection weights offline so one matrix absorbs both. Defer the scalar divide so the matrix unit and the vector unit run at once instead of one idling for the other. And in newer architectures that normalize twice in a row, drop one, because the operation is scale invariant and the second adds nothing. Together they buy a 33 to 35 percent speedup on the norm plus projection operation, and the folded checkpoint works with torch compile and quantized models. The deferral is where it got interesting, because you cannot do it from Python. He wrote the CUDA to run the matmul on tensor cores and the RMS reduction on CUDA cores in parallel. Unit tests passed, perplexity looked normal, and then over a long generation the model began repeating itself with a one step lag, outputs from the past. The join between the two streams was implicit. One stream had not finished, so the post scale read a stale buffer from an unfinished multiply. The fix was to mark the end of each stream explicitly and make the post scale wait on both. He closes on why the second half needed Superlinked's open inference engine to deploy a modified checkpoint, since you cannot do kernel surgery on a rented endpoint. Speaker info: - https://x.com/f_makraduli - https://www.linkedin.com/in/filipmakraduli/ - https://filipmakraduli.substack.com/ Timestamps: 0:00 - Two lines of algebra for RMS norm 2:07 - Why a layer with no math costs so much time 3:44 - Weight folding, deferred division, dropped pre norm 6:33 - The output that came from the past 7:25 - Tensor cores and CUDA cores in parallel 8:45 - An implicit join and a race condition 9:58 - Making the post scale wait on both streams 10:42 - Results on open models, and what works out of the box 13:54 - Why you cannot do kernel work on a rented endpoint 16:00 - Paper, repo, and where to find him