FP8 & Precision

Gradient Underflow in Reduced Precision Training

FP8 has a 4-bit exponent. That’s the entire problem. Everything else — the quality loss, the instability, the silent failures — is a symptom of that one architectural fact.

⏲
In progress
Full write-up coming soon
The bit-level breakdown is written. Cleaning up the diagrams now.
Built on this research

Everything in this article is measured, not promised. See it running:

Talk to NOESIS liveSee the 33x benchmarkBack the research