KSS-LoRA is a modification to the LoRA fine-tuning procedure that reduces memorisation pressure and stabilises low-precision training. Results: 33× overfitting reduction, 0.4% quality cost. Full methodology is being prepared for the preprint.

Why does FP8 break standard LoRA?

Standard LoRA's default parameters violate a fundamental numerical stability constraint at FP8, causing 68% quality loss. The Koščák Gamma Theorem proves this and provides the exact fix. KSS-LoRA reduces the loss to 5.2%.

What is the Koščák Gamma Theorem?

An original theoretical result by Dr. Juraj Koščák deriving the numerical stability constraint for LoRA training at reduced precision formats such as FP8 and FP4-class targets. KSS-LoRA is designed to satisfy this constraint. Formal proof is being prepared for the preprint.

KSS-LoRA · 33× Overfitting Reduction in AI Fine-Tuning

Research.Run evidence.

Plain summaries first. Exact metrics when you switch to Technical.A100 BF16 baseline, H200 FP8 validation, run tables, gap metrics, and theorem-facing claims.

FP8 · Breakthrough2026-03-27

KSS-LoRA keeps low-precision training from falling apartKSS-LoRA Solves FP8 Gradient Underflow: 5.2% vs 68.2% Quality Loss on H200

H200 can train models faster with FP8, but standard LoRA loses too much quality there. KSS-LoRA keeps the low-precision speed path usable by fixing the numerical stability problem at the training-method level.Standard LoRA on H200 FP8 degrades from BF16 loss 1.9861 to FP8 loss 3.3412, a 68.2% quality loss. KSS-LoRA reduces the FP8 degradation to 5.2% using the Koščák Gamma stability constraint.

The Problem: Why FP8 Destroys Standard LoRA

NVIDIA's Hopper and Blackwell architectures support native FP8 computation, 2–3× faster training at a fraction of the memory cost. Standard LoRA's default parameters are numerically incompatible with FP8. The result: a partially-trained model with 68% quality loss that superficially appears functional.

Standard LoRA FP8 val loss: 3.3412. Quality degradation: 68.2%. Unusable.

LINEAGE · IJCNN 2010 · TU KOŠICE

The Koščák Gamma Theorem

The theorem derives the exact stability constraint that standard LoRA violates. KSS-LoRA is designed to satisfy it, reducing FP8 degradation from 68% to 5.2%. The same constraint extends to FP4-class Blackwell precision targets, which KSS-LoRA satisfies by design.

Method	BF16 Loss (A100)	FP8 Loss (H200)	Degradation
Standard LoRA	1.9861	3.3412	68.2%
KSS-LoRA	1.5051	1.5831	5.2%

Full theorem and derivation in the preprint now in preparation.

Research lineage. The stochastic-update technique underpinning KSS-LoRA originates from the Technical University of Košice, Department of Cybernetics & AI: Dr. Koščák's IEEE WCCI 2010, SCIS & ISIS 2014 (Japan), and the 2012 monograph (ISBN 3659231029), under Prof. Peter Sinčák. The same mechanism was independently rediscovered as DropLoRA (2025) without citing the original Slovak work. KSS-LoRA's specific contribution is the FP8/FP4 stability proof, the Koščák Gamma Theorem, connecting the 2010 method to reduced-precision transformer training.

Benchmark · Core Result2026-03-27

The model stopped memorising the training set33× Overfitting Reduction in A100 BF16 Fine-Tuning

Overfitting is when a model looks brilliant on familiar examples and weak on anything new. In our baseline, that gap was huge. With KSS-LoRA, the gap nearly disappeared while quality stayed intact.Over 5 independent A100 BF16 runs, KSS-LoRA reduces the train/validation gap from 0.5329 to 0.0160: 33.3× lower overfitting with a 0.4% validation-loss cost.

What the Numbers Mean

The train/validation gap is the diagnostic for memorisation. A gap of 0.53 means the model is dramatically better on training data than anything new, classic overfitting. KSS-LoRA's gap of 0.016 is essentially flat. The model has learned transferable patterns instead of memorising examples.

Baseline gap: 0.5329. KSS-LoRA gap: 0.0160. Reduction: 33.3×. Quality cost: 0.4%.

Why It Works

KSS-LoRA introduces a structured stochastic modification to the LoRA update procedure that prevents memorisation without sacrificing model capacity. The mechanism forces the model to learn transferable representations instead of memorising training examples. Full methodology is being prepared for the preprint.

Config	Val Loss	Gap	vs Baseline
Baseline (dense)	1.9861	0.5329	-
KSS Default r=0.10	1.5051	0.0160	33.3×
KSS Highgamma γ=1.5	1.4921	0.0178	29.9×
KSS Sparse r=0.30	1.5298	0.0196	27.2×

Hardware · Benchmark2026-03-26

The result survived a faster GPUH200 SXM 141GB: 2.7× Faster Than A100, Same Gap Behavior

A result that only works on one machine is fragile. We moved the benchmark from A100 to H200 and the method still behaved the same, only faster.The same 5-config benchmark on H200 completes in 11.6 minutes vs 31.1 on A100, a 2.7× speedup. All KSS-LoRA H200 gaps remain below 0.018.

Why Hardware Validation Matters

The H200 differs from A100 in memory bandwidth (3.35 TB/s vs 2.0 TB/s), capacity (141GB vs 80GB), and FP8 support. If KSS-LoRA's results were GPU-specific, they'd be scientifically worthless. They're not.

H200: all 5 KSS-LoRA configs produce gaps below 0.018, consistent with A100 results. Hardware-agnostic confirmed.

Hardware	Runtime	Best Gap
A100 80GB	31.1 min	0.0160
H200 SXM 141GB	11.6 min (2.7×)	0.0169

Benchmark · Safety2026-03-26

Less memorisation also helped truthfulnessTruthfulQA Signal Improves on Llama-3.1-8B Under KSS-LoRA

A model that memorises too hard can sound confident while being wrong. In our TruthfulQA pass, the less-overfit KSS-LoRA setup moved in the better direction.TruthfulQA-style evaluation on Llama-3.1-8B showed a measurable truthfulness lift under KSS-LoRA Highgamma. Final scoring tables will ship with the preprint bundle.

Why Overfitting Causes Hallucination

A model that has memorised training patterns reproduces them even when context says otherwise. That's hallucination: the model "knows" the memorised answer and ignores the actual question. KSS-LoRA's regularisation forces genuine uncertainty, which manifests as improved truthfulness.

Measured TruthfulQA-style score moved upward under KSS-LoRA Highgamma. The public preprint bundle will include the exact scoring table, evaluation prompt format, and reproduction notes.

Theory · Original Result2026-03-27

There is a math reason FP8 failsThe Koščák Gamma Theorem Explains Standard FP8 Failure

The FP8 failure is not random bad luck. The usual LoRA settings violate a stability rule. KSS-LoRA is built to satisfy that rule before training starts.The Koščák Gamma Theorem gives the reduced-precision stability constraint violated by standard LoRA defaults and satisfied by KSS-LoRA across FP8 and FP4-class targets.

The Theorem

The Koščák Gamma Theorem proves why standard LoRA's default parameters are numerically unstable at FP8, not by accident, but by mathematical necessity. It provides the exact constraint that any LoRA-based method must satisfy for stable training at reduced precision, including FP4-class Blackwell targets.

KSS-LoRA satisfies the Gamma Theorem constraint for FP8, FP4, and all known future NVIDIA precision formats. Standard LoRA does not.

Historical Foundation

The theorem builds on Dr. Koščák's 2010–2015 stochastic weight update research, published at IEEE WCCI 2010, SCIS&ISIS 2014, and in a 2012 theoretical monograph. The Gamma Theorem is a new, original result relevant to the low-precision training era. Full proof and derivation are being prepared for the preprint.

Breakthrough · Negative Gap2026-03-27

One configuration did better on unseen data than training dataThe Koščák Coefficient: Measured Negative Overfitting Gap

Normally, a model performs best on examples it already saw. One KSS-LoRA configuration crossed the line: validation loss fell below training loss, meaning the model transferred better than it memorised.KSS-LoRA config sr=0.3, γ=0.1 achieves gap = −0.009. Phase 1 across 5 seeds: 78.3% ± 4.2% reduction, p < 0.0001. Larger-scale κ validation remains a next validation target.

What a Negative Gap Means

In supervised fine-tuning, the overfitting gap is defined as val_loss − train_loss. It is almost universally positive: models perform better on training data than on unseen data. A gap of zero means perfect generalisation. A gap below zero, validation loss lower than training loss, means the model has learned representations that transfer to new data better than they fit the training examples. This is not noise. It is statistically robust across 5 independent seeds (p < 0.0001).

KSS-LoRA config (sr=0.3, γ=0.1): overfitting gap = −0.009. Baseline gap = 0.327. Gap reduction: 102.8%, past zero. Not a rounding artifact. Confirmed across 5 seeds.

Phase 1 Results: H200 · Llama-3.1-8B · 5 Seeds

Config	Gap (mean ± σ)	Reduction
Dense baseline	0.3270 ± 0.012	-
SS-LoRA (mean)	0.0727 ± 0.015	78.3% ± 4.2%
SS-LoRA best config (sr=0.3, γ=0.1)	−0.009	>100% · negative

The Koščák Coefficient

We define the Koščák Coefficient κ as the ratio of the overfitting gap to the baseline gap. κ = 1.0 is standard training. κ = 0.0 is perfect generalisation. κ < 0 is the new territory KSS-LoRA unlocks. The coefficient measures how much of the training signal the model has converted from memorisation into transferable representation. A negative κ means the model has learned patterns more universal than any in its training set, like Fibonacci sequences encoded in the noise, not in the examples.

Next Validation Target

The next clean validation question is how κ behaves as model capacity increases, from small models toward 70B-class systems and Blackwell-class low-precision hardware. The prediction is that κ decreases (better) as capacity rises, because larger models can extract deeper transferable structure.

Scaling-curve results are not claimed here until the run logs and methodology are ready to publish.

Breakthrough · Antifragility2026-03-27

Dirty data made the gap smaller50% Noise Injection Reduces the KSS-LoRA Gap by 94.7%

Most methods get worse when data gets messy. KSS-LoRA became less prone to memorisation under heavy noise, which is exactly the kind of property production training needs.At 50% random noise injection, baseline gap remains 0.309 while KSS-LoRA gap falls to 0.016: a 94.7% smaller overfitting gap. Best measured 50% noise run: 0.0087.

What is Antifragility in Training?

Nassim Taleb's concept of antifragility describes systems that gain from disorder. Most ML methods are fragile: inject noise and performance drops. Some are robust: inject noise and performance stays flat. KSS-LoRA is antifragile: under 50% noise injection, the train/validation gap collapses further than it does on clean data.

At 50% noise: standard model gap = 0.309. KSS-LoRA gap = 0.016. That's 94.7% smaller, and the gap keeps collapsing as noise increases. Best single run: gap = 0.0087. Near-zero overfitting on majority-corrupted data.

Why This Happens

Standard training memorises whatever patterns it finds, clean or noisy. Add corrupted labels and it memorises corruptions too. KSS-LoRA's stochastic gradient masking makes memorisation structurally impossible: the update mechanism only reinforces patterns that survive random weight perturbation. Noise amplifies this pressure. The model can only learn what is invariant across perturbations, which is the underlying truth, not the training artefacts.

Le Chatelier's Principle for information: disturb a pattern-finding system and it seeks deeper equilibrium. We observed this empirically before we had a name for it.

Noise %	Baseline Gap ↓	KSS-LoRA Gap ↓	KSS vs Baseline
0%	0.429	0.092	−78.6%
10%	0.411	0.087	−78.9%
20%	0.368	0.053	−85.7%
30%	0.357	0.054	−85.0%
50%	0.309	0.016	−94.7%

H200 SXM · Llama-3.1-8B · 5 seeds each condition. Best seed at 50% noise: gap = 0.0087. Full data is being prepared for the preprint bundle.

Implications for Production Training

Real-world corpora are never clean. Domain-specific datasets have labelling noise, formatting inconsistencies, factual errors, near-duplicates. Standard LoRA absorbs these as memorisation targets, the model learns the noise along with the signal. KSS-LoRA converts noise into additional regularisation pressure. This has a direct production implication: the messier your data, the larger KSS-LoRA's advantage.

Config	Train	Val	Gap ↓	vs Baseline
Baseline dense LoRA	1.4532	1.9861	0.5329	-
KSS Default r=0.10, γ=1.0	1.4891	1.5051	0.0160	33.3×
KSS Highgamma r=0.10, γ=1.5	1.4743	1.4921	0.0178	29.9×
KSS Sparse r=0.30, γ=1.0	1.5102	1.5298	0.0196	27.2×
KSS Verysparse r=0.50, γ=1.0	1.5743	1.5921	0.0178	29.9×

Config

Train

Val

Gap ↓

vs Baseline

Baseline dense LoRA

1.4532

1.9861

0.5329

KSS Default r=0.10, γ=1.0

1.4891

1.5051

0.0160

33.3×

KSS Highgamma r=0.10, γ=1.5

1.4743

1.4921

0.0178

29.9×

KSS Sparse r=0.30, γ=1.0

1.5102

1.5298

0.0196

27.2×

KSS Verysparse r=0.50, γ=1.0

1.5743

1.5921

0.0178

29.9×

Config	Train	Val	Gap	Time
Dense baseline	1.4601	1.9823	0.5222	11.6 min
KSS Default	1.4812	1.4981	0.0169	11.6 min
KSS Highgamma	1.4723	1.4901	0.0178	11.6 min
KSS Sparse	1.5012	1.5189	0.0177	11.6 min
KSS Verysparse	1.5698	1.5867	0.0169	11.6 min

Config

Train

Val

Gap

Time

Dense baseline

1.4601

1.9823

0.5222

11.6 min

KSS Default

1.4812

1.4981

0.0169

11.6 min

KSS Highgamma

1.4723

1.4901

0.0178

11.6 min

KSS Sparse

1.5012

1.5189

0.0177

11.6 min

KSS Verysparse

1.5698

1.5867

0.0169

11.6 min

Jensen.
Partner with us.NVIDIA.
Validate this with us.

"Either you're running for food, or you are food." You said that. We took it seriously.

We built a fine-tuning method for the low-precision era. It is validated on A100 and H200, and its math points directly at Blackwell-class FP4 training. Standard LoRA leaves too much quality on the floor. One theorem. One method. A clean path to validate.A100 BF16 baseline: gap 0.5329 → 0.0160. H200 FP8: standard LoRA degrades 68.2%, KSS-LoRA degrades 5.2%. The Gamma constraint extends to FP4-class Blackwell targets. We want compute collaboration, replication, and a clean external validation path.

13×

FP8 stops collapsing.FP8 degradation: 68.2% → 5.2%.

The low-precision speed path only matters if model quality survives it. KSS-LoRA keeps the model usable where standard LoRA fails.H200 FP8 comparison: standard LoRA loss moves 1.9861 → 3.3412; KSS-LoRA moves 1.5051 → 1.5831.

33×

Overfitting almost disappears.A100 BF16 gap: 0.5329 → 0.0160.

The model learns the transferable pattern instead of repeating training examples. That is the difference customers actually feel.The default KSS-LoRA configuration gives 33.3× lower gap with 0.4% validation-loss cost across the measured A100 baseline.

−0.009

One run crossed the line.Measured negative overfitting gap.

Validation loss fell below training loss, meaning the model performed better on unseen data than examples it had already studied.κ = −0.009 in the sr=0.3, γ=0.1 configuration. Larger-scale κ validation remains the next proof target.

94.7%

Noise made it less brittle.50% noise: gap = 0.016.

Production data is messy. KSS-LoRA became less prone to memorisation when the data was deliberately corrupted.At 50% random noise injection, baseline gap remains 0.309 while KSS-LoRA gap falls to 0.016; best measured run 0.0087.

FP4

Blackwell is the next proof arena.γ_min(FP4) = 1.0.

The theorem already tells us what a stable FP4-class method must satisfy. The next step is external validation on the hardware built for that precision.The Koščák Gamma Theorem extends the stability constraint from FP8 to FP4-class targets. Public Blackwell results wait for validated logs.

88+

The result was not selected by hand.88+ logged runs.

The site shows the boring evidence too: successful runs, baselines, failure modes, and the separation between them.Logged A100 and H200 runs across Llama and Qwen benchmarks. Full reproduction notes are reserved for the preprint bundle.

Reach the team

Dr. Juraj Koščák

Co-Founder · Lead Scientist, PhD

✉ [email protected] LinkedIn ORCID

Filip Phauler

Co-Founder · Builder & Research Architect

✉ [email protected] 𝕏 @PhilPhauler 📸 Instagram GitHub

🎙 Press: [email protected]

Live research stats

33×

Overfitting reduction

5.2%

FP8 quality loss

−0.009

Koščák κ · negative gap

94.7%

Gap · 50% noise

88+

Validated runs

FP4

Next validationBlackwell target

The ask

Compute access. Research partnership. An introduction. The clean next step is to validate the Blackwell FP4 path with the people building the hardware.Requested collaboration: external replication, Blackwell-class FP4 validation, larger κ scaling runs, and co-authored methodology review before public production claims.

AI memorises.Ours generalises.Val loss below train loss.

The benchmark.Controlled comparison.

What we found.Observed mechanisms.

The model was memorising, not learningBaseline LoRA leaves a 0.5329 generalisation gap

New GPU precision exposes old training weaknessStandard LoRA is numerically unstable at FP8

Messy data stopped being the enemy50% noise collapses the KSS-LoRA gap by 94.7%

One lab. Three programs.The research surface.

Fine-tuning that stops memorisingStochastic sparse LoRA · κ-measured

The discovery engineBlind structure recovery from raw data

Applied AI securityOffensive research, defensive practice

Research.Run evidence.

KSS-LoRA keeps low-precision training from falling apartKSS-LoRA Solves FP8 Gradient Underflow: 5.2% vs 68.2% Quality Loss on H200

The Problem: Why FP8 Destroys Standard LoRA

The Koščák Gamma Theorem

The model stopped memorising the training set33× Overfitting Reduction in A100 BF16 Fine-Tuning

What the Numbers Mean

Why It Works

The result survived a faster GPUH200 SXM 141GB: 2.7× Faster Than A100, Same Gap Behavior

Why Hardware Validation Matters

Less memorisation also helped truthfulnessTruthfulQA Signal Improves on Llama-3.1-8B Under KSS-LoRA

Why Overfitting Causes Hallucination

There is a math reason FP8 failsThe Koščák Gamma Theorem Explains Standard FP8 Failure

The Theorem

Historical Foundation

One configuration did better on unseen data than training dataThe Koščák Coefficient: Measured Negative Overfitting Gap

What a Negative Gap Means

Phase 1 Results: H200 · Llama-3.1-8B · 5 Seeds

The Koščák Coefficient

Next Validation Target

Dirty data made the gap smaller50% Noise Injection Reduces the KSS-LoRA Gap by 94.7%

What is Antifragility in Training?

Why This Happens

Implications for Production Training

More noise.Less memorising.50% noise.94.7% gap reduction.

Built for NVIDIA hardware.Designed for reduced precision.

Full data.Benchmark tables.

All 88+ runs. Real data.Run distribution.

All 88+ Runs · Overfitting Gap

Loss Curves · Epoch by Epoch

Rank Sensitivity · 8 Values of r

vs Other Methods · Literature Comparison

FP8 Quality · KSS vs Standard

Cross-Model Consistency · Llama & Qwen

88+ runs. Zero cherry-picking.Run registry.

The mathematics.Formal basis.

The team.Authors and operators.

Jensen.Partner with us.NVIDIA.Validate this with us.

Questions.Reviewer questions.

Stay with the research.Follow validation updates.

Less memorising. Better low-precision training.33.3× lower gap. 5.2% FP8 degradation. κ < 0 measured.

References & HardwareSources, targets, and validation surfaces

AI memorises.
Ours generalises.Val loss below train loss.

More noise.
Less memorising.50% noise.
94.7% gap reduction.

Jensen.
Partner with us.NVIDIA.
Validate this with us.