A 125M-parameter language model, trained from scratch and fine-tuned to answer legal/financial questions — grounded in a passage you supply, or from general knowledge.
| parameter | value | what it means |
|---|---|---|
| total parameters | 125.8M | size of the model — small enough to run inference on CPU with no GPU |
| vocabulary size | 16,384 | number of unique tokens the custom byte-level BPE tokenizer can produce |
| hidden size | 768 | width of each token's internal representation as it flows through the network |
| layers | 12 | number of stacked transformer blocks the input passes through |
| attention heads | 12 | parallel attention patterns computed per layer (head dim 64, standard multi-head attention) |
| feed-forward size | 3,072 | width of the expansion layer inside each block (SwiGLU activation) |
| context length | 1,024 | maximum number of tokens the model can attend over at once |
| tied embeddings | yes | input and output embedding matrices share weights, reducing parameter count |
| tokens per parameter | ~16.0 | 2.02B training tokens ÷ 125.8M params — a Chinchilla-style ratio; ~20 is considered compute-optimal |
Next-token prediction only, no instruction data. Teaches the model to speak the legal/financial register fluently.
| learning rate | 6e-4 → 6e-5 | step size for weight updates; cosine decay from a high starting rate |
| warmup | 200M tokens | gradually ramps up the learning rate at the start to avoid early instability |
| global batch size | 524,288 tok | total tokens averaged per optimizer step, via gradient accumulation |
| weight decay | 0.1 | regularization that discourages overly large weights |
| gradient clipping | 1.0 | caps the gradient norm to prevent destabilizing update spikes |
Continues training from the pretrained checkpoint on Q&A pairs generated by Gemini, at a much lower learning rate.
The open-domain slice was added after the first fine-tune reflexively refused every no-passage question — see "What this is" below. Grounding accuracy is a separate quality check: a second LLM pass verified whether each generated answer was actually supported by its source passage, on a partial judged sample taken early in generation (full-dataset judging was skipped afterward since accuracy was already high, deferred to a later spot-check).
| epochs | 3 | full passes over the fine-tuning dataset |
| learning rate | 2e-5 → 2e-6 | much lower than pretraining, since we're adapting an already-trained model |
| warmup | 30 steps | short ramp-up — fine-tuning starts from an already-stable checkpoint |
| weight decay | 0.01 | lighter regularization than pretraining |
Both models generated answers to the identical 200 held-out questions (never trained on by either model — same shuffle seed used for both). A blinded LLM judge graded each answer independently against the reference answer, never told which model produced it.
| metric | n | SLM-125M | Gemma-2-2B | Gemma-RQ (compressed) |
|---|---|---|---|---|
| grounded accuracy | 155 | 51.6% | 92.9% | 92.9% |
| open-domain accuracy | 45 | 15.6% | 80.0% | 86.7% |
| grounding / faithfulness rate | 155 | 57.4% | 93.5% | 93.5% |
Gemma wins clearly on every axis — expected given the ~21x parameter gap (125M vs. 2.64B) and Google's much larger pretraining. This isn't a fair fight on raw quality: 125M-from-scratch-and-fully-fine-tuned vs. 2.64B-pretrained-and-QLoRA-adapted differ in more than one variable at once (different tokenizers — 16K custom BPE vs. 256K SentencePiece, so the val-loss numbers above aren't comparable across the two models — full fine-tune vs. adapter, and pretraining scale). Try both side by side on the Gemma demo, or see Gemma's base and fine-tuned weights on Hugging Face. The Gemma-RQ column is the same fine-tuned Gemma with its 590M-parameter embedding compressed to 3.59 bits/param via a unified residual codebook — quality essentially unchanged; see the compression demo.
80% of random pulls come from the 45 held-out questions never used in training (a true generalization test); 20% come from the ~1,964 open-domain training questions (more variety, but the model may recall a memorized answer). Each pull is labeled below.
This model was fine-tuned on two distinct skills. With a passage, it's a grounded reader — it reads whatever text is in the passage box at that moment and answers strictly from it, including saying "Not stated in the passage." when the answer genuinely isn't there. It isn't recalling memorized facts about that passage; it's doing live in-context extraction, the same skill it would apply to any new text, including the random excerpts pulled above.
With no passage, it answers from general legal/financial knowledge learned during pretraining and reinforced by ~2,000 open-domain Q&A examples — added after the model's first version reflexively refused every no-passage question, since none of its original fine-tuning examples omitted a passage.
At 125M parameters it can miss specific numbers, repeat itself, or give an imperfect answer in either mode — treat answers as a starting point, not a citation.