Learning with Verifiable Rewards
- Model Completed
- Training Completed
- SFT baseline In progress
- GRPO Planned
- Ablations & analysis Planned
A research-engineering study of whether GRPO earns its cost over a properly built SFT baseline on GSM8K, and where it fails. Model and training foundations are built and tested; the SFT baseline is in progress; GRPO, ablations, and findings are not yet run.
The question
Does GRPO materially improve held-out GSM8K accuracy over a properly built SFT baseline at small scale, and where does it fail?
This is a study, not a tutorial and not an RL library. It implements one algorithm, varies it along a small number of axes, and is finite by design. Everything in the repository exists to answer that question honestly: a protocol frozen before any result is seen, seed variability reported next to point estimates, and a failure taxonomy built from inspected examples rather than assumed.
Status at a glance
Only the first two rows are finished work. Everything below them is either in progress or planned, and this page makes no claim about GRPO until the runs exist.
| Component | Status | Evidence |
|---|---|---|
| Model foundations: decoder-only transformer | Completed | Code, 112 model-layer tests in CI, verification write-up |
| Training foundation: tokenizer, pinned data, loop, reference run | Completed | One-command CPU reference run, reproduced on two machines (below) |
| GSM8K data, answer verifier, evaluation harness | Built and tested | Content-addressed train/dev cut, verifier tests, frozen eval config. No study result yet |
| SFT baseline | In progress | Config frozen before any GRPO number exists. Not yet trained |
| GRPO | Planned | Base hyperparameters frozen. Rollout and group-advantage code not yet written |
| Controlled ablations | Planned | Axes chosen in advance (below). Not run |
| Findings | None yet | This page will publish only measured conclusions |
What is finished
Model foundations
A decoder-only transformer built from raw PyTorch tensors: scaled dot-product and multi-head attention, RoPE / sinusoidal / learned / ALiBi positional encodings, RMSNorm and LayerNorm, SwiGLU and GELU feed-forward layers, pre- and post-norm blocks, an incremental KV-cache, and greedy / temperature / top-k / top-p sampling.
It is pinned by 112 tests covering shapes, causality, centred finite-difference gradient checks, and the property that is easiest to get silently wrong: incremental decoding with the KV-cache matches the full forward pass at atol=1e-5. The cache also has a measured benefit. With greedy decoding on CPU, it raises median throughput from 317.9 to 814.3 tok/s at 128 new tokens, and from 185.2 to 699.5 tok/s at 256 (median of 5 runs, d_model=256, 4 layers).
The close-out post, A Transformer from Raw Tensors: What 112 Tests Taught Me, covers what broke and how each failure was pinned. The theory is in Part 1, Part 2, and Part 3 of the Transformer Internals series.
Training foundation
A character-level tokenizer, corpus fetching against a pinned sha256, batching, token-level cross-entropy, a frozen run config, and the training loop (warmup plus cosine schedule, decoupled weight decay, gradient clipping, deterministic evaluation). One command, uv run rlvr-train --config configs/tiny.yaml, reproduces the reference run from a clean clone.
The reproducibility claim is stated so it can be checked. The same seed should reproduce the final validation loss within ±0.05 nats. Measured on two machines:
| Machine | torch | Wall clock | Final val loss |
|---|---|---|---|
| Apple Silicon, macOS, CPU | 2.11.0 | 37 s | 1.7677 nats |
| x86-64, Linux, CPU | 2.13.0 | 80 s | 1.7677 nats |
This is an engineering check on the training path, not a research result.
Experimental protocol
These decisions were frozen before any SFT or GRPO number existed. Changing any of them after a result is seen counts as a deviation and gets recorded with a date and a reason.
| Base model | Qwen 2.5-0.5B, pinned to a model revision |
| Task | GSM8K. The train/dev cut is content-addressed and re-derivable from its seed. Held-out is upstream’s own test split, untouched until the study closes |
| Baseline | SFT on answer tokens only, 2 epochs, evaluated zero-shot. The base model’s few-shot score is a harness sanity check, not the baseline |
| GRPO base arm | Group size 8, 8 questions per step, 60 steps, lr 1e-6, KL coefficient 0.04, clip ε 0.2. Rollouts are sampled, evaluation is greedy |
| Seeds | RQ1 (SFT vs GRPO) runs three seeds for both arms. Each ablation runs one seed, and the RQ1 seed spread sets the scale for judging it |
| Comparison | Paired per question on a frozen 200-question dev prefix. At n=200 a confidence interval is roughly ±7 points, so two overlapping intervals are not treated as a conclusion |
| Hardware and budget | Apple Silicon (MPS), float32. Measured cost is 3.3 s per completion, so one GRPO run takes about 3.5 h and the study can afford 6–8 runs |
Research questions
| RQ1 | Does GRPO materially improve held-out GSM8K accuracy over the SFT baseline? |
| RQ2 | How sensitive is the result to group size? |
| RQ3 | What effect does KL regularization have? |
| RQ5 | Where does reward improve without a corresponding improvement in held-out accuracy? |
RQ4, the effect of a format reward, was dropped before any training. The base model already scored 200/200 on format with zero missing answers in the harness check, which leaves no headroom to measure. The third ablation slot goes to either learning rate or rollouts per question instead.
What this page will contain when the study is done
The case study is not finished until each of these exists. Items marked pending have no results behind them yet.
- Research question and hypothesis. Stated above. The hypothesis is written down before the first GRPO run.
- Baselines and protocol. Frozen above.
- Dataset, model, seeds, and hardware. Frozen above, plus the exact model revision each run used.
- Main results with uncertainty. Pending. Per-seed accuracy for both arms, the seed spread, and paired per-question deltas.
- Controlled ablations. Pending. One variable at a time against the base GRPO arm. A delta smaller than the seed spread is reported as below the noise, and that counts as a result.
- Failure taxonomy and negative results. Pending. Categories derived from inspected completions, including any case where reward rose but held-out accuracy did not.
- Reproduction instructions and exact code version. The commit and config behind every reported number.
Out of scope
To keep the study finite and answerable:
- DAPO, GDPO, or any other algorithm added for breadth
- A general PPO/RL framework or RLHF library
- Large-scale pretraining or distributed training
The algorithms are covered as explainers in the Reinforcement Learning for LLMs series. This project is where one of them gets measured.
Tech stack
Python 3.13 · PyTorch · uv · ruff · ty · pytest · GSM8K · Qwen 2.5-0.5B