Benchmark Your Benchmark

rates = [time_once(model, prompt, n_new, use_cache)]                          # bug
rates = [time_once(model, prompt, n_new, use_cache) for _ in range(N_RUNS)]   # fix

The model had 112 tests; the benchmark had none — and that’s exactly where the bug shipped. N_RUNS was defined, never used, and every “median ± spread” was one measurement wearing statistics as a costume.

The fix came with a rule attached: prove both configs compute the same thing before timing anything.

assert torch.equal(ids_nc, ids_c), "cache != no-cache in greedy — bug, not benchmark"

Benchmarks are code. Review them like code, and never report a number without its spread. Full story in the Phase 1 close-out.