After GDPO: A Map of the GRPO Family

Part 5 of 5: The Map

Updated 31 Oct 2026: Split “length and std normalization” into two axes, loss aggregation and advantage scaling: DAPO’s token-level loss is a loss-aggregation change, not a granularity one. Toned down two papers’ conclusions, which the old text overstated: ScaleRL finds that some knobs do move the ceiling, and the off-policy REINFORCE paper doesn’t conclude that the family converges on PPO. Added RLOO, GDPO’s position on the std axis, and a caveat on using Dr. GRPO with several rewards. Corrected the memory claim.

TL;DR: DAPO, Dr. GRPO, GSPO, GDPO, CISPO, VAPO, GiGPO, GMPO, REINFORCE++… it looks like GRPO fragmented into a zoo. It didn’t. Every post-GRPO method keeps the same skeleton — drop the critic, compute a group-relative advantage — and varies along a small set of design axes: token vs. sequence granularity, clip width, how per-token losses are aggregated, whether advantages are divided by the reward std, whether multiple rewards are pooled before normalizing, KL on or off, and how groups are sampled. Hold the axes and the zoo collapses into a handful of deliberate trade-offs. This post maps the four variants worth treating in depth, gives you a decision guide for when to reach for each, and equips you to reason about the next variant you meet — not just the ones listed here.

Reading time: ~18 minutes

Prerequisites: This is the capstone of the series. It assumes you already know GRPO. If not, start with Part 3: GRPO — and ideally Part 2: PPO and Part 4: GDPO — then come back. This post sits on top of those; it does not re-derive them. Symbols follow the series notation.

The Zoo Is Real

Open any RL-for-LLMs reading list from the last year and you will drown in acronyms. DAPO. Dr. GRPO. GSPO. GDPO. CISPO. GMPO. VAPO. GiGPO. REINFORCE++. There is even a curated index whose entire job is to keep up with the post-R1 GRPO family. The natural reaction is the one everyone has: the field fragmented, and I now have ten algorithms to learn instead of one.

That reaction is wrong, and it is wrong in a useful way. These are not ten algorithms. They are one algorithm — GRPO — being paid down along a handful of engineering debts that GRPO left on the table. Once you can name those debts, every “new” method announces itself as which debt it pays and how. You stop memorizing the zoo and start reading coordinates on a map.

This post is that map. It is deliberately a synthesis, not a listicle: the value is in the axes and the judgement, not in summaries you could get from each paper’s abstract.

The Shared Skeleton (What Nobody Changes)

Start with what every member of the family keeps. Strip away the branding and they all make GRPO’s two core moves:

1. Drop the critic. No learned value network $V_\psi$. PPO’s four-model architecture loses the most expensive kind of model: a trainable one. For a 7B model with Adam, that is about 112 of 252 GB of model state (Part 2), before activations, the KV cache and the generation engine, and before sharding changes the per-GPU picture.
2. Compute a group-relative advantage. For a prompt $q$, sample a group of $G$ completions $\{o_1, \ldots, o_G\}$ from the old policy, score them, and use the group statistics as the baseline:

That is the whole skeleton. At heart it is REINFORCE with a baseline — the mean reward of the group is the baseline — which is exactly the lineage Ahmadian et al. (2024) argued we should return to for RLHF. Everything is then plugged into a PPO-style clipped surrogate:

Every variant in this post inherits these two equations. What they argue about is the fine print — and the fine print turns out to be a short list.

The Axes of Variation

This is the intellectual core of the post. Spend time here; the variants below are just coordinates in the space these axes define.

Granularity. Is the importance ratio $\rho$ and its clipping computed per token or per sequence? GRPO is token-level. A single rare token can produce a wildly large or small ratio, and that noise compounds — badly in mixture-of-experts models, where the active experts can change between the sampling policy and the training policy. Sequence-level methods average the signal over the whole completion so the unit of optimization matches the unit of reward.

Clip width. PPO/GRPO clip symmetrically, $[1-\varepsilon, 1+\varepsilon]$. But the clip ceiling quietly caps upside: it limits how much probability mass a good-but-currently-unlikely token can gain in one step. Those low-probability tokens are often the interesting ones — the “wait,” “let me reconsider,” fork-and-reflect tokens that long chain-of-thought reasoning depends on. Widening or de-symmetrizing the clip (“clip-higher”) controls the explore/exploit edge and is the main lever against entropy collapse, where the policy sharpens into a narrow, repetitive mode and stops learning.

Loss aggregation. How the per-token losses become one number. GRPO averages within each response — dividing by its length $|o_i|$ — then across responses. That harmless-looking convenience biases the gradient: a token in a long response counts for less than one in a short response, so a long wrong answer spreads its negative advantage thinly and is penalised less per token. The optimizer drifts toward longer wrong answers, which is the main reason responses balloon during training without getting better. DAPO’s token-level loss averages over every token in the batch (token-mean); Dr. GRPO divides by a constant. Both remove the per-response length term.

Advantage scaling. Whether the advantage is divided by the group’s reward standard deviation — and, with several rewards, whose. Dividing by the std over-weights prompts that are too easy or too hard, where rewards barely vary (Part 3 works through the numbers). Dr. GRPO drops the division. GDPO keeps it, but applies it per reward.

KL term. Keep the KL penalty to the reference model, or drop it? KL anchors the policy and prevents drift, but it costs a reference model in memory and can hold the policy back when you want it to move far from the base model (as in R1-Zero-style training from a base checkpoint). Many recent recipes drop it entirely.

Reward pooling. With several rewards, are they summed and then normalized together, or normalized separately and then combined? Pooled, groups with different reward profiles can collapse to identical advantages, and a rarely-satisfied objective is drowned out by an easy one. This is the failure Part 4 is built around. (A related choice is whether the baseline and normalization are per-group or global across the batch.)

Sampling. Static groups versus dynamic sampling. A group where every completion is correct (or every one is wrong) has zero advantage variance and therefore contributes no gradient — it is wasted compute. Dynamically filtering or resampling degenerate groups keeps every batch informative.

Seven axes. That is the entire design space the family lives in. Here is the same idea as a picture — each axis, and the variants that move along it:

Two patterns jump out of this picture. First, most variants move along one or two axes — they are surgical. Second, DAPO moves along several at once; it is less a single idea than a curated bundle. Keep both observations in mind as we walk the variants.

What do these axes look like in a training run? The two biases that motivate Dr. GRPO and DAPO show up as characteristic curve shapes:

The Variants: Problem → Change → When

With the axes in hand, here are the four worth knowing in depth, plus the frontier cluster. Read each as a triple: the problem it noticed, the change it made, and the situation that should make you reach for it.

Dr. GRPO — debias the gradient

Understanding R1-Zero-Like Training: A Critical Perspective (Liu et al., 2025) noticed that GRPO’s length and standard-deviation normalization are not neutral conveniences — they are biases. The length term systematically rewards longer wrong answers, which is exactly the runaway-verbosity pathology practitioners kept seeing. The fix is almost embarrassingly small: delete both normalizers. No division by response length, no division by group std. The gradient becomes unbiased, responses stop inflating, and the model is markedly more token-efficient. Built on the lightweight Oat framework, with a clean reference implementation. Reach for it the moment you see length climbing without accuracy following.

DAPO — the production bundle

DAPO: An Open-Source LLM Reinforcement Learning System at Scale (Yu et al., 2025) is the one that is really four things at once — Decoupled Clip and Dynamic sAmpling Policy Optimization. It ships clip-higher (de-symmetrize the clip ceiling to keep low-probability exploratory tokens alive and fight entropy collapse), dynamic sampling (drop degenerate all-correct/all-wrong groups so every batch carries gradient), token-level policy-gradient loss (token-mean loss aggregation, so long completions are not under-weighted in long-CoT training — the same lever as Dr. GRPO’s length fix), and overlong reward shaping (soften the penalty for responses that run past the length budget, reducing reward noise). Together they reach 50 points on AIME 2024 with a Qwen2.5-32B base, with code and data fully open on top of verl (repo). DAPO is what you reach for when you are scaling up real long-CoT training and instability — diverging loss, collapsing entropy — is the thing standing between you and a good model.

GSPO — match the unit of optimization to the unit of reward

Group Sequence Policy Optimization (Zheng et al., 2025, the Qwen team) made a single sharp observation: the reward is assigned to a sequence, so why is the importance ratio computed per token? Token-level ratios are noisy, and that noise is catastrophic for mixture-of-experts models, where the set of active experts can differ between the sampling and training passes. GSPO defines the importance ratio on the sequence likelihood and clips at the sequence level. The result stabilizes MoE RL training — enough that it underpins the latest Qwen3 models. Reach for it when you are training MoE models or chasing large-scale stability, and when token-level ratios are giving you grief. (Implementations live in verl and TRL; the Qwen write-up is the readable intro.)

GDPO — keep multiple rewards from colliding

This is Part 4, so I will be brief. GDPO: Group reward-Decoupled Normalization Policy Optimization (Liu et al., 2026, NVIDIA) attacks the multi-reward case. Sum several rewards and normalize the total, and groups with different reward profiles can map to the same advantages; a rarely-satisfied objective gets drowned out. GDPO normalizes each reward independently before combining. In effect it is GRPO on a reweighted sum, each reward divided by its own group std (Part 4’s reweighting view) — so it moves on the advantage-scaling axis too, per reward. Reach for it whenever you are optimizing more than one reward at once — tool calling with format-plus-correctness, math with correctness-plus-length, coding with pass-rate-plus-length-plus-bugs. (If you suspect this is already happening to you, the advantage-collapse checklist is a quick diagnostic.)

The frontier cluster

Beyond the four, a frontier worth knowing as coordinates, not as separate disciplines:

• CISPO, from the MiniMax-M1 technical report (2025), clips the importance-sampling weights rather than the token updates. The motivation is precisely the clip-width axis above: standard clipping silently discards the low-probability fork-and-reflect tokens, so CISPO keeps every token contributing and reports a 2× speedup over DAPO in their setting. Reach for it when you suspect your clip is eating your model’s “aha” moments.
• SPO (Single-stream Policy Optimization, Xu & Ding, 2025) goes group-free: it replaces the on-the-fly group baseline with a persistent, KL-adaptive value tracker and normalizes advantages globally across the batch. It is a glimpse of where the family heads once you decide the group itself is the bottleneck.

You can now place the rest of the zoo — VAPO, GMPO, GiGPO, REINFORCE++ — without my help: each one moves along one or two of the seven axes. That is the entire point of the map.

A Decision Guide

The variants reduce to a flowchart. Start from the symptom, not the acronym.

One caveat on the first row. Dropping std normalisation with several rewards failed in the GDPO paper: GRPO without std reached 0% correct format on BFCL-v3 (Part 4). With more than one reward, keep per-reward std (GDPO) and take only Dr. GRPO’s loss-aggregation fix.

The honest version of this flowchart has a second caveat: the moves compose. You can run Dr. GRPO’s debiasing and clip-higher and drop KL at the same time; that is roughly what the minimal from-scratch repos do by default. The flowchart tells you which lever a symptom points to, not that you may pull only one.

”Is This Just PPO With Extra Steps?”

A fair skeptic will push back: if everything is REINFORCE-with-a-baseline wearing a PPO surrogate, did we really invent anything?

Two recent papers give the skeptic real ammunition — though neither says quite as much as the slogan.

First, Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm (Yao, Chen et al., UCLA and Alibaba; ICLR 2026) derives group-relative REINFORCE from first principles and shows it admits an off-policy reading: the within-group mean baseline doesn’t need the strict on-policy assumption usually attached to it. Its practical takeaways are that adapting REINFORCE to off-policy data comes down to two moves — regularising the policy update, and shaping the training data — and it reinterprets Kimi’s OPMD and Meta’s AsymRE as regularised REINFORCE. My own synthesis, not the paper’s conclusion: read next to the map above, much of the family looks like REINFORCE with a group baseline plus different regularisers, rather than a set of rival algorithms.

Second, The Art of Scaling RL Compute for LLMs (Khatri, Madaan et al., 2025) — a 400,000+ GPU-hour study — fits compute-versus-performance curves to many recipes. Its first finding is that recipes do not all reach the same asymptote. Many of the knobs on this map — loss aggregation, normalization, curriculum, the off-policy algorithm — mainly change compute efficiency: how fast you approach the ceiling. But some move the ceiling itself. The ones it reports include the loss type, FP32 precision at the LM head, model size and generation length. Its own recipe, ScaleRL, uses a CISPO-style loss — the CISPO coordinate on this map.

That is the judgement this map is really teaching. Most knobs on this map change how fast you reach the ceiling, not where it is. A few — the loss type, numerical precision, model size — do move it. Know which kind of knob you are holding before you expect it to out-level your data.

The Production Lens

Which points straight at the thing that actually decides outcomes: every variant on this map optimizes against a reward, and every reward is a proxy.

In math and code you get lucky — a verifier exists. The answer is right or it is not; the unit tests pass or they do not. Reinforcement learning with verifiable rewards (RLVR) works there precisely because the proxy is nearly the real thing. Step outside that world — helpfulness, faithfulness, tone, “did the agent actually accomplish the task” — and there is no clean verifier. You build one (a reward model, an LLM judge, a heuristic), and the optimizer’s job becomes finding its seams. Whatever gap exists between your proxy and what you actually want, a good optimizer will widen it. That is reward hacking, and a better optimizer hacks a flawed reward faster.

So the practical ranking is the opposite of the one the zoo implies. Outside math and code, the bottleneck is almost never the optimizer; it is the verifier. Picking GSPO over DAPO is a second-order decision. Building a reward your optimizer cannot game is the first-order one — and it is the genuinely hard, under-glamorous engineering that decides whether any of these algorithms help you. Hold this as a principle: spend your scrutiny on the reward, then let the axes tune the run.

Feeling the Differences on One GPU

You do not need a cluster to build intuition for the axes — you need a small model and an afternoon. The deltas between these methods are legible at tiny scale, and a handful of repos are built to make them legible:

• GRPO-Zero implements GRPO from scratch with DAPO’s improvements (token-level loss, no KL, overlong filtering) on the Countdown task. It is the single best repo to dissect if you want to see the axis-moves in code rather than read about them.
• simple_GRPO is ~200 lines across two files and trains in under an hour on a single A800 — teaching-oriented, nothing hidden.
• TinyZero reproduces R1-Zero-style training on verl for under $30, a clean baseline to fork.
• Unsloth’s GRPO notebooks get the R1-Zero “aha” on 5–7 GB of VRAM via QLoRA — about as close to a laptop as this gets.

For the research framing of why small-model RL is even worth your time, Reinforcement Learning for Reasoning in Small LLMs (Dang & Ngo, 2025) trained a 1.5B model to 46.7% on AIME 2024 for roughly $42 on 4×A40 in 24 hours — and was candid about what broke. If you would rather run the full family under one roof, verl implements almost all of it (PPO, GRPO, GSPO, DAPO, Dr. GRPO, RLOO, REINFORCE++, PRIME) with verifiable rewards; TRL is the gentlest on-ramp via its GRPOTrainer. This is exactly the territory my own rlvr-from-scratch project lives in.

Where the Family Is Heading

The axes will keep being the right lens, even as the methods drift past the ones named here. Three directions are already visible. Group-free methods like SPO ask whether the group baseline — the move that defined GRPO — is itself the next bottleneck, replacing it with a persistent value tracker. Agentic and hierarchical advantages push group-relative credit assignment down to steps and sub-trajectories for tool-using, multi-turn agents. And verifier-free RL tries to escape the proxy problem from the other side — learning a reward signal without a hand-built verifier at all, which, if it works, attacks the actual ceiling rather than the path to it.

None of these will look like a clean break. Each will arrive as another coordinate: a different granularity, a different clip, a different normalization, a different way to sample. GRPO never fragmented into a zoo. It exposed a design space — and once you can read the axes, you are not learning the next algorithm so much as recognizing where on the map it sits.

That is the whole series in one sentence: from PPO’s critic, to GRPO’s group baseline, to GDPO’s decoupled rewards, the field has been walking a small set of axes the entire time. You can see the full arc on the series page.

References

Foundations (linked, not re-derived here):

• Schulman et al. (2017), Proximal Policy Optimization Algorithms — arXiv:1707.06347
• Ahmadian et al. (2024), Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs — arXiv:2402.14740
• Shao et al. (2024), DeepSeekMath (GRPO origin) — arXiv:2402.03300
• DeepSeek-AI / Guo et al. (2025), DeepSeek-R1 — arXiv:2501.12948

The four core variants:

• Liu et al. (2025), Understanding R1-Zero-Like Training: A Critical Perspective (Dr. GRPO) — arXiv:2503.20783
• Yu et al. (2025), DAPO: An Open-Source LLM Reinforcement Learning System at Scale — arXiv:2503.14476
• Zheng et al. (2025), Group Sequence Policy Optimization (GSPO) — arXiv:2507.18071
• Liu et al. (2026), GDPO: Group reward-Decoupled Normalization Policy Optimization — arXiv:2601.05242

Frontier and critical perspective:

• MiniMax (2025), MiniMax-M1 (introduces CISPO) — arXiv:2506.13585
• Yao, Chen et al. (2025; ICLR 2026), Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm — arXiv:2509.24203
• Khatri, Madaan et al. (2025), The Art of Scaling RL Compute for LLMs — arXiv:2510.13786
• Xu & Ding (2025), Single-stream Policy Optimization (SPO) — arXiv:2509.13232

Small-model / single-GPU:

• Dang & Ngo (2025), Reinforcement Learning for Reasoning in Small LLMs (Open-RS) — arXiv:2503.16219
• Frameworks and minimal repos: verl, TRL, Oat, GRPO-Zero, simple_GRPO, TinyZero, Unsloth, and the curated rl-llm-nlp index.

@misc{sousa2026grpofamilymap,
  title={{After GDPO: A Map of the GRPO Family}},
  author={Sousa, Vitor},
  year={2026},
  howpublished={\url{https://www.vitorsousa.com/blog/grpo-family-map/}},
  url={https://www.vitorsousa.com/blog/grpo-family-map/}
}