Honey I judged the rollouts: How your RL Judge affects your models reasoning

# Honey I judged the rollouts: How your RL Judge affects your models reasoning

## Introduction 
Credit assignment, especially with agentic judges, has been a hot topic for a while now, and deservedly so: if you can get more learning out of the same rollouts you're already paying for, why wouldn't you? I'd had a few hypotheses about this rattling around for some time, and with my compute grant about to expire, I decided to sprint it out and actually test them.

The question that this blog tries to answer is that with an agentic judge, can we expect the RL to teach good and efficient reasoning and not just the correct outcome? 


## Hypotheses and background 

LLM RL post-training has evolved over the years from value-function and critic based methods to largely critic free methods such as [GRPO](https://arxiv.org/abs/2402.03300), that enabled RL scaling at previously unseen heights, with huge success. But as RL runs get more expensive and good training data gets scarcer, a question comes up: can we get more out of the rollouts we already have?

For those who don't know, GRPO is a critic-free way of doing RL on LLMs. The working principle is that the model is already stochastic, so you can just sample a group of rollouts for the same problem, and some will be better than others. GRPO scores each rollout, compares it to the group average, and nudges the model toward the above-average ones and away from the rest. In effect, it sharpens a distribution the model already has. To learn more about GRPO and how it works, check out my blog [here](https://chinmaykarkar.com/blog/reward_structure_blog/)

That simplicity comes at a cost: GRPO gives every token in a rollout the same credit. Your model could rickroll you in the middle of a math problem, and as long as the final answer is correct, GRPO would upweight every token in that rollout, relevant or not.
<div class="fig-mount" data-src="/llm_credit_assignment_figures.html" data-fig="fig-comic"></div>
As you can see, this is quite bad for learning things, as you are not inducing the right priors/lessons via outcome RL, and thus credit assignment is an ongoing topic of research in the field of RL. 

The question I wanted to test was quite simple: can an RL judge give a better reward, one that makes the model's chain of thought better and more efficient, rather than just optimising for the correct answer? 

## Methodology and Setup

I started with the data. I pulled 3k problems each from [DeepScaleR](https://huggingface.co/datasets/agentica-org/DeepScaleR-Preview-Dataset) and [DAPO-Math-17k](https://huggingface.co/datasets/BytedTsinghua-SIA/DAPO-Math-17k), deduplicated them (the two overlap heavily), and ran a pass@4 screen with a 16k-token cap on two candidate models: [Qwen3-4B-Thinking-2507](https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507) and [Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B) in thinking mode. The point of the screen is to find prompts with learning signal: GRPO only learns from a prompt when the model sometimes gets it right and sometimes doesn't, so all-pass and all-fail prompts get filtered out.

| Model × dataset | pass@1 | pass@4 | Truncated at 16k |
|---|---:|---:|---:|
| Qwen3-4B-Thinking-2507 × DeepScaleR | 69.1% | 75.5% | 19.5% |
| Qwen3.5-4B × DeepScaleR | 42.1% | 58.2% | 51.4% |
| Qwen3-4B-Thinking-2507 × DAPO | 59.6% | 69.0% | 38.3% |
| Qwen3.5-4B × DAPO | 31.5% | 45.1% | 68.0% |

Qwen3-4B scores higher here, and Qwen3.5 truncates a lot at 16k, mostly because it thinks for much longer. But Qwen3.5 gave far more prompts with learning signal (523 vs 196 on the first 1.5k DeepScaleR problems), and in my earlier work Qwen3-4B models looped badly. So I went with Qwen3.5-4B, re-screened it with a 32k cap, and ended up with 975 usable prompts (875 for training, 100 held out).

I compared three runs:

- **Outcome-only GRPO (the baseline):** [Dr. GRPO](https://arxiv.org/abs/2503.20783)-style advantages (reward minus the group mean, no std normalisation) and [prime-rl](https://github.com/PrimeIntellect-ai/prime-rl)'s IPO loss, which drops any token whose probability moved by more than 0.3 and caps the importance ratio.
- **Judge credit, λ = 0.25:** the same, plus judge credit on specific spans, capped so the judge's correction averages at most 0.25 per token. It nudges the outcome reward rather than overriding it.
- **Judge credit, λ = 1.0:** the judge turned up, taking its share of the total advantage from about 6% to about 20%.

Everything else was constant: 40 steps of 8 prompts × 8 rollouts, a 32k response cap, rank-16 [LoRA](https://arxiv.org/abs/2106.09685) (α = 32), learning rate 1e-5, temperature 1.0, and gpt-6-luna as the judge.

### How the judge hands out credit

For every rollout, the judge gets the problem, the model's full reasoning trace, its final answer, the verifier verdict, and the reference answer plus a worked solution when one exists.

The judge chooses the spans itself. Before it sees the trace, I insert markers like `⟦1234⟧` at the start of every line (and every ~64 tokens inside long lines), pointing into the model's *original* generated token ids. The judge then picks the stretches that matter, i.e. the original errors, claims that depend on them, genuine corrections, key progress and wasteful loops, and returns each one as an exact token range with a label, a confidence and a quote from inside the span as evidence. It doesn't have to cover the whole trace: every token it leaves out gets zero credit. It also has tools: it can view exact token ranges, search the trace, and run sandboxed Python or a symbolic-equivalence check to verify arithmetic before it calls something right or wrong. 

Each label maps to a score:

| Label | What it means | Score |
|---|---|---:|
| Verified correction | explicitly repairs an earlier mistake, and the repair is verifiably right and actually used | +1.00 |
| Valid progress | sound reasoning that moves the solution forward | +0.50 |
| Useful exploration | a reasonable investigation, even if abandoned | +0.25 |
| Neutral / abstain / low confidence | neither progress nor error, or the judge isn't sure | 0 |
| Unproductive repetition | re-doing or re-checking with no new progress | −0.25 |
| Unsupported | locally fine, but built on an earlier error | −0.50 |
| Invalid | the original material error | −1.00 |

That score becomes per-token credit $c_{i,t}$ for every token in the span, and gets added on top of the usual outcome advantage:

$$
\widetilde A_{i,t} = A_i + \lambda\, c_{i,t},
\qquad
\frac{1}{T_i}\sum_{t=1}^{T_i}|\lambda\, c_{i,t}| \leq \kappa = 0.25
$$

where $A_i$ is the outcome advantage shared by every token in rollout $i$, $\lambda$ is the judge's strength, and $T_i$ is the rollout length. If the judge's correction averages more than $\kappa$ per token, it's scaled down to fit. So the answer checker still sets the overall direction for each rollout, and the judge reshapes credit *within* it, pushing good spans up and bad spans down. The good part of this is that previously unusable groups which had either all pass or all fail, could still produce some learning signal. 

<div class="fig-mount" data-src="/llm_credit_assignment_figures.html" data-fig="fig-spans"></div>



## The RL runs and results

<div class="fig-mount" data-src="/llm_credit_assignment_figures.html" data-fig="fig-traincurves"></div>

<div class="fig-mount" data-src="/llm_credit_assignment_figures.html" data-fig="fig-scores"></div>

If you look at the numbers, the difference in the benchmark scores is just barely above noise, and even considering the small number of steps, how can we say from the scores alone if the extra credit was even doing anything? So I put my trace police hat on, and went into the data mines to look for answers. 

### How to know the judge helped? 

The clearest sign would have been materially better reasoning traces, so I started by reading them, and the result was a bit disappointing. The judge-credited model wasn't doing dramatically cleverer maths. What it mostly did was *commit* to answers faster.

<div class="fig-mount" data-src="/llm_credit_assignment_figures.html" data-fig="fig-loop"></div>


So maybe the judge just wasn't giving enough credit? At λ = 0.25 it only contributed about 6% of the total advantage signal, so I ran λ = 1.0, which pushed that to about 20%. To my disappointment, the results were basically identical, with similar reasoning patterns. 

That left me with an uncomfortable question. Is the judge's credit actually pointing in the right direction, or is it just extra noise that happens not to hurt? Benchmarks and trace mining couldn't answer that, so I needed a method to measure, independently of the judge, whether the spans it rewarded actually made progress toward the answer.

To measure this, I borrowed a trick from an unlikely place: story generation, where there isn't even a right answer to check against. The [VR-CLI](https://arxiv.org/abs/2503.22828) paper needed to reward a model's plan for the next chapter without any verifier. Their trick: take a frozen LM and the real next chapter, and see how much more probable that chapter becomes once the plan is pre-filled. If the plan was useful, the real chapter becomes much more likely. 

Since math is verifiable, we replace the real next chapter with a reference answer. Take a reasoning trace, cut it off at some point, and force the model to commit right there by appending `</think> The final answer is $\boxed{`. Then ask the base model how likely the correct answer is as the next tokens (the mean log-probability of the answer tokens). If the model is already confident in the right answer at that point, the reasoning so far has basically solved the problem; if it's lost, it hasn't.

Doing this at the start and at the end of every span the judge labelled gets each span its own progress score: how much more confident the model is in the right answer *after* the span than *before* it. Now there's a measurement of each span's progress that never saw the judge's labels, and the question becomes testable: **do the spans the judge rewarded actually move the model toward the right answer?**

<div class="fig-mount" data-src="/llm_credit_assignment_figures.html" data-fig="fig-probe"></div>

### Does the judge reward real progress?

With a way to measure progress in hand, the test is simple: if the judge's credit points the right way, spans it scored higher should move the model further toward the right answer. I ran the probe on all 3,373 judged spans from 400 rollouts of the λ = 0.25 run. A naive comparison could fool the observer in a few ways, so before looking at a single score, I wrote down a control for each:

- **Position:** later spans naturally look more "answer-ish", since the model has done more work by then. 
- **Length:** a long span can move the needle more just by being long.
- **"Answer-likeness":** a span could make any answer more likely just by sounding conclusive. So on a subset of 100 traces I also scored a **wrong answer** at the same points (the trace's own wrong answer when it had one), and measured progress toward the right answer relative to the wrong one.
- **Wording:** maybe it's an artifact of the exact "The final answer is" phrase. So I repeated it with a different phrasing. 

The judge passes every one of them:

| Test | Progress per point of judge score | 95% CI |
|---|---:|---:|
| Main test (same trace, position and length controlled) | **0.105** | [0.078, 0.134] |
| Relative to a wrong answer | **0.098** | [0.041, 0.161] |
| Different answer wording | **0.106** | [0.058, 0.162] |

Shuffling the judge's labels within each trace never once produced an effect that large in 10,000 tries (p = 1e-4). The effect holds in traces that ended up correct and in traces that ended up wrong, so the phenomenon is quite general. It also holds in the groups where all 8 rollouts got the same reward, exactly where outcome-only GRPO has nothing to reward (0.108 per judge point, [0.064, 0.155]), so the extra signal the judge adds there is pointing the right way too. Inside a single trace, the spans the judge called good gained **0.29** more answer-confidence than the spans it called bad.

Broken down by label, the ordering is what you'd hope for:

<div class="fig-mount" data-src="/llm_credit_assignment_figures.html" data-fig="fig-progress"></div>

"Valid progress" is where the model actually moves toward the answer (+0.27); "unsupported" steps, reasoning built on an earlier mistake, actively move it away (−0.16). Repetition sits at roughly zero, which is exactly what repetition *is*: it doesn't change what the model believes, it just burns tokens. 

(The probe itself broke in a funny way the first time I ran it, by tripping over a tokenizer seam. The full story is in the [appendix](#appendix).)

### Does it make reasoning more efficient?

The probe tells us the judge points at the right steps. But did training on it actually change how the model reasons? When I read the traces earlier, the judge-credited model seemed to commit to its answer instead of looping, and the same probe can check that properly, across every held-out trace rather than a few hand-picked ones. 

For 1,600 held-out traces (100 problems × 4 samples, for the base model and each of the three trained policies), I paused the model every ~4k tokens and asked: *if you had to answer right now, how sure are you of the right answer?* Early in a trace that might be 20%; once the model has worked the problem out, it climbs to 90% and beyond. I call a trace **locked in** at the first pause where the model is at least 90% sure of the right answer and stays that sure at every pause after it. Everything the model writes after that point is spent on an answer it already has.

<div class="fig-mount" data-src="/llm_credit_assignment_figures.html" data-fig="fig-lockin"></div>

Every RL-trained policy learned to stop sooner once it had the answer, and the judge-credited policy stopped the fastest: about **1,900 fewer tokens** compared to the base model, against about 1,400 fewer for the outcome-only run. 


## Thoughts on credit assignment

Stepping back from this one experiment, as compute scales and models get smarter, we're starting to realise that GRPO-like methods are inefficient. They burn a lot of compute and leave a lot of learnable signal in the rollouts on the table, which is exactly the problem that I wanted to see if a well tuned judge could fix.

In recent times, methods have shifted to more PPO-style methods ([SAO](https://arxiv.org/abs/2607.07508)) where we want to bring the critic back, because even though GRPO is cheap, the critic gave fine grained credit, which is really important these days to get maximum learning out of your rollouts. 

Agentic judges help solve this, and I think it is where most of the labs are heading/standardising. Xiaomi's [MiMo-V2.6](https://arxiv.org/abs/2610.11959) report describes "groupwise agentic grading": "instead of assigning the same reward to all solutions that pass the test cases, we compare solutions within each group", which "steers the model towards shorter, more token-efficient solutions". That's the same kind of efficiency effect I saw here, at a vastly bigger scale. Thinking Machines trained [Inkling](https://thinkingmachines.ai/news/introducing-inkling/) with a rubric grader plus a claims grader that "performs agentic web search for claim verification", and openly notes that rubric graders "emphasize recall and can be hacked by models spraying plausibly relevant facts". For coding in [Inkling 2 Small](https://thinkingmachines.ai/news/inkling-2-small/), they "moved beyond unit-test rewards alone", with "an agentic RL grader that evaluated how the model approached the problem, not just whether its code passed". [TEMPO](https://studio.dots.ai/dots/tempo-blog.html) is really another cool method that uses Agentic Judges with test time scaling to really give fine-grained credit, though large scale verification of this is yet to be seen. 

I think even with my experiments, it has further reinforced (pun intended) in my mind that credit assignment is a problem that we need to make a lot of progress on, and will truly unlock frontier capabilities in even non-verifiable and very hard domains such as writing. 

In further posts, I honestly would love to explore more non verifiable domains and how do we ensure/instill a sense of taste in the models.

## Conclusion

I started with a simple question: can a judge that grades individual spans of a rollout give RL a better signal than "right or wrong"? It can. The judge does know which steps matter towards the correct answer, but given the small scale of these experiments, we were only able to get the policy to commit faster. Honestly, it was also just a lot of fun: it pulled me deep into RL and back to pen and paper, figuring out how to measure these effects in a methodical way.

I think post-training and large-scale RL are here to stay. We're already seeing signs of post-training compute matching or exceeding pre-training compute, and I'm excited for it. Post-training feels a lot more creative than pre-training to me.

I had a ton of fun working on this, and I hope you learned something from it too! If you enjoyed this, I'd recommend checking out [arocket](https://x.com/ar0cket1)'s writing on this line of work; it's excellent. I'm always happy to debate and discuss any of this. Thank you for reading :)

## Appendix

<details>
<summary>Appendix A: tokenizer tripping the probe</summary>

The first time I ran this, the probe failed its own sanity check. Before testing the judge, the analysis checks that the meter works at all: at the very end of a finished trace, it should easily tell correct traces from wrong ones. I'd set the bar at an AUROC of 0.9 in advance. The first run came in at **0.885**, so by my own rules the probe was declared invalid and the judge results didn't count, even though they looked great.

The culprit was tokenization. In normal text, `\boxed{-5}` is tokenized as `{-` + `5`, but my probe built the prefix and the answer separately, so it fed the model `{` + `-` + `5`, a split it essentially never sees. Correct traces with negative answers were scored as if the model had no idea. 40 of the 177 distinct answers had a seam like this. I fixed it by tokenizing the forced ending and the answer together, wrote the change down before rerunning, and changed nothing else. The rerun passed (**0.918**), and the judge effect barely moved (0.104 → 0.105).

</details>

<details>
<summary>Appendix B: checking the judge's rewards</summary>

It would be easy to assume the judge mostly learned to say "stop repeating yourself". It didn't. Most of its credit went to actual mathematics:

<div class="fig-mount" data-src="/llm_credit_assignment_figures.html" data-fig="fig-share"></div>

Valid progress and verified corrections together account for over 60% of everything the judge handed out. 

My favourite example comes from a trig minimisation problem early in training. The judge rewarded a correct derivative (+12.75), penalised a broken second-derivative calculation, and then gave **+39.5** to a first-derivative sign argument that genuinely repaired the solution. 
<div class="fig-mount" data-src="/llm_credit_assignment_figures.html" data-fig="fig-trigtrace"></div>

It wasn't perfect, though. In the same trace, the judge labelled the model's first clean write-up of the solution as "repetition" and penalised it (−57.4), enough that the whole trace ended up with negative judge credit despite being correct.

</details>

---

A small note: experiments like this take a *lot* of compute. Every training run, eval and probe here ran on GPUs from a compute grant from [Lium](https://lium.io), so a huge thank you to them for making this possible. I'd love to keep going down this line of work, so if you're interested in sponsoring compute for experiments like these, please reach out via X or email.