R-Zero Cracks Self-Evolving LLMs with Zero Human Data

The primary scaling bottleneck for frontier reasoning models is no longer compute—it is data curation. While reinforcement learning with verifiable rewards (RLVR) powered breakthroughs like DeepSeek-R1 and OpenAI's o-series, RLVR has remained tethered to vast corpora of human-crafted problems, test cases, and formal verification harnesses. When an AI can only practice on problems humans already know how to write, what it can learn is limited by what people have written.

Enter R-Zero, an autonomous self-evolving reasoning framework developed by researchers at Tencent AI Seattle Lab and Washington University in St. Louis that generates its own training questions and labels from a base model, with no human-written tasks or answer labels. (The base model itself was still pretrained on human text.)

By initializing two distinct agents from a single base model—a Challenger and a Solver—and having them co-evolve using Group Relative Policy Optimization (GRPO), R-Zero creates an autonomous curriculum generator. In the paper's results, this lifts a small model's average benchmark scores by roughly 6 to 7 points, without any external training data. The same paper also reports that the answer labels get less accurate over time and that scores eventually fall, covered below.

                      ┌────────────────────────────────────────┐
                      │             Base LLM Checkpoint        │
                      └───────────────────┬────────────────────┘
                                          │
                         ┌────────────────┴────────────────┐
                         ▼                                 ▼
               ┌───────────────────┐             ┌───────────────────┐
               │    Challenger     │             │      Solver       │
               │    (Q_theta)      │             │     (S_phi)       │
               └─────────┬─────────┘             └─────────┬─────────┘
                         │                                 │
                         │ Proposes Problems               │ Generates N Rollouts
                         ▼                                 ▼
               ┌─────────────────────────────────────────────────────┐
               │  Uncertainty Engine (Reward peak at 50% accuracy)   │
               │  Majority-Vote Pseudo-Labeling via Self-Consistency │
               └─────────────────────────┬───────────────────────────┘
                                         │
               ┌─────────────────────────┴───────────────────────────┐
               │ Challenger: GRPO update on solver uncertainty       │
               │ Solver: GRPO update on pseudo-labeled challenges    │
               └─────────────────────────────────────────────────────┘

The Engine: The Edge-of-Capability Uncertainty Reward

Previous attempts at autonomous data generation suffered from two failure modes: triviality collapse (models generate easy problems they already know how to solve) or verifiability collapse (models generate impossible, ill-posed, or uncheckable riddles).

R-Zero bypasses this by framing problem formulation as an uncertainty-maximization game. The framework splits learning into two alternating optimization phases:

• Challenger Training Phase: The Challenger ($Q_\theta$) autoregressively generates candidate problems $x_i$. For each problem, the frozen Solver ($S_\phi$) samples $m = 10$ independent rollouts. The Challenger is rewarded based on the Solver's empirical uncertainty:

$$\text{Reward} \propto 1 - 2\left|\hat{\rho}(x_i \mid S_\phi) - 0.5\right| - \lambda \cdot \text{RepetitionPenalty}$$

where $\hat{\rho}$ represents the solver's self-consistency score. When the Solver agrees with itself on every try, the problem is too easy and earns zero reward; when it never agrees, the reward is also zero. The Challenger is maximally rewarded when the Solver's success rate hovers around 50%—the edge of its capability.

• Solver Training Phase: Once the Challenger produces a batch of boundary-pushing tasks, the Solver is unfrozen. Each generated task is pseudo-labeled by majority vote over the Solver's ten tries, and only questions where a middle share of the tries agree are kept (the paper keeps 3 to 7 of 10), which filters out both the too-easy and the confusing ones. The Solver is then updated via GRPO on these self-generated problems using binary verification against its own consensus pseudo-labels.

As the Solver masters these edge cases, the Challenger's reward for those specific prompts drops to zero, forcing it to discover more sophisticated problem structures, longer reasoning chains, and complex logical constraints.

Why Two Separate Models Beat One

The R-Zero ablation tested whether the Challenger and Solver really need to be separate models. In Single-R-Zero, one shared-parameter model plays both roles.

• Earlier peak: the unified model's math score peaks after the very first iteration and degrades right after. The two-model version peaks later (49.12 at step 45) and only falls after the third iteration.
• Noisier labels: the single model's pseudo-labels are less accurate at every stage. In the first iteration, 63.4% against 71.0% for the two-model setup. The authors hypothesize that having the problem-setter and the solver come from the same model leads to overconfidence from internal bias.
• Both degrade: the separate-model version is not immune. Its math score also drops at step 60.

Math accuracy (%) across training steps (paper's Table 6):

Step 15   R-Zero, two models: 48.06    Single-R-Zero: 47.31
Step 30   R-Zero, two models: 48.44    Single-R-Zero: 46.95
Step 45   R-Zero, two models: 49.12    Single-R-Zero: 45.57
Step 60   R-Zero, two models: 46.52    Single-R-Zero: 43.89

Empirical Results: What the Paper Reports

R-Zero was tested on small open models (the Qwen3 and OctoThinker families, up to 8 billion parameters), so read the numbers as a proof of concept rather than a general result:

• Qwen3-4B-Base: the abstract reports +6.49 points on math-reasoning benchmarks and +7.54 points on general-domain reasoning benchmarks. The paper's own tables list slightly different sizes (for example +7.36 for math in Table 1 and +4.81 for general reasoning in Table 2), and it does not report seeds or confidence intervals.
• A harder curriculum: testing a fixed Solver (Step 15) on questions the Challenger wrote later, with GPT-4o supplying the reference answers, its accuracy fell from 59.0% on the Step 15 questions to 47.0% on the Step 45 questions. The Challenger really does write harder questions as training goes on.
• A worsening answer key: the same analysis found the accuracy of the self-made labels dropping from 79.0% to 63.0% by the third iteration (200 questions per set). The authors call this a critical trade-off and a potential bottleneck.
• Later degradation: the authors also note a "consistent and concerning trend of performance degradation" in later iterations across all models, such as the 4B model's math score falling from 49.12 to 46.52 at step 60.
• Limits: the method relies on domains where correctness can be objectively verified; open-ended tasks such as creative writing remain difficult.

What R-Zero Does and Doesn't Show

R-Zero is a concrete proof of concept for label-free, task-free self-evolution in domains where an answer can be checked, such as math. A Challenger that keeps the Solver at the edge of its ability, plus a majority vote as the answer key, lifted a small model's scores by several points without new human-written data.

It does not yet show open-ended self-improvement. The base model still came from human text, the self-made answer key got less accurate as the questions got harder, and scores slipped when training went on. Whether this scales to larger models and fuzzier tasks is still open.