Get the app

Language Models Are Hidden Reasoners: How LaTRO Unlocks Logic Without External Reward Models

Salesforce AI Research treats reasoning as a latent variable, enabling LLMs to self-reward and outperform supervised fine-tuning using only question-answer pairs.

Language models do not need massive datasets of human-annotated thought chains or brittle external process reward models (PRMs) to master complex multi-step reasoning. A research team at Salesforce AI Research has introduced LaTent Reasoning Optimization (LaTRO), a framework that suggests standard pre-trained LLMs already harbor dormant reasoning capabilities within their weights. By framing step-by-step thinking as an unobserved latent distribution and optimizing it via variational inference, LaTRO enables models to autonomously generate, evaluate, and reinforce their own rationales using nothing more than raw question-and-answer pairs.

On the standard GSM8K mathematical reasoning benchmark, LaTRO boosted zero-shot accuracy across Phi-3.5-mini, Mistral-7B, and Llama-3.1-8B by an average of 12.5% over base models and 9.6% over supervised fine-tuning (SFT)—all without ever exposing the models to gold-standard human reasoning traces during training.


The Pitfalls of Supervised Rationales and External Verifiers

The dominant paradigms for training reasoning models suffer from fundamental scaling bottlenecks:

  • Supervised Fine-Tuning (SFT) Overfitting: Training an LLM directly on curated Chain-of-Thought (CoT) traces forces the model into rigid syntactic patterns. When the model encounters out-of-distribution questions, its borrowed reasoning structures collapse.
  • The External Reward Bottleneck: Reinforcement Learning from Human Feedback (RLHF) and Process Reward Models (PRMs) require training a separate verifier network or collecting step-by-step human annotations. PRMs are notoriously prone to reward hacking, value drift, and out-of-distribution failure.
  • Distillation Dependency: Distilling reasoning traces from closed frontier models introduces data contamination, licensing issues, and ceiling effects capped by the teacher model's idiosyncrasies.

LaTRO bypasses this entire pipeline by asking a simpler question: Can a model use its own understanding of the correct answer to judge the validity of the path it took to get there?


The Variational Formulation: Reasoning as a Latent Variable

In standard language modeling, the network maximizes the direct conditional probability of target answer y given question x, denoted as log π_θ(y | x). However, for complex multi-step problems, jumping straight to y produces low likelihoods because the model lacks the intermediate compute to resolve dependencies.

LaTRO introduces reasoning trajectory z as an unobserved latent variable. The marginal likelihood of reaching answer y is the integral over all possible reasoning paths z:

p(y | x) = ∫ p(y | x, z) p(z | x) dz

Because computing this integral directly across vocabulary space is intractable, Salesforce researchers applied the variational principle to derive an Evidence Lower Bound (ELBO):

log π_θ(y | x) ≥ E_{q(z | x)} [log π_θ(y | x ⊕ z)] - D_KL[ q(z | x) || π_0(z | x) ]

Here:

  • q(z | x) is a variational proposal distribution representing the rationales sampled by the model.
  • π_θ(y | x ⊕ z) is the conditional likelihood that the model generates the ground-truth answer y after appending rationale z to question x.
  • D_KL[ q(z | x) || π_0(z | x) ] acts as an information-theoretic regularizer, preventing the reasoning distribution from drifting too far from the reference base policy π_0.
+--------------------------------------------------------------------------+
|                             LaTRO Pipeline                               |
|                                                                          |
|  [Question x]                                                            |
|       │                                                                  |
|       ▼                                                                  |
|  [Policy π_θ] ───> Sample Candidate Rationales: {z_1, z_2, ..., z_K}     |
|       │                                                                  |
|       ▼                                                                  |
|  [Self-Reward Evaluation]                                                |
|       │ Compute intrinsic likelihood: log π_θ(y | x ⊕ z_k)               |
|       │ High likelihood  ──> High Reward (valid logical bridge)          |
|       │ Low likelihood   ──> Low Reward  (hallucination / dead end)      |
|       ▼                                                                  |
|  [Variational Update]                                                    |
|       │ Maximize ELBO with KL regularization against base policy π_0     |
|       ▼                                                                  |
|  [Self-Improved Policy π_θ+]                                             |
+--------------------------------------------------------------------------+

Self-Rewarding: Intrinsic Verification Without a Critic

The central breakthrough of LaTRO is its self-rewarding mechanism. Rather than training an external critic or reward model to output scalar scores, the model leverages its own next-token generative probability as the reward function.

  1. Candidate Exploration: For a training instance (x, y), the current model π_θ samples K diverse candidate reasoning rationales {z_1, z_2, ..., z_K}.
  2. Likelihood Attribution: The model evaluates how easily it can emit the correct final answer y when primed with each rationale x ⊕ z_k. If rationale z_1 contains coherent logical deductions, the probability π_θ(y | x ⊕ z_1) spikes. If rationale z_2 contains mathematical errors or hallucinations, the probability drops sharply.
  3. Policy Alignment: The model updates its parameters to increase the likelihood of producing rationales with high self-reward scores while pruning degenerated trajectories.

Because the model concurrently acts as both the reasoning generator and the evaluator, improvements in its reasoning ability directly enhance its evaluation accuracy, creating a compounding self-improvement loop.


Empirical Performance Across Model Families

Salesforce AI Research evaluated LaTRO across multiple architectures, testing small-to-medium scale open models on mathematical reasoning (GSM8K) and scientific question answering (ARC-Challenge).

Key Benchmark Results on GSM8K:

  • Phi-3.5-mini: Rose from a base zero-shot baseline of 72.9% to 87.6% with LaTRO fine-tuning (and 90.5% with self-consistency), outpacing standard SFT (75.8%) by 11.8 percentage points.
  • Mistral-7B: Improved from 47.8% base accuracy to 67.3% under LaTRO greedy decoding, beating golden-trace SFT (57.2%) by 10.1 points.
  • Llama-3.1-8B: Jumped from 76.8% base accuracy to 80.1% (and 87.0% with self-consistency), demonstrating that the variational objective generalizes seamlessly across diverse pre-training corpora.
  • ARC-Challenge: Models achieved an average 5.2% accuracy increase over SFT under greedy decoding (and 8.1% with self-consistency), confirming that the self-rewarding signal applies equally well to multi-hop scientific reasoning.

Why Latent Optimization Reshapes Post-Training

LaTRO demonstrates that the bottleneck in open-source reasoning models has not been parameter scale or pre-training volume, but objective misalignment during post-training.

  • Decoupling from Proprietary Traces: Smaller models no longer need to be spoon-fed millions of synthetic step-by-step traces scraped from proprietary APIs. Given verified problem-answer pairs, models can explore and stabilize their own internal reasoning topologies.
  • Resistance to Goodhart's Law: Because the reward signal is derived directly from the model's exact conditional likelihood over ground-truth tokens rather than a surrogate reward head, the policy cannot "hack" the reward model with verbose, pleasant-sounding nonsense.
  • Synergy with Test-Time Search: Latent training refines the underlying generative distribution q(z | x). When combined with inference-time scaling techniques—such as self-consistency, beam search, or tree search—the improved base rationale quality provides a significantly better foundation for test-time compute.

As the industry transitions from raw pre-training scale to reinforcement learning and self-play post-training, LaTRO establishes a mathematically rigorous foundation for autonomous model self-improvement.

Sources

The day's AI news, in one Telegram post.

The stories that matter for people who build with AI, with the source for every claim. Free, no spam.

Join on Telegram
The daily AI brief, on your phone.

Feed, daily deep-dive and bytes — readable offline, with push alerts for the topics you follow.

Get it on Google Play