Get the app

What is test-time compute?

Test-time compute is the computational budget an AI model spends generating and refining reasoning steps during inference to solve difficult queries.

What is Test-time compute?

Test-time compute (also called inference-time compute or inference scaling) is the computational resource budget allocated to an AI model while it generates and evaluates answers at runtime, rather than during its initial training phase. Instead of relying on a single, rapid forward pass to output a response, the system spends additional floating-point operations (FLOPs) exploring multiple solution paths, generating internal step-by-step reasoning tokens, and verifying candidate outputs before returning a final answer. This paradigm allows models to scale their accuracy on complex tasks by thinking longer on hard inputs.

An everyday analogy is taking a difficult math examination. A standard language model acts like a student forced to write their final answer instantly from memory without pausing. Test-time compute equips the student with scratch paper, extra time, and the ability to draft intermediate steps, double-check equations, discard dead ends, and refine the solution before handing in the exam paper.

How it works

Test-time compute operates by trading runtime compute (latency and token generation FLOPs) for higher output accuracy on challenging queries. In practice, practitioners scale test-time compute using four primary mechanisms:

  1. Extended Autoregressive Thinking (Chain-of-Thought): Rather than outputting the answer directly, the model is prompted or reinforcement-learning-tuned (e.g., via Group Relative Policy Optimization, or GRPO) to output long sequences of internal reasoning tokens inside delimited blocks (such as <think> tags). The model deconstructs the problem, explores hypotheses, identifies mistakes in prior lines, and corrects its own trajectory before outputting the final response.
  2. Parallel Sampling and Best-of-N Rejection Sampling: The system samples N distinct candidate completions in parallel from the base model at a non-zero temperature. A scoring mechanism evaluates each candidate and selects the highest-scoring response. Scoring can be performed using:
    • Outcome Reward Models (ORMs): Evaluates the entire final response as a single scalar score.
    • Self-Consistency (Majority Voting): Clusters multiple sampled solutions and selects the consensus answer.
  3. Step-Level Search with Process Reward Models (PRMs): For structured problems like mathematics or code generation, parallel sampling can be inefficient. Instead, search algorithms such as Beam Search, Lookahead Search, or Monte Carlo Tree Search (MCTS) evaluate intermediate reasoning steps using a PRM. The PRM scores each reasoning step individually, enabling the search algorithm to prune incorrect paths early and backtrack to promising intermediate branches.
  4. Adaptive Compute-Optimal Allocation: Difficult problems require substantial search and verification, whereas simple factual queries do not benefit from extra thinking tokens. Compute-optimal scaling strategies assess query difficulty dynamically, allocating large verification and token budgets to complex prompts while serving standard single-pass decodings for trivial requests.

Test-time compute vs Pre-training compute

Pre-training compute refers to the massive, fixed compute budget (typically thousands of GPU/TPU cluster hours) spent upfront updating model weights across trillions of tokens of text. Once pre-training ends, the model parameters remain frozen. Test-time compute, by contrast, is a dynamic runtime budget spent per query during inference without modifying model weights. While scaling pre-training compute produces larger or more capable base models, scaling test-time compute allows even smaller base models to dedicate extra runtime compute to search through possibilities and verify complex logic on demand.

A practical example

The following Python example uses Hugging Face TRL (trl.extras.BestOfNSampler) to implement parallel test-time compute scaling. It generates multiple candidate responses for a prompt and uses a reward pipeline to select the highest-scoring answer.

from transformers import AutoTokenizer, pipeline
from trl import AutoModelForCausalLMWithValueHead
from trl.core import LengthSampler
from trl.extras import BestOfNSampler

model_name = "gpt2"
device = "cuda:0"

model = AutoModelForCausalLMWithValueHead.from_pretrained(model_name).to(device)
tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token

# Proxy reward model pipeline to score candidate reasoning quality
reward_pipe = pipeline("sentiment-analysis", model="distilbert/distilbert-base-uncased-finetuned-sst-2-english", device=device)

def score_fn(samples):
    return [float(out["score"]) for out in reward_pipe(samples)]

# Allocate test-time compute: sample_size=8 generates 8 candidates per query
sampler = BestOfNSampler(model, tokenizer, score_fn, length_sampler=LengthSampler(30, 50), sample_size=8)

query_tensors = tokenizer("Solve the step-by-step logic puzzle:", return_tensors="pt").input_ids.to(device)
best_output = sampler.generate(query_tensors, device=device)
print(tokenizer.decode(best_output[0], skip_special_tokens=True))

In this setup, setting sample_size=8 increases the test-time compute budget by a factor of eight relative to single-pass greedy decoding. The sampler generates eight diverse candidate sequences, scores each against the reward model, and returns the top candidate, filtering out low-scoring or incoherent generations.

Tools and implementations

Tool What it is for Link
Hugging Face TRL Training Process Reward Models (PRMTrainer) and running Best-of-N candidate search huggingface.co/docs/trl
vLLM High-throughput LLM serving engine with parallel sampling, speculative decoding, and reasoning token generation github.com/vllm-project/vllm
SGLang High-performance inference engine optimized for structured decoding and multi-branch tree search execution github.com/sgl-project/sglang
DeepSeek-R1 Open-weights reasoning models trained via GRPO that scale test-time compute natively through reasoning tokens github.com/deepseek-ai/DeepSeek-R1

Where it goes wrong

Scaling test-time compute is not a universal fix and introduces distinct failure modes:

  • Reward Hacking and Verifier Exploitation: When search algorithms evaluate thousands of candidate paths against an imperfect Process or Outcome Reward Model, the generator often discovers adversarial token sequences that score highly on the reward model despite being logically invalid (Goodhart's Law).
  • Hallucination Cascades: If a model lacks foundational domain knowledge or starts with an incorrect premise, extending its reasoning chain often amplifies errors. Additional compute is spent rationalizing an incorrect assumption rather than self-correcting.
  • Diminishing Marginal Returns: Studies (such as Snell et al., 2024) demonstrate that on simple queries, additional test-time compute yields zero accuracy gains, while on impossible queries outside the model's base capability, search returns no valid candidates regardless of budget size.
  • Latency and Cost Multipliers: Generating hundreds of reasoning tokens or branching across multiple tree paths multiplies GPU memory usage, bandwidth consumption, and Time-to-First-Token (TTFT), making large test-time budgets impractical for latency-critical user applications.

Why it matters now

Test-time compute provides a second axis of AI scaling alongside traditional pre-training compute, enabling developers to achieve frontier-level reasoning capabilities without training ever-larger models from scratch.

On January 22, 2025, DeepSeek released DeepSeek-R1 (arXiv:2501.12948), showing that reinforcement learning without preliminary supervised fine-tuning incentivizes emergent step-by-step reasoning tokens during inference, scaling pass@1 performance on the AIME 2024 mathematics benchmark from 15.6% up to 71.0%, and reaching 86.7% with majority voting.

FAQs

How does test-time compute relate to models like OpenAI o1 and DeepSeek-R1? Models like OpenAI o1 and DeepSeek-R1 are trained with reinforcement learning to generate explicit internal chains of thought before producing a final answer. They scale test-time compute directly by dynamically allocating more reasoning tokens to complex queries.

What is the difference between an Outcome Reward Model (ORM) and a Process Reward Model (PRM)? An ORM evaluates the final output of a completed sequence as a whole, providing a single scalar score. A PRM evaluates each intermediate reasoning step, allowing search algorithms to prune faulty reasoning branches early in generation.

Can test-time compute make a small model outperform a larger model? Yes. Research by Snell et al. (2024, arXiv:2408.03314) found that on tasks where a smaller base model attains non-trivial initial success, compute-optimal test-time search can allow the smaller model to outperform a 14x larger baseline model without additional inference compute.

Does allocating more test-time compute guarantee a correct answer? No. If the base model lacks the factual knowledge or core capability required for the problem, generating more tokens or searching larger candidate trees often leads to circular reasoning or reward model exploitation.

How does majority voting differ from Best-of-N search? Best-of-N search relies on an external reward model to rank and select the top candidate from N samples. Majority voting (self-consistency) extracts the final answer from each candidate and selects the answer that appears most frequently across all sampled generations without requiring a separate reward model.

Sources

Primary

Written by an AI pipeline from the sources above and checked against them. Methodology · Report an error

The daily AI brief, on your phone.

Feed, daily deep-dive and bytes — readable offline, with push alerts for the topics you follow.

Get it on Google Play