Why Majority Voting Fails Agents: Google Unveils VeriHarness for Long-Horizon AI

When scaling test-time compute in mathematical reasoning and coding, LLMs rely on unambiguous ground truths: compilers catch syntax bugs, unit tests validate runtime logic, and formal verifiers prove mathematical lemmas. But when enterprise AI agents perform long-horizon work—drafting 40-page financial memorandums, refactoring complex codebases across nested directories, or updating multi-tab financial models—those neat verification oracles vanish.

Standard industry practice has been to rely on self-consistency and majority voting across repeated rollouts. New research from Google Cloud AI Research and the University of Cambridge reveals that this heuristic critically breaks down in production. In their paper, VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks, researchers Caiqi Zhang, Rujun Han, Zifeng Wang, Zoey CuiZhu, Nigel Collier, Tomas Pfister, and Chen-Yu Lee demonstrate that 34% of unanimous, consensus claims across frontier agent rollouts are completely false. Meanwhile, in 74% of cases where rollouts disagreed, the correct answer existed inside the candidate pool—yet naive voting discarded the true solution over half the time.

To solve this "verifier dilemma," the team developed VeriHarness, an architecture that transforms the base generator itself into an autonomous verifier equipped with environment tools, adversarial challenger loops, and a self-improving skill library.

---

The Illusion of Consensus: Why Voting Fails Complex Agents

In frontier deployment, systems cannot rely on a "super-intelligent judge" model because the generator is already the most capable model available. When researchers sampled 10 independent rollouts per task using frontier models like Claude Opus 4.8 and Gemini 3.5 Flash across high-stakes benchmarks like APEX-Agents (spanning banking, consulting, and law), they uncovered two failure modes:

• Common-Mode Hallucination and Omission: When multiple rollouts agree, they frequently share the same underlying inductive biases, misinterpret subtle prompt constraints identically, or skip the same obscure edge case. As a result, consensus is not a proxy for correctness—over one-third of consensus claims failed rigorous expert rubrics.
• The Disagreement Paradox: When rollouts diverge on numerical calculations, factual dates, or code logic, naive frequency-based voting selects the correct candidate in only 47% of instances. However, at least one rollout in the disputed set had the exact correct answer in 74% of cases.

In short: consensus conceals errors, while disagreement conceals solutions.

                                [ N Candidate Rollouts ]
                                           │
                   ┌───────────────────────┴──────────────────────┐
                   ▼                                              ▼
       [ Disagreement Resolver ]                       [ Consensus Challenger ]
   • Isolates disputed claims (V_p)                • Flags unanimous claims (C_shared)
   • Gathers environment evidence                  • Actively seeks refuting proof
   • Updates belief distributions                  • Checks for omitted requirements
                   └───────────────────────┬──────────────────────┘
                                           ▼
                             [ Evidence-Backed Adjudicator ]
                               • Scores candidate artifacts
                               • Generates targeted patch plan
                               • Executes final artifact revision

---

Inside VeriHarness: Resolver, Challenger, and Adjudicator

Instead of treating LLM verification as a passive zero-shot evaluation, VeriHarness wraps the model in an active agentic harness with direct terminal and workspace access. The system operates through three primary mechanisms:

1. The Disagreement Resolver

For every disputed claim $p$ with candidate values $\mathcal{V}p = {v_1, v_2, \dots, v_m}$, the resolver does not vote. Instead, it extracts the underlying computational graph or citation trail and executes targeted verification actions in the workspace—querying raw CSVs, running localized Python assertions, or grepping file logs. It maintains a Bayesian belief state $b_p$ over candidate values based on accumulated environment evidence $e{<k}$, updating its confidence until an empirical claim is proven or refuted.

2. The Consensus Challenger

Rather than accepting unanimous outputs as settled truth, VeriHarness deploys an adversarial challenger agent. The challenger is tasked with:

• Searching for refuting evidence: Interrogating agreed-upon metrics against source files and constraint boundaries.
• Requirement audit: Cross-referencing delivered multi-file artifacts against the original task specification $s$ to locate omitted business rules or overlooked formatting constraints that every single rollout neglected.

3. Evidence-Backed Adjudication & Targeted Revision

Once the resolver and challenger compile structured evidence logs, the Adjudicator scores each rollout. Crucially, VeriHarness does not merely select the best rollout (Best-of-$N$ reranking); it formulates an explicit revision plan. The harness executes a surgical editing pass that splices verified claims into the highest-scoring candidate artifact, fixing isolated errors while leaving sound logic intact.

---

Empirical Results: Outperforming Best-of-N and Majority Voting

The authors evaluated VeriHarness across five long-horizon workspace benchmarks spanning complex enterprise workspace tasks:

• APEX-Agents in investment banking, legal drafting, and management consulting.
• Workspace-Bench Lite requiring navigation through large file-dependency graphs.
• WorkBuddy Bench covering multi-domain code patches, office document generation, and web navigation.
• SpreadsheetBench 2 & JobBench: Complex financial spreadsheet modeling and professional job-simulation workloads.

Benchmark	Single Rollout (Flash)	Majority Voting (Flash)	VeriHarness Select (Flash)	VeriHarness + Revision (Flash)	VeriHarness + Revision (Opus 4.8)
APEX-Agents	48.1	47.3	54.8	54.8 (+6.7)	58.2 (+7.1)
Workspace-Bench Lite	56.5	57.0	65.4	65.4 (+8.9)	69.8 (+8.2)
WorkBuddy Bench	73.1	74.0	78.2	78.2 (+5.1)	82.4 (+5.8)
SpreadsheetBench 2	25.8	27.0	32.2	32.2 (+6.4)	36.1 (+6.0)
JobBench	31.5	32.4	36.2	36.2 (+4.7)	45.8 (+4.9)
Overall Average	47.0	47.5	53.4	53.2 (+6.2)	58.5 (+6.4)

Across all benchmarks, standard majority voting delivered negligible gains (+0.5 points on average over a single rollout). When evidence-backed revision was applied, performance surged by +6.2 points (Flash) and +6.4 points (Opus 4.8) over baseline rollouts.

Furthermore, the researchers evaluated VeriHarness across standard developer CLIs (Gemini CLI, Claude Code, and Codex scaffolds), demonstrating that the protocol retains its advantages even when embedded in general-purpose coding agents.

---

Self-Improving Verification Skills and Open Dataset Release

Beyond static rules, VeriHarness introduces an evolving verification skill library. When the verifier fails to detect an error during development tasks, failure feedback is converted into reusable programmatic verification skills (such as specialized cross-table consistency validators or regex discrepancy finders). Evolving from empty adds 11.0 points on held-out tasks.

To catalyze research in agentic evaluation where ground truth answers cannot be trivially extracted, the Google Cloud AI team open-sourced the complete rollout corpus at veriharness.com and GitHub (google-research/veriharness).

The release includes ~26,000 full-horizon rollouts generated across Gemini 3.5 Flash and Claude Opus 4.8—representing more than $100,000 in API compute costs—providing the AI research community with an unprecedented foundation for studying test-time verification, multi-rollout consensus dynamics, and autonomous revision in enterprise workflows.