Build-A-Chip-Bench: Why AI Agents Fail at Physical Silicon Even When Code Compiles

The era of judging hardware AI agents by whether they can output passing Verilog snippets is officially over. On Wednesday, semiconductor AI startup ChipAgents—in close technical collaboration with NVIDIA—unveiled Build-A-Chip-Bench, the first holistic evaluation suite testing whether autonomous AI agents can take an electronic design specification from natural language all the way to a verified, timing-clean Graphic Design System (GDSII) layout ready for fabrication.

The results deliver a sobering reality check to claims that autonomous agents are on the verge of designing next-generation silicon: across five frontier language models tested under standardized containerized conditions, the top-performing agent—Anthropic's Claude Fable 5.1—achieved an overall score of just 16.6%. OpenAI's GPT-6 Astra, despite passing functional simulation on 85% of tasks, scored an aggregate 8.7% because its designs collapsed during downstream physical placement and routing.

On average, the industry's most advanced reasoning agents passed functional verification on roughly 40% of benchmarks, but only 10% produced physical implementations that satisfied real-world power, performance, and area (PPA) constraints.

The Saturation of 'Point Benchmarks'

Over the past eighteen months, AI labs have claimed sweeping victories on hardware benchmarks. On suites like ChipBench, frontier models surged from 30.7% pass rates in early 2026 to over 90.6% with Claude Opus 5.5. Similarly, on the Chinese Academy of Sciences' CVDP, models clustered into an uninformative ceiling around 65% to 70%, with benchmark saturation masking severe real-world shortcomings.

The flaw with existing benchmarks was structural: they evaluated isolated "point tasks." An agent was handed a markdown description of an arithmetic unit, a FIFO buffer, or an SPI controller, and asked to generate the Register-Transfer Level (RTL) code or write a corresponding testbench. If the Verilator simulation executed without assertions firing, the benchmark chalked up a win.

In commercial silicon engineering, functional correctness at the RTL level is barely 20% of the battle. An elegant architectural description that behaves identically to spec in cycle-accurate simulation can be completely unroutable, blow past thermal and clock budgets, or trigger catastrophic hold-time violations once mapped onto actual standard cells.

Build-A-Chip-Bench closes this loop by forcing agents through the complete 12-stage digital integrated circuit (IC) lifecycle:

1. Workload Specification
2. High-Level Architecture
3. Microarchitectural Specification
4. RTL Implementation (SystemVerilog)
5. Functional Verification (Hidden Verification IP)
6. Logic Synthesis
7. Floorplanning
8. Standard Cell Placement
9. Clock-Tree Synthesis (CTS)
10. Global and Detailed Routing
11. Static Timing Analysis (STA) & Power Signoff
12. Physical Verification (DRC & LVS) to Clean GDSII

The Hard Gates: How the Score Is Calculated

Developed by ChipAgents researchers Surya Gunukula, Jerry Yicheng Wang, and Kevin Zhang alongside NVIDIA's computational engineering group, Build-A-Chip-Bench evaluates agents via a dual-gate scoring function.

To prevent reward hacking, each agent runs in an isolated sandbox driven by the OpenCode harness. Grader scripts and gold verification IP sit in separate, inaccessible containers. Each problem presents a real-world task—such as architecting a custom INT4 matrix-vector multiplication engine for a Qwen3-0.6B inference workload using the open Nangate45 cell library at 400 MHz, restricted to a die area under 6.0 mm² and a power budget under 10 W.

Scoring requires passing two non-negotiable gates before quality is even measured:

• Gate 1 (Functional Pass): The synthesized RTL must pass complete verification suites against the hidden testbench.
• Gate 2 (GDS Pass): The design must successfully finish physical flow via OpenROAD—achieving zero Design Rule Check (DRC) violations, zero Layout Versus Schematic (LVS) mismatches, and clean hold timing.
• PPA Optimization: Only designs passing both Gate 1 and Gate 2 receive a PPA score (0–100%) measuring normalized frequency, silicon area, and dynamic power against constraints.

The overall benchmark score is strictly calculated as: $$\text{Score} = \text{Gate}_1 \times \text{Gate}2 \times \text{Score}{\text{PPA}}$$

If a design fails routing or introduces a hold violation, its score is an uncompromising zero.

Model	Functional Pass	GDS Pass	PPA vs. Constraint	Aggregate Score
Claude Fable 5.1	65.0%	75.0%	25.5%	16.6%
GPT-6 Astra	85.0%	85.0%	10.2%	8.7%
Gemini 3.8 Flash	20.0%	45.0%	4.1%	0.8%
Kimi K3	25.0%	25.0%	1.3%	0.2%
GLM-5.3	15.0%	5.0%	0.0%	0.0%

Why GPT-6 Astra Lost Despite 85% Functional Accuracy

The most revealing data point in the release is the divergence between OpenAI's GPT-6 Astra and Anthropic's Claude Fable 5.1.

GPT-6 Astra proved to be a computational powerhouse at code generation. It achieved an 85% functional verification rate and an 85% GDS completion rate—leading all models in raw routability. Yet its final score was nearly half that of Fable 5.1 (8.7% vs. 16.6%).

The failure mode boils down to what the researchers term destructive optimization. When Astra identified a negative slack timing violation during Static Timing Analysis, its reactive agentic loop attempted to rewrite RTL modules to pipeline critical paths or reduce logic depth. In doing so, it routinely broke previously verified functional behaviors. As the authors observed:

"Agents rarely treat keeping existing behavior intact as a constraint while they optimize."

While Astra frequently synthesized code that completed routing, its solutions took the path of least resistance—oversizing gates, inserting excessive buffer trees, and blowing through thermal limits, netting a dismal 10.2% PPA score. Fable 5.1 demonstrated superior macro-planning: it was more conservative in altering working RTL, reasoned more effectively across abstraction boundaries, and made trade-offs at the microarchitecture level rather than hacking netlists after the fact.

The Real Bottleneck: Tool Latency and Heterogeneous Feedback

Why are general reasoning agents struggling so acutely in Electronic Design Automation (EDA)?

• Asynchronous, Multi-Hour Feedback: Unlike software development, where a compiler responds in seconds, running placement, clock-tree synthesis, and global routing through OpenROAD takes tens of minutes to hours. Frontier agents trained on conversational fast-twitch loops struggle to preserve state across extended horizons when error logs span 100,000 lines of timing reports.
• Cryptic Error Manifolds: When a modern compiler encounters a syntax error, it outputs a precise file and line number. When a physical synthesis engine reports negative slack or congestion in a metal layer, the root cause could be an unbuffered long wire, improper pin placement, or a flawed architectural choice made ten steps earlier. Current LLMs treat these reports as flat text rather than multidimensional spatial problems.
• Fragility of Iterative Repair: The moment an agent modifies RTL to satisfy a physical constraint, it restarts the verification cycle. Current models lack the hierarchical memory required to maintain architectural intent across cyclic debugging loops.

As Tim Costa, vice president and general manager of industrial and computational engineering at NVIDIA, stated during the announcement:

"Chip design is a defining test of AI's ability to make decisions that hold up under the demands of the physical world. Build-A-Chip-Bench tests whether agents' decisions hold up from design intent through physical implementation, giving developers a clear view of where reasoning and tool use must improve to deliver impactful autonomous engineering."

Where Chip-Design Agents Go From Here

The launch of Build-A-Chip-Bench marks a vital inflection point. Just as the release of SWE-bench in late 2023 dispelled the illusion that LLMs were autonomous senior software engineers, Build-A-Chip-Bench punctures the hype that tapeout-ready silicon can be conjured from prompt engineering.

Solving end-to-end chip design will not come simply from larger context windows or brute-force chain-of-thought tokens. Closing the 83.4% gap will require:

1. Domain-Specific System-One Surrogates: Integrating non-generative ML surrogates for rapid timing and congestion prediction directly inside the agent loop, rather than forcing the LLM to wait on monolithic tool passes.
2. Formal Verification Guards: Implementing strict invariant lockers that programmatically prevent an agent from altering functionally proven RTL modules during physical PPA tuning.
3. Multimodal Spatial Reasoning: Equipping agents to ingest visual routing heatmaps and congestion grids rather than forcing them to parse dense ASCII routing summaries.

Silicon remains the bedrock of the AI ecosystem. If frontier models are to participate in architecting the hardware that trains their successors, they must prove they can respect physics, timing, and area—not just write code that simulates in a vacuum.