FDE-Gym: Why Frontier AI Agents Resolve Just 14 of 95 Real Enterprise Tasks

For the past two years, the AI industry has celebrated soaring pass rates on software agent benchmarks. Models now resolve synthetic GitHub issues, pass coding unit tests, and navigate command-line interfaces with claimed success rates topping 70% to 90%. Yet when those same agents are deployed into enterprise software delivery, they routinely stall in pilot purgatory.

A new benchmark, FDE-Gym: The Last Mile of AI Delivery (submitted by DeepLethe), exposes precisely why: real customer engineering is fundamentally not about clean code generation. It is about discovering unstated business constraints, steering misdirected client requests, navigating live production systems, and managing stakeholder communication costs without causing operational outages.

Evaluating 13 frontier model configurations across 95 end-to-end delivery tasks spanning 18 industry sectors, FDE-Gym delivers a brutal reality check: the highest-performing model, Claude Opus 5.5, achieves an average FDE score of just 0.56 and resolves only 14 out of 95 customer tasks.

---

The Forward-Deployed Engineer (FDE) Dilemma

In standard evaluations like SWE-bench, an agent receives a clearly scoped issue description, a reproduction script, and a test suite. Success is binary: did the unit tests pass?

In contrast, the Forward-Deployed Engineer (FDE) paradigm—pioneered in enterprise enterprise AI deployment—operates under chaotic conditions:

• Misdirected Requests: Enterprise sponsors frequently ask for the wrong solution or misdiagnose their core operational bottleneck.
• Unstated Constraints: Critical institutional knowledge (such as deprecated legacy database tables, unwritten compliance requirements, or rate-limiting firewalls) is never written in initial task briefs.
• Interactive Stakeholders: To find out what is actually needed, agents must query scripted human personas (directors, DBAs, frontline staff) who have finite patience, strict disclosure boundaries, and escalating communication costs.
• Business-Metric Evaluation: The output is not evaluated against unit assertions, but against Normalized Business Uplift ($U$) on held-out customer data, where 0 represents the baseline "do-nothing" state and 1.0 represents the human expert reference solution.

                      [ Client Problem Statement (Often Misdirected) ]
                                              │
                                              ▼
                                   ┌────────────────────┐
                                   │   Agent Iteration  │ ──► [ Query Scripted Stakeholders ]
                                   │  in Live Sandbox   │      (Incurs Contact Cost 'C')
                                   └────────────────────┘
                                              │
                         ┌────────────────────┴───────────────────┐
                         ▼                                        ▼
             [ System Incident / Outage ]              [ Modified Live System ]
             - Uplift dropped to U = 0                 - Evaluated on Held-Out Data
             - Task Marked Failed                      - Yields Business Uplift (U)
                                                                  │
                                                                  ▼
                                                      Final Score = U - C
                                              (Resolved = Score ≥ 0.8 & 0 Incidents)

---

The Architecture of FDE-Gym

FDE-Gym builds an operational sandbox across 20 real-world customer delivery scenarios adapted by industry experts from 18 distinct sectors.

To prevent memorization and surface robust agentic reasoning, each base scenario includes 2 to 7 variants altering crucial dependencies (such as which stakeholder possesses authorization keys or which database index is corrupt), yielding 95 distinct tasks tested against more than 130 scripted interactive personas.

The Four Instruction Levels (L0–L3)

To pinpoint where agentic reasoning breaks down, FDE-Gym evaluates agents across four progressive levels of specification:

1. L0 (Oracle Specification): The exact business problem, unstated constraints, schema gotchas, and evaluation metrics are explicitly laid out in the initial prompt.
2. L1 (Problem Stated): The core root cause is stated, but operational constraints and edge cases must be uncovered in the environment.
3. L2 (Vague Request): The client's intent is conceptually aligned with the goal, but poorly specified.
4. L3 (Raw Sponsor Request): The realistic deployment setting. The agent receives only the executive sponsor's verbatim request—which is often misdirected toward a symptom rather than the root problem.

Penalties, Outages, and Contact Overhead

Crucially, FDE-Gym incorporates real-world penalties: $$\text{FDE Score} = \max(0, U_{\text{adjusted}}) - C$$

Where:

• $U$ (Normalized Business Uplift): Gains achieved relative to the do-nothing baseline ($0.0$) and human reference ($1.0$).
• Incidents: If an agent triggers an unauthorized drop, database corruption, or security bypass, $U$ is immediately reset to $0.0$.
• $C$ (Stakeholder Contact Cost): Every ping to an engineer, clarification message, unnecessary trial run, and simulated waiting step deducts score points to penalize "shotgun debugging" and human-in-the-loop fatigue.
• Resolution Criterion: A task is considered "Resolved" only if the final score is $\ge 0.80$ with zero incidents.

---

The Results: Frontier Models Hit the Enterprise Wall

The benchmark evaluated 13 model configurations from 7 leading AI labs. The results demonstrate a stark disparity between theoretical reasoning capability and production execution.

Model Configuration	L0 Score (Oracle)	L3 Mean Score (Realistic)	95% Confidence Interval	Business Uplift ($U$)	Contact Cost ($C$)	Resolved Tasks (/95)	Negative Scores ($< 0$)
Claude Opus 5.5	0.82	0.56	[0.50, 0.62]	0.70	0.14	14	5
Claude Haiku 4.5	0.11	-0.33	[-0.42, -0.24]	-0.23	0.09	0	79

Critical Takeaways from the Data

• 1. Discovering Constraints is the Primary Chasm: When provided with full oracle instructions (L0), leading models hit respectable scores between 0.74 and 0.82. However, the moment constraints must be actively surfaced (moving from L0 to L1), scores collapse by 0.28 to 0.35 points. Interestingly, the jump from L1 to L3 (dealing with misdirected sponsor requests) caused minimal further drop, proving that constraint discovery, not prompt misdirection, is the primary agent bottleneck.

• 2. Information Retrieval Does Not Equal Task Delivery: A common assumption is that agents fail because they fail to ask the right questions. The benchmark data disproves this: on the 23 tasks where Claude Opus 5.5 uncovered 100% of all hidden facts from stakeholders, it only managed to achieve $\ge 0.8$ business uplift on 10 of them. Synthesizing complex multi-file codebase modifications in live production environments without introducing secondary bugs remains profoundly difficult.

• 3. Smaller Models Can Be Net-Destructive: Smaller lightweight models proved outright dangerous in production. Claude Haiku 4.5 yielded negative scores in 79 out of 95 tasks, meaning an enterprise would achieve better business outcomes by doing literally nothing. The model repeatedly spam-queried stakeholders, attempted incorrect database migrations, or introduced breaking modifications that triggered catastrophic incident zeroes.

• 4. Ensembles and Compute Scaling Offer Diminishing Returns: An oracle ensemble that selects the best-performing model run per task still only reaches a score of 0.70 with 30 resolved tasks. Furthermore, total token expenditure and reasoning chain length exhibited nearly zero statistical correlation with final delivery success.

---

Why FDE-Gym Resets the Agent Evaluation Standard

Until now, enterprise AI evaluation has suffered from goodharting: benchmarks tested models on sterile, single-turn pull requests or sandboxed algorithmic puzzles. But enterprise buyers do not pay for code tokens; they pay for business metric uplift and operational stability.

FDE-Gym highlights that creating an autonomous "AI engineer" requires solving three orthogonal challenges:

1. Conversational Epistemics: Knowing what you don't know, and asking the minimum sufficient set of questions to clarify ambiguity without exhausting human stakeholders.
2. Defensive Modification: Safely modifying live production databases and microservices without breaking underlying business contracts.
3. Metric-Driven Verification: Validating code against downstream business uplift distributions rather than simplistic unit test suites.

Until frontier models learn to navigate these unstated organizational dynamics, the vision of fully autonomous AI delivery teams will remain confined to the demo stage.