MLExpert: How a 35B MoE Doubled Kaggle Medal Rates via Windowed Interaction RL

Autonomous machine learning engineering (MLE) is widely regarded as the cornerstone of recursive self-improvement (RSI), yet training an agent to function as an end-to-end ML engineer has remained bottlenecked by long-horizon credit assignment. When an AI agent spends thirty CLI commands configuring an environment, refactoring a data pipeline, and writing a PyTorch training loop, token-level reward models fail because no intermediate step is inherently "good" until the validation metric arrives—while end-of-trajectory rewards collapse under severe variance.

A newly released research paper introduces MLExpert (arXiv:2610.22344), a dedicated 35-billion-parameter Mixture-of-Experts (MoE) agentic model designed explicitly for autonomous, model-directed ML engineering. By restructuring post-training around Windowed Interaction Reinforcement Learning (WIRL) and hierarchical supervision, MLExpert nearly doubles the competitive medal rate of its base model on MLE-Bench Lite—from 37.9% to 72.7%—matching frontier-scale systems while running at a fraction of their parameter footprint.

---

The Credit Assignment Trap in Autonomous MLE

Existing attempts to automate competitive and applied ML engineering generally rely on rigid prompt scaffolding or brittle tree-search workflows layered over generalist frontier LLMs. When these systems switch to model-directed autonomy—where the model itself chooses when to write exploratory code, run an ablation, inspect stdout logs, or pivot to a new architecture—standard reinforcement learning breaks down:

• Dense intermediate rewards misguide search: Rewarding code syntax correctness or partial execution encourages agents to write safe, trivial scripts rather than bold, high-performing modeling iterations.
• Sparse trajectory rewards suffer from credit dilution: A full Kaggle competition run can involve dozens of separate experiments spread across hundreds of turns. Rewarding only the final test submission metric makes it mathematically impossible for policy gradients to isolate which specific hyperparameter tweak or feature engineering step drove the improvement.
• Execution latency stalls on-policy training: Real ML training loops take minutes or hours to run. Waiting for an entire trajectory to resolve before calculating policy updates renders large-scale RL post-training economically and computationally infeasible.

To break this deadlock, the researchers behind MLExpert targeted the interaction boundary between successive empirical outcomes.

---

Windowed Interaction RL (WIRL)

The core algorithmic contribution of MLExpert is Windowed Interaction RL (WIRL). Rather than treating a multi-hour MLE session as one monolithic trajectory or a series of disconnected single-token steps, WIRL segments the interaction history into discrete execution windows defined by consecutive experimental feedback.

[Hypothesis & Code Gen] ──► [Bash Execution] ──► [Log & Metric Output] 
└─────────────────────────── Execution Window k ──────────────────────────┘
                                    │
                                    ▼
                  Shared Outcome Reward Δ Metric (k)

1. Window-Level Credit Attribution: Every decision, tool invocation, and code edit between experiment setup and validation result is bound into a single execution window. The policy assigns a shared, outcome-based reward to all actions within that window based directly on the delta of validation performance ($\Delta \text{Loss}$, $\Delta \text{AUC}$, or benchmark metric).
2. Asynchronous Trajectory Collection: Because ML experiments exhibit non-deterministic execution times, WIRL implements an asynchronous buffer that computes policy corrections without blocking worker threads.
3. Off-Policy Gradient Updates: The trainer updates the agent's policy mid-run as soon as an execution window closes, allowing the model to refine its optimization heuristics without waiting for the full multi-turn benchmark attempt to conclude.

This architecture transforms autonomous MLE into a tractable MDP where the agent learns the direct causal link between its architectural hypotheses and real validation swings.

---

Kaggle-MLE Factory and Hierarchical Agentic SFT

Reinforcement learning in complex software environments requires strong initialization. MLExpert bootstraps its base 35B MoE architecture through a multi-tiered data engine:

• Kaggle-MLE Factory: A specialized simulation and execution harness that compiles diverse, executable machine learning challenges spanning tabular, vision, NLP, and multimodal domains with fully sandboxed runtime environments and ground-truth validation oracles.
• Hierarchical Agentic SFT: Rather than standard next-token code cloning, supervised fine-tuning is divided into high-level strategy planning and low-level CLI execution. High-level plans decompose raw problem descriptions into exploratory data analysis (EDA), baseline validation, iterative feature engineering, model exploration, and ensembling.
• Outcome-Guided Pruning: Trajectories used during the SFT phase are filtered so that only successful debugging episodes and positive metric deltas are preserved, ensuring the agent learns effective error recovery rather than looping on broken package dependencies.

---

Benchmark Performance: Leveling Up on MLE-Bench Lite

Evaluated on MLE-Bench Lite—a rigorous benchmark consisting of real-world Kaggle competition tasks requiring end-to-end data preparation, model training, and submission generation—MLExpert demonstrated massive post-training gains:

• Medal Rate Surge: MLExpert lifted the base model's average medal rate from 37.9% to 72.7%.
• Parity with Frontier Giants: At 35B parameters (with sparse active parameter routing), MLExpert's medal acquisition and submission accuracy matched or exceeded frontier generalist models such as Kimi-K3 and GLM-5.3.

Model Architecture	Parameter Class	MLE-Bench Lite Medal Rate	Optimization Style
Base 35B MoE Baseline	35B MoE	37.9%	Unspecialized Tool-Use
Frontier Generalists (e.g., Kimi-K3 / GLM-5.3)	Large Frontier	~70–73%	Prompted Agentic Harness
MLExpert (WIRL + SFT)	35B MoE	72.7%	Model-Directed WIRL

---

Generalization Beyond Kaggle: The AI4AI Flywheel

Crucially, the authors evaluated whether MLExpert’s learned optimization policy was merely overfitting to Kaggle heuristics or genuinely internalizing empirical ML engineering principles.

When transferred without additional fine-tuning to out-of-domain tasks, MLExpert demonstrated robust transfer:

• NatureBench Lite: Maintained high hypothesis-generation and data-fitting scores on scientific discovery benchmarks outside standard competitive ML.
• Agent Scaffold Optimization: The model successfully acted as a meta-optimizer, iteratively modifying the Python orchestration code of other LLM agents to improve their downstream success rates.
• Neural Architecture & Algorithm Tuning: Demonstrated the ability to autonomously refine custom training loops, optimize learning rate schedules, and modify optimizer implementations to beat baseline training efficiency on novel datasets.

---

What This Means for Engineering Loops

The arrival of MLExpert, alongside concurrent work like Tsinghua's AutoSFT (which automates SFT dataset curation across agentic sub-loops) and Terminal-RSI (which iterates on terminal policy refinement), marks a clear inflection point for AI4AI.

For the past three years, AI code generation has operated primarily as open-loop completion—synthesizing code that compiles and hoping the logic holds. MLExpert proves that structuring RL around execution windows and empirical metrics produces closed-loop agents capable of running their own research programs. When a 35B open-weights model can reliably organize experiments, prune dead ends, and deliver competitive machine learning solutions autonomously, the barrier to automated engineering in production just collapsed.