ActionOPD: Why LLM Agent Distillation Should Target Decisions, Not Tokens

Traditional knowledge distillation forces student language models to mimic the exact token sequences and logits of their teachers. In static text generation, this works reasonably well. But in multi-turn agentic environments—where models invoke tools, query APIs, and navigate environments—token-level matching creates severe architectural bottlenecks. An agent can formulate three completely different chain-of-thought rationales and phrasing variations that execute the exact same tool call with the exact same parameters.

Forcing a 4B student model to mimic the exact verbose token distribution of an API-gated frontier model wastes capacity and locks distillation into identical tokenizer vocabularies.

ActionOPD, developed by researchers at Tsinghua University, fundamentally reframes agent distillation. Instead of matching token probabilities along teacher-generated trajectories, ActionOPD maximizes the expected likelihood of the teacher's semantic action over states sampled directly from the student's own policy rollouts. This allows smaller models to discover their own internal reasoning paths while learning to execute the correct decisions.

---

The Triple Dilemma of Agent Distillation

Training compact, high-performing agents traditionally forces researchers to pick between three flawed paradigms:

• Offline Imitation (Standard KD): The student is trained on static traces recorded by the teacher. When the student makes a minor deviation at test time, it enters an unseen state and cascades into compounding errors due to covariate shift (d_T ≠ d_S).
• Token-Level On-Policy Distillation (Token-OPD): The student rolls out actions in the real environment (s ~ d_S), and the teacher evaluates the student's token sequence. However, this requires access to the teacher's white-box token logits and demands that both models share an identical tokenizer. It also penalizes students whenever their reasoning phrasing differs from the teacher's, even if the final tool call is identical.
• Outcome-Based Reinforcement Learning (RL): The agent receives sparse terminal rewards (+1 on task success, 0 on failure). While this prevents distribution shift and tokenizer lock-in, multi-turn trajectories with dozens of tool calls suffer from severe credit assignment sparsity and slow sample efficiency.

ActionOPD resolves this trilemma by operating strictly at the semantic action boundary.

---

Formulating Action-Level On-Policy Policy Gradients

In ActionOPD, an agent's generation y produced at state s is mapped to an action space via a semantic parser ϕ_s(y) = a. Here, a represents the structured action: the specific tool name, its structured arguments, or the explicit conversational intent.

The student's probability mass for generating a specific teacher action a^T is the sum of probabilities of all possible text outputs that evaluate to a^T:

π_θ(a^T | s) = ∑_y π_θ(y | s) · 1{ϕ_s(y) = a^T}

Instead of minimizing Kullback-Leibler (KL) divergence over individual token logits, ActionOPD maximizes the expected alignment with the teacher's sampled actions a^T ~ π_T(· | s) over the student's on-policy state visitation distribution d_S:

J_ActionOPD(θ) = E_{s ~ d_S, y ~ π_θ(· | s)} [ r_T(s, ϕ_s(y)) ]

where the teacher reward r_T(s, a) is defined as the expected agreement indicator E_{a^T ~ π_T(· | s)} [ 1{a = a^T} ].

Differentiating this objective yields a standard REINFORCE-style policy gradient:

∇_θ J_ActionOPD(θ) = E_{s, a^T, y} [ 1{ϕ_s(y) = a^T} · ∑_{ℓ=1}^{|y|} ∇_θ log π_θ(y_ℓ | s, y_{<ℓ}) ]

Why This Derivative Matters

In standard policy gradient RL, an agent only updates its policy when an external environment verifier scores the final output. In ActionOPD:

1. At any state s reached during a student rollout, a black-box teacher (queried via standard API) is prompted with state s and samples candidate actions a^T.
2. The student generates multiple candidate responses y under its own current policy.
3. Every student candidate whose parsed action ϕ_s(y) semantically matches the teacher action a^T receives a positive advantage.
4. Crucially, the gradient backpropagates through the student's own tokens, reinforcing the student's idiosyncratic reasoning style rather than forcing it to mimic the teacher's syntax.

---

Breaking the Tokenizer and Model Family Barrier

Because ActionOPD treats the teacher strictly as an action oracle, it completely decouples the student's architecture from the teacher's.

• Cross-Model Distillation: You can distill a proprietary or non-compatible frontier model (e.g., DeepSeek-V4-Flash or proprietary closed APIs) directly into an open-weight edge model (e.g., Qwen3-4B or Llama-3-8B).
• Zero Tokenizer Alignment: Prior on-policy cross-tokenizer approaches required expensive byte-prefix marginalization or embedding projections. ActionOPD requires neither: only the semantic action strings (JSON payloads, API arguments) are compared.
• Autonomous Reasoning Compression: Smaller models cannot match the 32K-token latent reasoning depth of 100B+ models without severe degradation. ActionOPD allows the 4B student to formulate concise 500-token scratchpads that still hit the target API call.

---

Experimental Results: Scaling Cross-Benchmark Transfer

The Tsinghua team evaluated ActionOPD by distilling two distinct frontier teachers (DeepSeek-V4-Flash and Qwen3.7-Flash) into compact Qwen3-4B and Qwen3-8B student models across four challenging multi-turn benchmarks, including τ²-bench (Tau-2).

Key Performance Highlights

• Substantial OOD Generalization: Distilling Qwen3-4B using DeepSeek-V4-Flash via ActionOPD increased out-of-distribution tool-use performance by +6.9 percentage points over the base model, jumping from 30.8% to 37.7% average success rate.
• Teacher Capability Scaling: Students distilled from DeepSeek-V4-Flash consistently outperformed those distilled from Qwen3.7-Flash across all benchmark categories, confirming that ActionOPD successfully scales with teacher capability even when the teacher operates under an entirely different tokenizer and model family.
• Efficiency on τ²-bench: On complex dual-control conversational agent environments, ActionOPD matched or exceeded token-level OPD.

Model Configuration	Action Teacher	Out-of-Distribution Avg (%)	Delta vs Base
Qwen3-4B (Base Student)	—	30.8%	—
Qwen3-4B + ActionOPD	Qwen3.7-Flash	35.9%	+5.1%
Qwen3-4B + ActionOPD	DeepSeek-V4-Flash	37.7%	+6.9%
Qwen3-8B (Base Student)	—	35.6%	—
Qwen3-8B + ActionOPD	Qwen3.7-Flash	39.0%	+3.3%
Qwen3-8B + ActionOPD	DeepSeek-V4-Flash	40.0%	+4.4%

---

Self-ActionOPD: Privileged Self-Distillation

Beyond cross-model distillation, the authors introduced Self-ActionOPD, where the same model acts as both student and teacher.

During training, the teacher instance is given privileged context—such as ground-truth database schemas, hidden simulator variables, or unmasked environment state. The student instance operates under strict runtime observation constraints. By aligning the student's actions with the privileged teacher's decisions on student-generated rollouts, Self-ActionOPD enables models to self-improve on long-horizon reasoning tasks without human labels or larger external models.

---

The Broader Shift in Agent Alignment

For the past two years, LLM post-training has treated text generation and agent actions as the same mathematical object: a sequence of tokens under a vocabulary V. ActionOPD reinforces an increasingly critical realization in agentic AI:

Tokens are merely a transport protocol; actions are the semantic unit of intelligence.

By optimizing policy gradients against semantic decisions rather than linguistic realization, ActionOPD provides a clean, robust blueprint for compressing frontier agent capabilities into fast, lightweight models capable of running locally on phones, PCs, and edge robotics.