MemT-1.0 Internalizes Long-Term Memory to Slash KV Storage by 66x at 128K Tokens

Long-context language models and multi-agent frameworks are colliding with a fundamental infrastructure wall: serving extended conversations by re-streaming bloated KV caches or orchestrating brittle external RAG pipelines incurs prohibitive VRAM consumption, spiraling time-to-first-token (TTFT) latency, and staggering hosting costs.

MemTensor has introduced MemT-1.0 (arXiv:2610.26825), an open-weight foundation model family that abandons external context-stuffing plumbing in favor of an intrinsic, memory-native architecture. By augmenting transformer layers with parallel linear recurrence blocks driven by Gated DeltaNet-2 (GDN-2) and scaling capacity via a novel Mixture-of-Storage (MoS) mechanism, MemT-1.0 internalizes persistent, evolving state directly into the neural network's forward compute graph.

The operational gains are stark: at a history length of 128K tokens, MemT-1.0-Nano achieves a 66× reduction in persistent storage overhead and a 1.86× end-to-end inference speedup compared to standard full-context KV caching, while cutting TTFT by up to 4.19× at 256K tokens.

---

The Architecture: Why External Context Plugs Fail

Modern agentic workflows treat memory as an external systems problem. Every time an agent takes a step in a multi-turn environment, an orchestration framework queries a vector database, formats chunks into markdown, and concatenates past execution traces into the prompt window.

This approach imposes three major bottlenecks:

• Quadratic & Linear VRAM Bloat: Carrying KV states for tens of thousands of past interaction tokens quickly overwhelms high-bandwidth GPU memory.
• TTFT Spikes: Prefilling long context transcripts at every turn introduces massive compute overhead before generating a single new token.
• Information Degradation: External retrieval algorithms often discard critical intermediate agent reasoning steps and chronological nuances.

MemT-1.0 addresses these failure modes by transitioning memory from an input payload to a native computational state. Selected Transformer layers are augmented with parallel Native Memory Blocks. The model decouples memory operations into two complementary systems:

1. Dynamic Local Memory: A recurrent state structure that updates token-by-token within active trajectories to track immediate interaction context.
2. Trainable Hyper Memory: Higher-order parametric memory components that govern how long-term knowledge and cross-session procedural patterns are integrated.

[ Input Tokens ] ────────┬─────────────────────────────┐
                         ▼                             ▼
             [ Transformer Attention ]      [ Native Memory Block (GDN-2) ]
                         │                             │
                         │                   [ Mixture-of-Storage ]
                         │                             │
                         └──────────────┬──────────────┘
                                        ▼
                            [ Unified Output State ]

---

Gated DeltaNet-2 and Mixture-of-Storage

Previous native-memory prototypes struggled with numerical instability during online updates or relied on discrete token-dropping heuristics that corrupted long-term semantic context. MemT-1.0 stabilizes token-level evolution by implementing Gated DeltaNet-2 (GDN-2).

GDN-2 decouples three fundamental memory operations into explicit linear attention gates:

• Erase Gate: Explicitly decays or zeroes out superseded facts or stale scratchpads.
• Write Gate: Selectively commits novel, high-value tokens into the recurrent memory matrix.
• Fade Gate: Regulates gradual contextual forgetting over extended interaction steps.

To prevent the memory parameters from exploding the model's active compute footprint, the researchers introduced Mixture-of-Storage (MoS). Analogous to Mixture-of-Experts (MoE) in feed-forward layers, MoS distributes memory states across dynamically routed memory slots. This allows the system to massively expand native memory capacity without linearly scaling the parameter footprint or memory bandwidth overhead per head.

The Open Model Family

MemTensor released two core model variants:

• MemT-1.0-Nano: 36B total parameters (3B active per token) with 10M dynamic memory capacity, built upon a Qwen3.6-35B-A3B base.
• MemT-1.0-Flash: 288B total parameters (17B active per token) with 336M dynamic memory capacity, built upon a DeepSeek-V4-Flash backbone. MemT-1.0-Flash adds only 3.61B trainable memory parameters to its 288B foundation—a mere 1.25% parameter overhead.

---

Two-Stage Training on 34B Tokens

Transforming a frozen pretrained backbone into a memory-native model requires carefully calibrating how memory reads and writes interact with core language modeling capabilities. The MemTensor team executed a two-stage training pipeline supported by a curated 34B-token dataset:

1. Continued Pre-Training (CPT): The model is trained on multi-turn dialogue histories, document archives, and procedural interaction sequences to establish fundamental stateful language modeling. Native memory modules learn stable recurrent updates while the backbone attention layers remain anchored.
2. Post-Training Memory Alignment & Distillation: Using knowledge distillation from frozen backbone teachers alongside task-specific supervision, the model learns downstream memory operations—such as multi-hop recall, implicit user profiling, and procedural skill execution—without suffering catastrophic forgetting.

Evaluations across standard capability benchmarks confirm that general reasoning remains uncompromised. On MMLU-Pro, GPQA, IFEval, and LongBench, both Nano and Flash variants perform within 1 to 2 percentage points of their base teacher backbones, demonstrating that memory internalization does not degrade zero-shot reasoning.

---

Benchmark Performance: Speed, Footprint, and Agentic Success

MemT-1.0 was evaluated across 11 standard memory benchmarks, including MemBench, PersonaMem, LoCoMo, LongMemEval-S, MemoryAgentBench, and BEAM-1M.

Model Variant	Avg. Memory Benchmark Score	vs. Parametric Baselines	KV Storage Reduction (128K)	TTFT Speedup (128K / 256K)
MemT-1.0-Nano (36B/3B)	47.0	+8.9 pts	66×	— / —
MemT-1.0-Flash (288B/17B)	50.6	+12.5 pts	66×	2.44× / 4.19×

MemT-1.0-Flash outperformed the strongest parametric memory baselines by 12.5 points, operating purely on internal state representations without retrieving raw historical transcripts into the context window at inference time.

Procedural Memory on SWE-bench

The most compelling capability for agent developers appears in software engineering benchmarks. When retrieved procedural traces and repo structures were ingested directly into MemT-1.0's dynamic memory before task execution, the model achieved:

• +7.6 points improvement on SWE-bench Verified over the baseline backbone.
• +8.3 points improvement on SWE-bench Multilingual.

Crucially, once internalized into memory, the agent executed code generation steps with the run-to-run consistency and speed of a zero-context prompt, eliminating the token overhead of carrying historical documentation in every agent thought loop.

---

Serving Infrastructure & Production Reality

MemTensor released the model weights under an MIT license alongside implementation pathways for vLLM and SGLang.

For production deployments, MemT-1.0 shifts the serving paradigm:

• Constant Persistent State: Rather than scaling memory storage linearly with interaction length $O(N)$, the session state size remains completely constant once initialized.
• Distributed Memory Isolation: Inference engines maintain shared frozen model weights across instances while routing isolated, lightweight dynamic memory vectors per user session or agent trajectory.
• The Textual Retrieval Tradeoff: The authors note that while MemT-1.0 excels at behavioral consistency, procedural knowledge, and contextual synthesis, traditional RAG remains preferable for tasks requiring exact verbatim document citations (e.g., legal or compliance audits).

By moving memory from an external context-window hack into native neural state dynamics, MemT-1.0 offers open-source developers a scalable blueprint for building persistent, long-running agents that learn continuously without collapsing under the weight of their own past.