The Reality of Long-Context on Local LLMs: What a Thorough 13-Model Comparison Revealed About the "Real Bottleneck"

An exceptionally detailed local LLM benchmark result was posted in the LocalLLaMA community on Reddit. It compares 13 models (Dense, MoE, Mamba2 hybrid, MLA MoE) across five context lengths ranging from 512 tokens to 131K tokens. The test environment was an RX 7900 XT 20GB (Vulkan backend) running llama.cpp build 9860, and the whole run took 21 hours.

Finding 1: Prefill speed is everything — tg128 is nearly irrelevant

For agent workloads (tool use, coding, RAG), when generating a short output (300 tokens) over a 65K context, **prefill (prompt processing) accounts for 94–99% of the total time**. In other words, tg128 (token generation speed), the commonly cited metric, is nearly irrelevant for agent use cases. What matters is pp65K/pp131K (prefill speed at 65K/131K contexts).

Finding 2: KV head count matters more than parameter count

The most influential architectural factor turned out to be neither the parameter count nor the MoE/Dense distinction, but the **number of KV heads**. Models with 4 KV heads (×128 dim = 64 KB/token) were up to 4.4× faster at 128K context than those with 8 (160 KB/token), and this gap holds even within the same Dense size class.

Finding 3: The Mamba2 hybrid exceeds expectations

Granite-4.0-H-Small (IBM, 4 attention layers + 36 Mamba2 layers) retained 69% of its pp4K speed even at 131K context — notable when every transformer model dropped below 42%. Mamba2 layers use a fixed recurrent state, which keeps the KV cache from growing, so long-context scaling stays flat. While reasoning quality still has issues today, the architecture is very promising.

Finding 4: An F16 KV cache can be faster than a quantized one

Contrary to the conventional wisdom that "you should quantize the KV cache," cases were observed where an F16 KV cache was 20–53% faster than Q8/Q4. The effect was especially pronounced in MoE models and smaller Dense models. This is because the computational cost of dequantization grows in proportion to context length.

Practical takeaways

Benchmark with pp65K/pp131K instead of tg128

When choosing a model, check `n_kv_heads` and `head_dim` before the parameter count

MoE models offer the best balance for agent workloads on consumer VRAM

It's worth trying an F16 KV cache at your actual context length

---

Hugging Face and Cerebras Announce speech-to-speech — a Voice Conversational AI That Runs with a Single Command

On July 1, Hugging Face and Cerebras released a "speech-to-speech" tool. It is designed to shrink the pause between you speaking and the response arriving as much as possible, and it can be set up in a local environment with a single command.

Given that a few hundred milliseconds of latency greatly shapes how conversational AI feels, this is a noteworthy approach to achieving low-latency voice interaction. See the original article for detailed setup instructions and performance verification.

---

Hooking DeepSeek V4 Pro Directly into Codex Failed — Even Though It Works with Claude Code

An attempt to connect DeepSeek V4 Pro directly to Codex (OpenAI's coding agent) has been reported. Although it works fine with Claude Code, the conclusion was that it can't be recommended for Codex.

Regarding this "seemingly contradictory behavior," it has been suggested that compatibility between each agent environment and the model, as well as differences in tool-use protocols, may be playing a role. It's a useful case study when evaluating tool choices and model pairings in real-world development.

---

AI Industry Weekly: Karpathy Joins Anthropic, and More (dera AI Weekly Vol.32)

Vol.32 of "dera AI Weekly," a weekly AI news roundup, has been published on Qiita. The May 25, 2026 issue covers the following topics:

Andrej Karpathy, OpenAI co-founder, moves to Anthropic

New announcements at Google I/O

The evolution of agent technology and accelerating talent movement

As competition in the AI industry intensifies, this roundup is a helpful way to track the movement of top talent and the direction of agent technology.

---