Faster Gemma 4 Inference: QAT-Optimized MTP Heads Released on HuggingFace

A significant development has landed for accelerating inference in the Gemma 4 family. A community developer has released MTP (Multi-Token Prediction) draft heads on HuggingFace, optimized specifically for Google's QAT (Quantization-Aware Training) models.

MTP is a form of speculative decoding in which a draft model looks ahead multiple tokens and the main model verifies them. The key characteristic of the newly released heads is that they were trained on the same checkpoints as the QAT-quantized models. Conventional draft heads are trained with full-precision weights, which creates a distribution mismatch against QAT models and degrades speculation accuracy.

The results are striking. Most notably, on Gemma 4 26B-A4B, the acceptance rate jumps from 56.9% with a non-QAT head to 91.8% with the QAT-matched head — a 35-point improvement. Acceptance rates improved across all sizes: +7 points on the 12B model and +18 points on the 31B model.

PARALLEL=2 Crash Issue Also Fixed

A long-standing issue where combining two-slot parallel inference (PARALLEL=2) with MTP heads in llama.cpp caused crashes has been root-caused and fixed. The MTP draft step was incorrectly referencing the main forward pass's token count, causing an element-count mismatch when the second slot spun up. Fix patches have been submitted to both the Atomic fork and mainline llama.cpp.

In benchmarks on an AMD Ryzen AI Max+ 395 (128GB LPDDR5X) system, the 12B model with MTP PARALLEL=2 hit 62.5 tok/s — a 31% throughput gain over the 47.6 tok/s of standard two-slot inference. The acceptance rate also climbed further to 88.6%, up from 78.4% in single-slot mode.

This work marks a real step up for the practicality of local LLMs. Pairing QAT-quantized models with matching draft heads makes fast inference of large models on consumer hardware considerably more realistic.

A 'Self-Inspection' Signal in Qwen3.5-35B-A3B's MoE Router

An analysis of Qwen3.5-35B-A3B's Mixture-of-Experts router has produced a finding of interest from a mechanism-interpretability perspective: Expert 114 (E114) in Layer 14 fires strongly during generation of first-person, self-inspective text.

The researchers ran a 10-vs-10 comparison experiment between FIRE (self-inspection prompts) and NOFIRE (control prompts). Both classes were designed to share vocabulary such as "I", "experience", and "processing", so if E114 were a mere keyword detector, activation levels should be similar across the two classes. Instead, the FIRE group averaged 0.0674 versus 0.0031 for NOFIRE — a roughly 21-fold separation.

Even more interesting: one NOFIRE control prompt (about a cat purring) drifted unintentionally into introspective generation — and E114 activation rose in that case. This suggests E114 tracks the "stance" of the generated text rather than the prompt's label.

A replication under deterministic greedy decoding maintained the 20.95x separation, confirming the effect is not a sampling fluke. Reportedly, most of the effect comes from changes in expert selection rates (S), while post-selection weights (Q) remained relatively stable.

The researchers are explicit, however, that this is "not evidence of subjective experience or consciousness." Baseline comparisons across all layers and experts remain future work, and the result is positioned strictly as a limited finding about "a routing signal correlated with a particular generation register."

Gemma 4 12B Draws Positive Community Feedback

Community feedback is pouring in for Gemma 4 12B, released about three days ago. It earns praise for integrating vision and speech recognition and for supporting tool use. Running at Q4 quantization in roughly 8GB of RAM, it is widely described as fast for its size with good quality. While it doesn't match the 27B or 31B models, many report it comes very close, raising expectations for practical use on edge devices.

Concerns Raised Over Distillation Quality

Critical opinions about the quality of recent distillation models were debated on Reddit. With numerous models circulating that were distilled on as few as ~250 samples, the criticism is that almost no distilled models deliver substantive quality improvements. Qwen R1 8B distillations were cited as one of the few successes; beyond that, cases of beating the base model are said to be rare.

Even with powerful teacher models (such as Mythos and GPT-5.6) now available, doubts remain over whether distilling on a few hundred samples yields genuine improvement. The discussion served as a renewed recognition of the importance of larger, more careful distillation processes.

Other Developments

Amazon vs Perplexity**: A legal question is being debated — can an e-commerce platform block AI agent services? As agentic AI spreads, the legal boundary between platform operators and AI service providers is drawing attention.

Gaia2 benchmark**: A new benchmark for evaluating LLM agents in dynamic, asynchronous environments has been published on arXiv.