Intel Optane Persistent Memory Enables Local Inference of a 1-Trillion-Parameter Model

A build report has been drawing attention on Reddit's LocalLLaMA community: a DIY PC built around Intel Optane Persistent Memory (PMem) running Kimi K2.5 (1 trillion parameters) locally at roughly 4 tokens per second.

The key to the configuration is the use of "Memory Mode," in which six discontinued Intel Optane DCPMM modules (768GB total) are installed with 192GB of DRAM acting as a cache. Despite a modest setup with only an RTX 3060 12GB GPU, it takes advantage of llama.cpp's hybrid GPU/CPU inference and the override-tensor flag: the attention weights of Kimi K2.5's MoE architecture are placed on the GPU, while the sparse expert weights are processed from PMem/DRAM.

Thanks to the nature of MoE architectures—not all parameters are needed at all times—inference on such limited hardware becomes possible. The builder commented, "It's a shame Intel discontinued Optane PMem. It should have paired well with the current direction of SSD offloading and memory-tiering approaches."

A GPU Compiler for LLMs Built from Scratch in 5,000 Lines of Python

"An LLM compiler stack built from scratch in 5,000 lines of Python" has been making waves on Reddit's MachineLearning community.

Against the backdrop of TVM being built with more than 500,000 lines of C++ and PyTorch stacking Dynamo, Inductor, and Triton, the author built a minimal compiler that converts small-scale models such as TinyLlama and Qwen2.5-7B into CUDA kernels through six IR stages.

In benchmarks on an RTX 5090, the FP32 kernels achieved an average of 1.11x over PyTorch eager and 1.20x over torch.compile, with speedups of up to 4.7x reported on small reductions, SDPA, and KV projection.

Using the optimization process of an RMSNorm layer as an example, the design progressively automates the optimizations CUDA engineers normally perform by hand—staging inputs to smem, resolving bank conflicts, and improving occupancy—which also makes it an excellent learning resource for ML compilers.

Large-Scale Testing of LLM JSON Output Reliability Across 288 Models

An investigation that systematically tested outputs from 288 models on what breaks when you ask an LLM for JSON has been gaining traction on Hacker News.

It is known that when retrieving JSON via APIs, models can exhibit a variety of problems—schema violations, missing escapes, and broken nesting structures—and this testing comprehensively brings the reality to light.

Even models equipped with Structured Output features were confirmed to behave differently on edge cases, making the results a valuable reference when building production systems that depend on LLM JSON output.

Simon Willison Introduces a Technique for Running LLMs in the Shebang Line

Simon Willison published on his TIL (Today I Learned) blog a technique for specifying an LLM in a script's shebang line (`#!`).

By writing the `llm` CLI tool into the shebang line, script files written in natural language can be executed directly. This approach, which blurs the boundary between the prompt and executable code, opens up new possibilities for one-liner-style task automation.

Willison's TIL entries are always practical and immediately actionable, offering valuable insights for developers interested in making the most of LLM CLI tools.

AI vs Microservices — Considering an Architectural Turning Point

Michael Nygard's article "AI versus Microservices" examines the impact the rise of AI agents will have on microservices architecture.

Whereas microservices assumed "coordination among loosely coupled components," we are shifting toward a model in which AI agents "autonomously carry out tasks given instructions in natural language." The article offers sharp analysis of how this change will affect existing service boundary design principles, and how far the level of infrastructure abstraction will rise.

Nygard is widely known in the software architecture field as the author of "Release It!", and his perspective carries important implications for rethinking the very assumptions of design.

Running 500k Context at 21 Tokens/sec on 48GB VRAM

Also in the LocalLLaMA community, a report was posted claiming that 500k tokens of context can be processed at 21 tokens per second on 48GB VRAM (dual TITAN RTX).

The model used was Nemotron-3-Super-64B-A12B-Math-REAP-GGUF, a quantized model published on HuggingFace. Although originally a math-focused tune, it reportedly performs better than expected on agentic coding tasks as well.

Being able to process a 500k-token context at practical speeds on consumer GPUs is an achievement that greatly expands the possibilities of local LLMs.

Concerns That Agentic AI Gives Cybercriminals "Nation-State-Level" Capabilities

According to Defense One, concerns are spreading that agentic AI brings nation-state-level attack capabilities to cybercriminals.

While Pentagon leadership actively embraces agentic AI, the technology's double-edged nature—the same technology also grants sophisticated automated attack capabilities to malicious actors—has been pointed out.

Concrete debate has begun over concerns that agentic AI's autonomous planning and execution capabilities could put cyberattacks that once required nation-state-level resources within reach of individual-level actors.

---

**Sources:**

[Computer build using Intel Optane Persistent Memory - Can run 1 trillion parameter model at over 4 tokens/sec](https://www.reddit.com/r/LocalLLaMA/comments/1taeg8h/computer_build_using_intel_optane_persistent/) — Reddit LocalLLaMA

[A hackable compiler to generate efficient fused GPU kernels for AI models](https://www.reddit.com/r/MachineLearning/comments/1tag07l/a_hackable_compiler_to_generate_efficient_fused/) — Reddit MachineLearning

[What breaks when you ask an LLM for JSON (288 model outputs tested)](https://thecrosswalk.news/what-breaks-when-you-ask-an-llm-for-json/) — The Crosswalk

[Using LLM in the shebang line of a script](https://til.simonwillison.net/llms/llm-shebang) — Simon Willison

[AI versus Microservices](https://www.michaelnygard.com/blog/2026/05/ai-versus-microservices/) — Michael Nygard

[500k context on 48gb VRAM!! - 21tok/s (coding)](https://www.reddit.com/r/LocalLLaMA/comments/1tag1ks/500k_context_on_48gb_vram_21toks_coding/) — Reddit LocalLLaMA

[Agentic AI is giving cyber criminals nation-state-like powers](https://www.defenseone.com/threats/2026/05/pentagon-leaders-love-agentic-ai-its-giving-cyber-criminals-nation-state-powers/413379/) — Defense One