September 18 was a day dominated by LLM infrastructure news. AWS announced the Amazon SageMaker HyperPod Inference Gateway, which routes inference requests by inspecting the internal state of GPUs, touting up to an 82% reduction in first-token latency. In China, details emerged about "Naive AI," a stealth LLM startup that reached a $1.4 billion valuation just seven months after its founding, while a $2 billion push into small-scale batteries got underway to meet AI data center power demand. And in the local LLM community, a DIY rig with 768GB of VRAM is making waves.
AWS's GPU-Aware Inference Gateway — Up to 82% Lower First-Token Latency
According to [AWS's announcement](https://aws.amazon.com/blogs/machine-learning/introducing-amazon-sagemaker-hyperpod-inference-gateway/), the SageMaker HyperPod Inference Gateway is a Kubernetes-native, GPU-aware routing system that can be deployed as a managed add-on to HyperPod on Amazon EKS. It leverages real-time GPU signals to deliver each inference request to the optimal pod, cutting first-token latency by up to 82% without any changes to either model servers or client applications.
The underlying problem is that standard Kubernetes load balancers cannot see the internal state of GPUs. Round-robin and least-connections algorithms have no visibility into which pod's KV cache is nearing saturation, which pod is midway through generating a long context, or which pod already has the LoRA adapter required by a request resident in memory. As a result, requests pile up behind busy pods while available GPU capacity sits idle. Under concentrated traffic, first tokens can take more than four seconds, which in turn drives teams toward over-provisioning with generous headroom.
The gateway takes a two-tier architecture. The first tier is an EKS add-on that runs in each cluster and consists of three components: Envoy Gateway (an L7 proxy), the Body-Based Router (which reads the model field of OpenAI-compatible requests and routes them to model pools), and the Endpoint Picker (which scores pods on weighted criteria derived from their Prometheus metrics). The Endpoint Picker's evaluation axes are KV cache utilization, queue backlog, LoRA adapter residency, prompt-prefix cache hit rate, and the number of in-flight requests, with weights adjustable through configuration. AWS cites an example where a chatbot user who had waited 4.4 seconds for the first token now receives it in under 800 milliseconds.
In benchmarks, four models ranging from 8B to 235B parameters were run on H100 (p5.48xlarge) and A10G (g5) instances and compared against a Kubernetes round-robin baseline. On fleets with mixed GPU generations, P95 first-token latency fell by 97–98%; under bursty traffic, by 94–98%; and throughput improved by up to 50% under certain conditions. On homogeneous hardware with steady traffic, however, results were on par with round-robin. The structure is such that "the further a fleet drifts from homogeneity, the greater the gains" — a design that pays off precisely under the conditions real production traffic creates: mixed hardware, sudden demand spikes, and shared prompt prefixes.
Deployment is complete once you install the add-on and apply the InferenceGatewayConfig custom resource, and the endpoint is OpenAI-compatible. It also supports co-located routing of multiple models and routing that favors pods where the needed LoRA adapters are already resident. A second tier, the "Global Inference Router" — which will handle failover across clusters and regions — is planned for future release.
China's Stealth Startup "Naive AI" — $1.4B Valuation in 7 Months, First Model to Be Open-Weight
[A report from The Information](https://www.theinformation.com/articles/tsinghua-professors-stealth-llm-startup-hits-1-4-billion-valuation) revealed that Chinese AI startup Naive AI has raised a total of $400 million across three funding rounds, reaching a valuation of over $1.4 billion ($1.42 billion post-money). Investors include Tencent, IDG Capital, Matrix Partners China, and Sequoia China. The company was founded in February of this year by Tsinghua University professor Dai Jifeng, has fewer than 100 employees, and has operated quietly for more than seven months.
Its technical strategy is distinctive as well. Rather than taking the high-cost route of pre-training from scratch, the company is said to modify the architecture of existing Chinese open-weight models and then layer on mid-training, post-training, and reinforcement learning to push performance further. Which base models it actually uses has not been confirmed. On top of that, the company is reportedly researching "recursive self-improvement," in which models iteratively refine themselves.
The company plans to release its first large language model, "Naive," as an open-weight model as early as this month. How far a zero-scratch pre-training approach can go remains an open question, but a new type of player may be about to join the competition among Chinese open-weight models.
$2 Billion in Batteries for AI Power Demand — EQT-Backed Push to Accelerate Data Center Connections
With the rapid growth of AI data centers making grid connection queues a mounting problem, it was [reported](https://www.bloomberg.com/news/articles/2026-09-18/eqt-is-backing-2-billion-small-battery-push-for-us-data-centers) that EQT Infrastructure is backing a $2 billion push into small-scale battery storage. Madison Energy Infrastructure, the distributed power development company under its ownership, plans to add one gigawatt of capacity by 2028, starting with PJM Interconnection, the largest power grid in the United States (covering 13 states).
The batteries to be installed offer four hours of discharge and will be deployed at industrial and commercial facilities, including wastewater treatment plants. The strategy is two-pronged: placing batteries on facility sites lowers power costs for industrial consumers while also enabling AI data centers to connect to the grid sooner. Madison CEO Richard Walsh discussed the plans in an interview.
Generative AI's power demand is growing faster than new power plants can be built, and the idea of building a distributed-storage "bridge" rather than waiting for major transmission upgrades looks set to become a factor that shapes where future data centers are sited.
768GB of VRAM for the Price of One RTX 6000 Pro — A New Sweet Spot for DIY Local LLM Builds
On Reddit's LocalLLaMA community, a [DIY rig that delivers 768GB of VRAM from twelve CMP170hx cards (64GB each)](https://www.reddit.com/r/LocalLLaMA/comments/1wjr59t/768gb_vram_for_less_than_the_price_of_one_rtx_6000/) is drawing attention. According to the poster, the total cost came in below that of a single RTX 6000 Pro. The CMP170hx is a cheap secondhand card that circulated during the mining boom, and for LLM workloads its cost per GB of VRAM is extraordinary. The rig is also connected to a separate rig over RPC via optical fiber, so memory can be expanded as needed.
Models running on the rig include GLM5.3, DeepSeek v4.1 Flash, Qwen3.8 Flash, Qwen3.8-2.4T, Kimi K3, and MiniMax M3. The poster claims "performance that a single RTX 6000 or an M3 Mac Studio cannot reach," with inference available on both vLLM and llama.cpp. As for the classic counterarguments — electricity bills, noise, APIs being cheaper — the poster's stance is that they are already factored in.
In DIY local LLM territory, setups like multiple RTX 3090s for 64GB of VRAM have long been the go-to. With huge models out of China being released one after another, budget-first builds that "stack VRAM to match the size of the model" have gained yet another round of new options.
From the Japanese Community — A "Cost-Benefit Ledger" for Official Claude Code Plugins on Qiita
Among developers in Japan, Qiita saw the publication of a [hands-on write-up of installing four plugins from Claude Code's official marketplace and removing two of them the same day](https://qiita.com/sumitsuke/items/67c7ea172584a90f72dc). The four plugins installed were pyright-lsp, typescript-lsp, security-guidance (v2.0.8), and hookify. Written as a "cost-benefit ledger for linters," it records what each tool caught and what it cost.
Claude Code can be extended with plugins and skills, but what to install is left to the judgment of each team. Even for officially distributed plugins, this kind of hands-on measurement — logging payback from both detection and cost perspectives — should offer a useful reference for other teams weighing adoption.