3.14 Million Papers, 16TB — The Complete arXiv Corpus Arrives on Hugging Face

A dataset called "[secemp9/arxiv-complete](https://huggingface.co/datasets/secemp9/arxiv-complete)" has been published on Hugging Face. It's a complete corpus bundling arXiv metadata, version history, submission files, and rendered documents, covering 3,148,796 papers and totaling 16.08TB. It spans preprints from 1986 through the end of August 2026.

The dataset is organized into nine configs, ranging from per-paper metadata (1.6GB) to resolved TeX full text (paper_text, 70GB), complete submission packages (source, 6.51TB), and a PDF mirror (8.65TB). PDFs are retained for 99.54% of versions with reported sizes, and 99.47% of papers have a PDF available. paper_text alone is estimated at 78–81 billion tokens, and you can try it out easily starting from the 26MB sample config.

That said, it isn't usable as a training corpus as-is. paper_text still contains LaTeX syntax, comments, and macros, along with withdrawal stubs and mistakenly selected templates. Licenses also vary from paper to paper, and the CC0 designation covering the compilation as a whole does not extend to the individual papers. The authors disclose these limitations in detail on the dataset card, making it a useful model of transparency for large-scale datasets.

Until now, arXiv datasets have mostly been partial subsets covering only metadata or TeX. This one stands out for packaging entire submission packages in a verifiable form with SHA-256 hashes, and it looks set to gain traction as a foundation for LLM training data and literature research.

Qwen 3.8 Flash Next Reaches Usable Local Speeds — FreeToken and halogen 0.12.0

In the local LLM community, two reports came out in quick succession of running Qwen3.8-Flash-Next at usable speeds on a single machine.

The first is [FreeToken](https://github.com/FlashML-org/FreeToken). On a DIY machine with a single RTX 5090 plus 128GB of DDR5, expert caching reportedly achieves around 50 t/s (40–60) for token generation and roughly 2,300 t/s for prompt processing. llama.cpp doesn't yet implement expert caching for MoE models, so this attempt to fill that gap is drawing attention. The model used is NVIDIA's NVFP4-quantized version, and the approach is said to work with other MoE models as well. It's still early days — KV quantization and MTP aren't supported yet.

The second is halogen 0.12.0 for Strix Halo (Ryzen AI Max+ 395, 128GB). Following feedback that performance degraded at deeper contexts, fixes were applied: at roughly one million tokens of context, decode improved from 27.3 tok/s to 38.3 tok/s, and prefill was shortened from 21.2 minutes to 17.9 minutes. On a second pass with the prompt cache warm, the first token reportedly comes back in 0.55 seconds.

After a steady stream of fast-inference reports from the Apple Silicon camp, Windows/Linux DIY machines and mini PCs are now reaching practical territory too. The options for running million-token-class contexts locally keep growing.

Jev and Local Models Go Head-to-Head in Doom

The AI agent "Jev" and small local models were compared directly in the FPS game Doom (ViZDoom). The input is a short text description generated from the game screen (labels for visible objects, bounding boxes, HP and ammo values), and the output is a single button action such as "left," "right," or "shoot." The local models ran on a single DGX Spark (NVIDIA GB10, 128GB unified memory).

For average kills in Defend the Center, Jev 1.13 scored 5.63, a LoRA-finetuned Qwen3.5-4B scored 3.63, and Laya and the fine-tuned ModernCE-base-nli each scored 1.25. Jev also topped Health Gathering with an average survival time of 13.03 seconds, though the margin was small. On latency, ModernCE posted a p50 of 7.6ms versus 117.3ms for Jev (including the API round trip). None of the models received any Doom-specific training.

Even with a text-only interface, it's interesting that a 4B-class local model can hold its own this well. At the same time, Jev's advantage is unmistakable — the gap between a purpose-built agent model and general-purpose LLMs showed up in hard numbers.

Designing Screens That Agents Can Operate — The Affora Research

Research called "Affora," which probes UI design for the era of "agents" — AI that operates apps and websites on people's behalf — has been released. According to a Japanese write-up, whether an agent can navigate a screen well is determined not by how it looks but by "how it's built underneath," and designs that make semantics explicit can improve operation success rates from 43% to as high as 90%.

For developers working on agent readiness, this means explicit semantics pay off separately from human-facing visual design. In parallel with advancing model performance, we've entered a stage where the design of the things being operated on is being reexamined as well.

AI Adoption at Japan's SMEs: The Reality of "Leaving the Last Step to People"

Turning to Japan, ITmedia AI+ ran a lineup of articles on AI adoption at small and mid-sized companies. Simply deploying generative AI doesn't lead to business improvement if frontline teams can't put it to use, and amid barriers like AI talent shortages, training, and cost, new support services are emerging — from engaging experts "only when needed" to full-service support covering everything from implementation through business transformation.

Meanwhile, Abeam and Notion are focusing on building "corporate knowledge" — putting the knowledge scattered across a company and individuals' rules of thumb into a form AI can use. Generative Partners has launched a service that splits the "last-mile human work" — exception handling and visual checks that remain even after RPA and generative AI are in place — between AI and human effort, taking it on entirely as BPO.

With tool deployment underway, the next points of debate are organizing knowledge and redesigning people's roles — a lineup that lays bare the realities on the ground at Japanese workplaces.