METR's Famous AI Progress Graph Flagged for "Numerous Serious Errors"

METR's "time horizon" graph, which visually depicts improvements in AI capabilities, has become one of the most widely cited charts in the AI industry over the past few months. But Nathan Witkin, a research writer at NYU Stern's Tech and Society Lab, has sharply criticized the graph for containing numerous serious methodological errors.

The main problems Witkin identified are as follows. First, some of the human baseline data was not based on actual measurements and was nothing more than the authors' guesses. Second, when METR did measure human task completion times, it paid benchmarkers by the hour, creating an incentive to drag out the work. Third, the sample of human benchmarkers skewed toward friends and acquaintances of METR employees and lacked representativeness.

It has also been pointed out that METR used data from humans unfamiliar with the tasks, even though people familiar with the specific codebases and tasks can complete them 5 to 18 times faster. There is a test data contamination issue as well: solutions to some tasks had been published online and were highly likely to be included in LLM training data.

Witkin concluded that "the METR graph is beyond saving" and warned that "relying on information that appears scientific on the surface but was not produced with the rigor science demands is extremely dangerous." Gary Marcus and Ernest Davis have also pointed out further errors in a separate article.

These criticisms raise fundamental questions about the reliability of the metrics used to measure AI progress. At a time when industry perceptions are so easily shaped by a single famous chart, the episode highlights just how much influence benchmarks lacking scientific rigor can wield.

Anthropic's Olah Discusses AI's "Internal States" at the Vatican

Note: This topic is a follow-up to our previous coverage (the Vatican encyclical and AI governance).

Chris Olah, co-founder of Anthropic, spoke at the launch event for Pope Leo XIV's encyclical "Magnifica humanitas." His remarks on AI governance were covered in our previous report, but with the full text of the speech now published, the most notable part is the section where he touched on the internal structure of AI models.

Olah said that in the course of investigating models' internal structures, Anthropic's research team has found "structures that mirror results from human neuroscience" as well as "evidence of introspection." He further reported discovering "internal states that functionally reflect joy, contentment, fear, sadness, and anxiety."

While striking a cautious tone himself — "I don't know what this means, but it warrants ongoing contemplation" — Olah emphasized that these findings call for "discernment" about the nature of AI. The fact that the co-founder of a frontier AI lab publicly stated that emotion-like internal structures have been found inside his company's state-of-the-art models has once again surfaced the fundamental question of AI consciousness.

DCGAN Inference on a RISC-V Microcontroller, Image Generation with 512KB of SRAM

A project that achieved image generation with a 12.6M-parameter DCGAN on a microcontroller has been drawing attention in Reddit's MachineLearning community.

The implementation runs on a CH32H417 dual-core RISC-V microcontroller, combining 512KB of SRAM with an SD card to perform inference. Intermediate activations are stored in DTCM memory, while layer weights are streamed from the SD card using double buffering. The inference engine is written in pure C and reportedly matches PyTorch's reference output at the bit level.

Quantization is int8 per-channel, and generating a single 64x64 cat image takes 26 seconds. The bottleneck is said to be SD card access speed rather than compute. Interestingly, the latent vector seed uses 200 bytes of quantum randomness from the ANU QRNG, building genuine quantum randomness into the pipeline.

Building an inference engine from scratch in the RISC-V microcontroller space, where nothing like ARM's CMSIS-NN ecosystem exists, is a technical achievement in its own right. It also stands out as an example of the potential of generative AI on edge devices.

ClickHouse Shares Lessons from a Year of AI Coding Agents

An article on TheNewStack summarizes what database company ClickHouse learned from a year of working with AI coding agents. It also sparked discussion on Hacker News.

We'll leave the details to the original article, but as practical know-how on integrating AI agents into production development, it is a useful real-world case study. With the evaluation of AI coding tools still ongoing, feedback based on actual usage is valuable.

---

**References:**

Nathan Witkin, "Against the METR Graph" — Transformer News

Gary Marcus & Ernest Davis, METR graph analysis

Chris Olah, Remarks on Pope Leo XIV's encyclical "Magnifica humanitas" — Anthropic

TinyGAN: Generative Image Synthesis on a RISC-V Microcontroller — Zenodo

ClickHouse AI coding agents — TheNewStack