LLMs Lie When They Say "99% Confident" — A New Technique Makes Them Verbalize Confidence Accurately

Ask an LLM "How confident are you in this answer?" and it will almost certainly reply "99% confident." Yet analyzing the model's internal hidden states reveals that it can actually distinguish correct answers from incorrect ones with an AUROC of 0.76–0.88. In other words, **the model cannot verbalize what it actually knows**.

A study that gained attention on Reddit's r/MachineLearning proposes "probe-targeted fine-tuning," which uses probe outputs from these hidden states as the LoRA fine-tuning objective. With just a few hundred examples and less than 10 minutes of training on an M3 Ultra, the model reportedly learns to verbalize its internal awareness honestly.

The method has been validated across 8 models from 4 families (7B to 70B), with causality confirmed through activation patching (ρ = 0.976). In 70B-class models, internal metacognitive signals do exist, but the researchers also observed a phenomenon where the argmax text-output bottleneck pins the stated confidence at "99%." A fascinating result that points to a new approach for evaluating LLM reliability.

Let a Coding Agent Speak HTML and It Can Draw SVG Diagrams

"AI agents keep outputting markdown all the time — what happens if we make HTML their primary language?" That's the experiment making waves on r/LocalLLaMA.

A developer changed the system prompt of a browser-based coding agent from markdown to HTML. As a result, Qwen3.6-27B became able to render SVG diagrams directly within the chat. Tables and layouts can also be built with HTML tags, dramatically improving expressiveness.

That said, the model still tends to drift back to markdown, and explicit instructions are reportedly needed for a complete switch to HTML. On Qwen3-VL-4, SVG quality was remarkably poor, and the analysis suggests that SVG drawing is likely "an emergent capability of the model." The code is available on [GitHub](https://github.com/sdfgeoff/HTML-agent).

The Pope's First AI Encyclical "Humanitas": Evidence That Most of It Was Written by AI

An analysis post on Substack presents **evidence that most of "Humanitas" — the Pope's first encyclical on AI — was written by AI (Claude)**, sparking debate on Hacker News.

The analysis identifies traces of AI generation from stylistic patterns, word choice, and structural features. Lively discussion has unfolded from both religious and AI-community perspectives on the ethical and social implications of using AI for faith documents.

Skill0.5: Unifying Skill "Internalization" and "Utilization" in Agent RL

Featured on Hugging Face Daily Papers, Skill0.5 resolves the binary dilemma in skill-based reinforcement learning between "full externalization (excess context) vs. full internalization (overfitting and knowledge conflicts)."

A difficulty-aware router sorts tasks into three tiers: hard tasks internalize general skills via privileged distillation, medium tasks maximize success rates with standard RL, and easy tasks penalize shortcuts to enforce faithful skill utilization. The method outperforms prior approaches on ALFWorld and WebShop.

Ptah: A Trustworthy Multimodal Report Generation Harness

Also featured on Hugging Face Daily Papers, Ptah is a multi-agent harness that coordinates textual research, claim-grounded evidence, and visual evidence of source consistency.

It produces reports in a format where images and text alternate, and the authors also propose PtahEval, a protocol that evaluates factual grounding, citation faithfulness, visual relevance, and cross-modal consistency.

---