Can Vision LLMs Beat OCR? A Real-World Long-Document QA Benchmark, a Comprehensive TTS Benchmark Arrives, and Perplexity Open-Sources a Supply Chain Defense Tool

Can Vision LLMs Beat OCR? — What a Real-World Long-Document QA Benchmark Shows

A benchmark result has been published that is drawing attention in both Reddit's LocalLLaMA and MachineLearning communities. It uses 30 long-form PDFs containing images, tables, and graphs to compare two approaches: feeding PDFs directly to a Vision LLM versus running them through an OCR pipeline before processing with an LLM.

The test used Claude Sonnet 4.5 and evaluated it on 171 QA tasks. The results are as follows:

| Approach | Accuracy | Cost per question |

|---|---|---|

| LlamaCloud premium + full-context | 59.6% | $0.1885 |

| Azure premium + full-context | 58.5% | $0.2051 |

| Azure basic + full-context | 54.4% | $0.1062 |

| Agentic RAG | 53.2% | $0.0827 |

| **Native PDF (Vision LLM)** | **52.0%** | **$0.2552** |

| LlamaCloud basic + full-context | 50.9% | $0.1049 |

Native PDF (direct reading by a Vision LLM) ranked 5th out of 6 methods and was also the most expensive. Vision's weakness was particularly noticeable on pages dense with charts and tables, while premium OCR + layout extraction proved more robust.

Additionally, the Native PDF approach showed a 7% intrinsic failure rate (failures that persist even after retries), caused by issues stemming from PDF file size. In contrast, OCR-based methods had a 0% failure rate after retries.

That said, the sample size of 30 documents is small: according to McNemar's test, statistically significant differences were confirmed in only 3 out of 15 pairs. While the Vision-vs-OCR comparison itself is statistically significant, keep in mind that some of the rankings may be noise.

AutoRubric-T2I — An Interpretable Reward Model for Text-to-Image Alignment

A study called "AutoRubric-T2I" has been featured on Hugging Face's Daily Papers. It is a method for automatically learning natural language rubrics (evaluation criteria) from a small amount of image preference data, applied to reward models that evaluate the quality of text-to-image generation models.

Conventional reward models require large amounts of annotated preference data, but AutoRubric-T2I cuts the required annotation data to less than 0.01%. It enables VLM (vision-language model)-based evaluation and works without fine-tuning. It is also highly interpretable — you can check in natural language why a given evaluation was reached.

A Comprehensive TTS Benchmark as of May 2026 Has Been Released

A Reddit user, noting that there was no proper existing TTS benchmark, has released "tts-bench", an open-source benchmark that comprehensively compares major TTS models.

Results for Windows and Mac are already available, and a Linux version (Radeon 5900XT + RTX 3090 workstation) is coming soon. The results can be viewed on an HTML page, making this a practical resource for choosing a local TTS tool.

Perplexity Open-Sources Supply Chain Defense Tool "bumblebee"

According to Qiita's GitHub daily report, Perplexity has open-sourced "bumblebee", a supply chain defense tool written in Go. It debuted on GitHub trending and has already racked up 1,300 stars.

Supply chain attacks have become a threat to the entire open-source ecosystem, and a recent large-scale poisoning incident by TeamPCP has also been reported. Against this backdrop, the arrival of a tool specialized in supply chain security is worth noting.

RAM Offloading for Massive Models — How Big Is the Performance Gap Between GPU VRAM and System DRAM?

On Reddit's LocalLLaMA, discussion is heating up over how to divide work between GPU VRAM and system DRAM when running massive models like DeepSeek V4 Pro locally.

GPU VRAM is expensive but fast, while system DRAM is cheap but narrow in bandwidth. In MoE (Mixture of Experts) architectures, experts switch frequently, so data transfer between VRAM and DRAM can become a bottleneck.

The focus of the debate is whether an actual performance difference emerges between the RTX 5090 (48GB) and RTX 6000 (96GB), and how effective VRAM caching of experts really is. No definitive conclusion has been reached yet, but it has been suggested that the decoding bottleneck may depend on system RAM bandwidth.

---