The latest RAG trend for the current agent harnesses (Codex, Cowork) is to do two passes of document processing to solve a knowledge work task over a data room of documents: 1️⃣ A fast and light pass, oftentimes using a free/OSS doc parsing tool. This can be cheaply run across 10-100-1k’s of files, and enables the agent to then do retrieval (e.g. grep, semantic) to find relevant subsets of context. 2️⃣ A “just-in-time” VLM-based pass. Once the agent finds the relevant pages of context, it will screenshot the documents can call its own VLM (or write code) to dissect the pages. The issue with only using VLM-based OCR tools over massive ad-hoc customer file dumps is that it’s slow and expensive. Doing JIT VLM OCR allows the agent to filter through the data cheaply, but still preserve accuracy for the context that’s needed for the task. The agent harnesses do two-pass document processing by default using off the shelf-tools: pdf2text as the first pass, and using itself (Opus 5) as the second pass. See the below video where Cowork runs over a bunch of PDFs to answer a question about a benchmark graph in the Kimi k3 paper. The main issues here with the “out of the box” doc processing these agents offer are: * Opus 5 is not the best VLM for OCR. It is also way too expensive at scale and lacks grounding * The OSS tools like pypdf, pdf2text, may not be versatile enough as the first pass. * The agent will write a lot of throwaway code to rewrite things an OCR tool would’ve provided out of the box, like chart processing, bounding boxes, confidence scores, leading to increased cost and speed. We have all the tools within @llama_index to help any agent do two-pass document processing with higher accuracy and lower cost. 1️⃣ We have liteparse for the first pass - a free/OSS parser written in Rust that’s faster/more accurate than other OSS parsers, and supports 50+ document types 2️⃣ We have LlamaParse for the second pass - an agentic document engine that uses VLMs+harnesses to achieve SOTA in accuracy and cost across various doc parsing and extraction tasks. It can be called from any agent harness as an MCP or skill. It takes in page numbers as input, so that the agent can choose to run LlamaParse over a subset of the doc instead of the full doc as a “zoom-in” pass. Come check it out! LiteParse: https://t.co/JNER0mVcB8 LlamaParse: https://t.co/XYZmx5TFz8 All the relevant docs, including MCP, are here: https://t.co/qc9Q5NT3Jr

+1 more

we've been iterating a lot on how DocWriter internally represents a user's writing style! the naive approach is to dump all a user's prior writing in context and pray the AI figures out how to sound like them. this doesn't really work. and the user can't just manually specify their style either, because nobody knows what their meaningful writing preferences (for an AI agent) actually are. so we've built a multi-agent pipeline that runs lexical, grammatical, and discourse analyses on the user's writing, extracts specific patterns with evidence, and then surfaces them in a UI where the user can confirm or reject each pattern by comparing two passages side by side TBD on how well it performs in our next round of user studies 😆 it feels like we are really trying to do anything possible to avoid fine-tuning, but perhaps we need to bite the bullet eventually

Connecting AI to hardware requires days or weeks of bespoke integration, with no standard way for agents to operate equipment safely. MHS cuts integration to hours or minutes, provides an interface that makes devices discoverable, and enables agents to operate them safely.

We need to normalize measuring and judging models against a standardized test harness "Oh but model X performs best in their own proprietary harness" I could not care less. When I take exams, I go to the standardized classroom, get the standardized pencil and exam sheet, and have to solve it under 2 hours This system arose because we have a LOT of people to test Guess what? We now have a LOT of models, and they are multiplying by the day "Oh but model X performs substantially better in ARC-AGI-3 with a custom harness" I don't care... Then imbue model X with enough knowledge so that it can reconstruct that harness on the spot The main harness could be mini-swe-agent, terminus 2, vanilla pi or something along those lines It needs to be simple, and stay roughly the same over time There is already too much complexity in the benchmarking space right now, and I feel like not enough people are putting their feet down to cut away some part of it

🧵Scaling Laws for T2I Diffusion We do a scaling laws ladder for t2i diffusion models all the way from 60M to 2B, spanning 3 orders of magnitude in training flops https://t.co/GtxmABJAcq

Grok 4.6 just took the #1 spot on CursorBench 3.2.....and the efficiency is insane Here's the cost comparison: • Grok 4.6 Extra High — 70.8% | $2.81/task • Fable 5 Max — 70.5% | $17.32/task • Opus 5 Max — 70.0% | $8.23/task • GPT-5.6 Sol Max — 67.2% | $5.69/task Grok achieved the highest score while costing roughly 6X less than Fable 5 Max and nearly 3X less than Opus 5 Max per task That’s what makes Grok so powerful for agents Top-tier intelligence is great.....but top-tier intelligence that can keep working across long coding tasks without burning ridiculous amounts of compute is even better Grok’s agentic coding efficiency is insane

📈 New blog post: Training and Finetuning Multi-Vector Embedding Models with Sentence Transformers. As a practical example, I finetuned a ColBERT-style model for medical retrieval. 14.5 hours on one RTX 3090, and it beats every general-purpose retriever I could find. Thread 🧵 https://t.co/zRWY3j3ivJ

Great new paper from AWS on agent handoff tax. If you build agents today, you need to understand the so-called handoff tax. (bookmark it) Escalating to a stronger model mid-run is usually the resort when a cheap agent stalls. New work from AWS AI Labs measures how much that switch actually costs. Coding agents run for dozens of model calls, so teams escalate when a weak model struggles and downshift once the hard reasoning is done. Every switch forces the receiving model to continue a trajectory another model wrote. Across pairs of Claude and GPT models, full-trajectory escalation recovers less than half the quality gap between the weak and strong model while adding a substantial cost premium. The authors call that penalty the handoff tax. Downshifting lands at a much better cost-quality point. Cutting the weak model's trajectory information improves escalation quality, while removing the strong model's trajectory hurts downshift quality. Paper: https://t.co/59ozKgugB8 Track more trending AI papers in our academy: https://t.co/1e8RZKs4uX

Today we’re introducing Navigator n2. It’s a frontier computer-use model, with just 27B parameters. https://t.co/8MiJNtk3CD

We pre-trained Atlas from scratch to take multimodal inputs, including camera movement, and turn it into 3D grounded views. Atlas puts you in control: direct the views, reconstruct real spaces with your inputs, and build explorable worlds. Read more: https://t.co/SHjFaNWedP