Digest & Practices
📌 Digest & Practices
What the feed surfaced, week by week — and what we currently think it means
Current understanding — applied AI
Distilled 2026-08-27 · 15 practices. Each is a dated claim by a named practitioner, not a consensus result — one post is a lead worth testing; agreement across independent accounts is a strong prior.
Evaluation
A Generalizability Theory decomposition across three agent benchmarks found the agent main effect accounts for under 3% of total variance, while the agent-by-task interaction accounts for 7-23%. Leaderboards rank specialisation, not capability. A 30-50 item golden set on the work you actually do is the only reliable signal.
@dair_ai · 2026-08-13Twelve open-weight models answered the same 3,679 items under 26 equally defensible harness configurations, varying only option order, prompt wording, and whether the answer is read from generated text or per-option likelihoods. One model landed anywhere from 31% to 89% on configuration alone. Pin and report your harness, or your number means nothing.
@dair_ai · 2026-08-25"Reporting benchmark results as a scalar number, e.g. '75% on XYZ' is completely meaningless at this point. You should always report efficiency scores, e.g. '75% at a cost per task of $10.'" Visible in practice: a model taking the top CursorBench slot at a fraction of the cost of those behind it was read as the real signal, because sustained affordability is what permits long agentic sessions.
@fchollet · 2026-07-06Work on coding-agent reward signals found test pass rates, LLM judges and execution traces each have a horizon beyond which they stop tracking real correctness. The metric you pick matters less than how long it keeps tracking the thing you care about. Benchmarks rot the same way: SWE-Bench Pro was audited, found ~30% broken and saturated at a ~70% noise ceiling, and formally retracted.
@omarsar0 / @OpenAI · 2026-07-08Document extraction
ExtractBench (370 enterprise documents, 14 systems) found the dangerous failure is not a wrong value but a missing row that looks like nothing went wrong. On the longest documents frontier models 'do not misread these docs, they abandon them': precision stays high while recall collapses to 8.9-35.8% F1. Every row returned looks correct, so a human spot check passes while most of the document is missing. Only a count against an independent source detects this.
@llama_index · 2026-08-12A fast, cheap (often free/OSS) parse runs across all files -- enough for the agent to retrieve the relevant subset by grep or semantic search. Only then does a just-in-time VLM pass run on the pages that matter. Keep single-pass where every page is genuinely needed (spec sheets, invoices); the two-pass split is for corpora you search rather than read.
@jerryjliu0 · 2026-08-23A vendor-reported jump from 57.1% to 80.6% on document reasoning did not transfer to parsing: on ParseBench the same model gained +42.3 on charts but regressed -2.5 on layout, with content and tables flat. Re-test on the shape of work your pipeline actually does.
@llama_index · 2026-04-16Agents
Microsoft's Thinkingbox grades 507 policy-conditioned workflows on the backend state the agent leaves behind; executable checks reject wrong, missing or extra effects, so collateral damage counts against you. The strongest model reached 65.36% pass@1 but only 25.25% pass^20, and many failed trials terminated cleanly -- the agent believed it had succeeded.
@dair_ai · 2026-08-22A comparison across 14 models on BFCL v4 found programmatic tool calling -- tools presented as typed Python stubs invoked through code, with execution and results handled in one agent turn -- matched or beat native JSON tool calling in 11 of 14 models, rising to 13 of 14 under parallel fan-out. The GPT-5.6 family gained 10.6% over the JSON baseline.
@dair_ai · 2026-08-10Re-implementing a harness with retained reasoning and context compaction moved a score by 188% while using 6x fewer output tokens -- same model. An eval measures 'a bundle of less visible choices about API settings, harness design, and prompting', not a model in isolation.
@OpenAI · 2026-07-29Context
The COMPINT suite found current context compactors retain only 17% of session constraints -- instructions like 'do not delete any emails until I confirm' that are meant to bind the agent for a whole session. Compaction drops them silently, and compactors 'often leave the task worse off than running it without compaction at all'. Re-assert standing rules structurally after every compaction.
@dair_ai · 2026-08-13Four named failure modes for long-running agent teams: working state vanishes when a terminal closes; compaction blurs each agent's working detail; decisions get trapped in compacted chats and accumulate as 'agentic technical debt'; and every handoff needs another long prompt. The fix gives each agent a workstation directory holding its state alongside its skills, hooks and scripts.
@omarsar0 · 2026-08-01Reliability
Hosted databases return read timeouts and Cloudflare 52x pages under load; a one-second outage cost this project a full hour of ingestion. Match on exception TYPE -- httpx.ReadTimeout's message is 'The read operation timed out', which contains no 'timeout' substring. And never swallow a blip into a more expensive path: returning 'no previous data' here would trigger a full re-fetch of a whole timeline, billed per tweet.
notes_app · 2026-08-24A per-item failure is recorded and the run continues; the job exits non-zero only when failures cross both an absolute and a proportional threshold. The same reasoning retired a CI check here that had failed 60 of 60 runs across a year -- a permanently-red signal trains everyone to ignore it, and a real failure then arrives looking identical to the noise.
notes_app · 2026-08-24Scope
GPU kernel work, quantisation formats, training throughput and inference-server tuning dominate the technical volume -- 166 high-signal posts in 180 days. It is important work, but every pipeline in this portfolio is an API consumer, not a model trainer. Cost control here means fewer and cheaper API calls, not faster kernels.
corpus analysis · 2026-08-26By week
Top posts per ISO week, last 8 weeks, relevance ≥ 0.6. Retweets excluded — a retweet duplicates a claim already here under its original author.
2026-W36 · week of Aug 31
77 high-signal posts from 45 accounts · partial week (clipped by the window)
Training tiny models for special purpose use cases works so incredibly well if you have a great self improving recursive flywheel. Shopify ML team is on fire. finetuned 0.8b model beats GPT 5.6-sol xhigh in this very specialized task. https://t.co/w6OCWyWRi5
open ↗MR-JEPA: A General Purpose Video Foundation Model for Cardiac MRI "We present MR-JEPA, a self-supervised video foundation model for Cardiac MRI that extends LeJEPA to 3D spatiotemporal inputs through tubelet tokenization, spatiotemporal masking augmentation, and initialization fr…
open ↗Code: https://t.co/2KHMVKk28t Model: https://t.co/Ea8AKBkv4S Paper: https://t.co/RoPmG7boHH Tech blog: https://t.co/aBTMmOL0WS Discord: https://t.co/7hr2hzUbPC
open ↗⚙️ Performance starts here. The Kernel Engineering Track at #PyTorchCon North America (Oct 20-21 in San Jose) explores compilers, custom kernels, optimization, and the low-level technologies that make AI run faster. Learn more: https://t.co/rg8DxFJpyv 📅 Full Schedule: https://t.c…
open ↗Loop engineering has emerged as a new skill for AI engineers But there is very little research measuring how effective it is. The best results on full tasks in a new benchmark is ~25%. LoopArena from AMAP evaluates the outer loop rather than the coding agent. A Controller model r…
open ↗2026-W35 · week of Aug 24
341 high-signal posts from 165 accounts
I made a video about what happens when you execute `c = a @ b` in PyTorch, from the code to silicon. Hopefully it will help you appreciate that simple line of code more. Made by Grok @bot. Enjoy! https://t.co/hdn0fFSIA8
open ↗Today, we're kicking off the first phase of the research preview for Model Hardware Standard (MHS): a new standard for AI agents to safely operate physical equipment in scientific research and advanced manufacturing. Read more: https://t.co/XQ2y9EW7Af https://t.co/kgyCvZ6iYc
open ↗Demystifying Reinforcement Learning Post-Training of Language Models A very useful exploration of RLVR in a simplified setup, lots of good insights... "Result 1: sparse RL cannot find what the model never samples" "Result 2: dense rewards break that ceiling" "Result 3: "spurious …
open ↗Introducing our new work, Code World Model—a new world-model paradigm with a language model as its brain. A Coding Agent continuously maintains and updates the world state, which then guides a video model to generate high-fidelity visuals. Two key motivations: 1. Complex world in…
open ↗The ultimate test for coding agents isn't local editing— it's whole-repo evolution, and right now, the survival rate is 5.4%. Today we’re releasing SWE Refactor Bench, a benchmark for long-horizon, whole-repository software stack migration. Coding agents are getting very good at …
open ↗If you're looking for a weekend project, how about training your own text-to-speech model from scratch on your own GPU, and then running it on any device's CPU? We just open-sourced the entire Pocket TTS training stack: data pipeline, recipes, and evals. It learns pretty damn fas…
open ↗2026-W34 · week of Aug 17
337 high-signal posts from 172 accounts
Someone asked what I'd do if I were 17. I'd learn how to build LLMs from scratch, and then train ones as powerful as I could with whatever hardware I could get access to.
open ↗sharing a new long-form blog post: ai chip architectures it covers the leading chip architectures (nvidia, amd, tpus, trainium, cerebras, groq) across architecture, scaling (scale-up and scale-out), and software stacks. it helps build an intuition for the architectures and their …
open ↗A couple of days ago, I did a quick explainer on Claude’s new watermarking process and implementation. Since it’s such a popular topic and sparked such a lively discussion, I thought it might be interesting to go into a bit more detail when explaining how it works. So, instead of…
open ↗FreeToken is fast. Comparing to Ollama, we have 3–4× faster decode, and 6–30× faster prefill How? We introduce bandwidth-adaptive CPU–GPU execution + semantic-aware caching across agent turns. More details in the technical report: https://t.co/V3sIxbPMGU https://t.co/IsJZfQJ1FL
open ↗Qwen 3.8 Max on the Surge Scorecard: Tuesday Work Index — 58.7 ComplexConstraints — 45.5% Chartography — 29.1% HANDBOOK.md — 16.5% Riemann-bench — 15.2% Hemingway-bench — 1006 Elo Antidote — 990 Elo https://t.co/3MzHkqo1m4
open ↗2026-W33 · week of Aug 10
372 high-signal posts from 186 accounts
Trying GLM 5.3 right now. Avg: Prefill ~1 ktok/s, thinking/output ~60 tok/s. In our harness, we already see with full thinking traces GLM 5.2 > Fable+Claude Code. But GLM 5.3 is just on another level. It is very precise and concise. Just testing long-task performance. Exciting…
open ↗Grok 4.6 from @SpaceXAI on ARC-AGI (Verified): - ARC-AGI-1: 87.5%, $0.30/task - ARC-AGI-2: 67.1%, $0.76/task - ARC-AGI-3: 2.11%, $5.6K On ARC-AGI-3, Grok 4.6 with xhigh reasoning scored comparably to GPT-5.6 Sol with high reasoning, but cost $5.6K versus Sol's $15.2K. https://t.c…
open ↗We wrote a 36-page ArXiv whitepaper on ExtractBench 🧑🔬 , our effort to create the most comprehensive, schema-guided, real-world document extraction benchmark. It’s extremely detailed and covers everything from comparisons with related work on document extraction, to the dataset …
open ↗Build a Reasoning Model (From Scratch) is now finally also available on Amazon! Thanks so much to everyone who preordered! One important note for readers in India: please avoid ordering through @amazon India (@amazonIN) for now since they are selling counterfeit black-and-white c…
open ↗We ran the largest open experiment on how frontier models do AI research. 100+ autonomous runs across 10+ models, sandboxed on 8xH200s for up to 8 days, iterating on the nanoGPT optimizer track. Best runs closed 82% of the gap to a record built by dozens of humans over months. ht…
open ↗32GB VRAM (e.g. RTX 5090): llama serve \ -hf ggml-org/Qwen3.8-27B-GGUF:Q4_K_M \ -hfd ggml-org/Qwen3.8-27B-GGUF:Q4_0 \ --spec-default \ --spec-type draft-mtp \ --ctx-size 196608 \ --cache-type-k q8_0 \ --cache-type-v q8_0 \ --reasoning-preserve --fit off --agent
open ↗2026-W32 · week of Aug 03
287 high-signal posts from 157 accounts
Qwen 3.8 Max is spectacular! I benchmarked it on the INDUCTION benchmark and it comes second, behind GPT 5.6 Sol and ahead of Fable 5 by one problem. Incredible progress from Qwen 3.7: it gets 42% correct, compared to 0%. INDUCTION, is a challenging reasoning benchmark, described…
open ↗We’re releasing the manuscripts, formal Lean certificates, and reasoning walkthroughs so mathematicians can examine these results and build on their ideas. https://t.co/oDT2J8F6Ez
open ↗Qwen3.8 Max did not beat every frontier model. It did something more important: it split the leaderboard. PaperBench: 93.0 vs GPT-5.6 Sol at 90.5 OSWorld-Verified: 86.1 vs Fable 5 at 85.0 IFBench: 82.8 vs GPT-5.6 Sol at 72.7 Dense200: 87.0 vs Gemini 3.1 Pro at 69.7 Fable still do…
open ↗this is the most complete 8-minute breakdown of Claude Code I’ve ever seen. it contains everything you need to know. save and watch this. https://t.co/VxakdeBxyd
open ↗🚀Excited to share Qwen-3D (ECCV 2026) — a generalist 3D vision–language model that does grounding, segmentation, VQA, and spatial reasoning in one place. VLMs are great at images and short clips, but long videos bury them in tokens. Our idea: put multi-view frames into a shared 3…
open ↗Introducing Magnitude: your actually local agent 100% private and offline. No token costs, no API keys. Open source. Today's agents are local. The model isn't. Every prompt, every file, every secret gets sent straight to Anthropic and OpenAI. Magnitude is built around local model…
open ↗2026-W31 · week of Jul 27
379 high-signal posts from 179 accounts
// Persistent Workspaces for Long-Lived Claude Code Agent Teams // Four issues to be aware of: > Working state vanishes when a terminal closes and the team cannot be resumed. > Compaction condenses the conversation and blurs each agent's working detail. > Decisions and operations…
open ↗DeepSeek V4 Flash 0731 can now be run locally! 🐳 Run DeepSeek V4 Flash lossless 4-bit on 168GB RAM and 3-bit on 110GB RAM. V4 Flash 0731 outperforms V4 Pro. Run via Unsloth or llama.cpp. Smaller quants coming today. Guide: https://t.co/mCZZkpa95X GGUF: https://t.co/ac6uOI8mZA htt…
open ↗We implemented the harness with the Responses API and turned on: → Retained reasoning → Context compaction On the public set, GPT-5.6 Sol’s score rose 188% while using 6x fewer output tokens. https://t.co/uN1IrKEugu
open ↗Opus 5 built this flight simulator in one go. Using only Three.js. The trick to getting this quality and higher (if you prompt harder) seems to be in building a good judge-executor harness. The judge keeps the loop running to improve output quality. https://t.co/PIivNHtZ0r
open ↗PyTorch 2.13 brings FlexAttention to Apple Silicon, cuts peak memory by up to 4× for large-vocabulary models with nn.LinearCrossEntropyLoss, and updates distributed training, compilation, profiling, and on-device inference. Our live Q&A examined CUDA version support, CuTeDSL in T…
open ↗2026-W30 · week of Jul 20
352 high-signal posts from 183 accounts
A diffusion LLM built for agentic work, from the inclusionAI team at Ant Group (@ant_oss). The weights and code are open. Model: https://t.co/f0OUWOrwhD Code. https://t.co/ZXF1dLPmD3
open ↗Claude Opus 5 from @AnthropicAI is the new SOTA on ARC-AGI-3: 30.2% The previous high score (7.8%) was set by GPT-5.6 Sol (Max) Throughout our analysis, we observed novel behavior that allows Opus 5 to solve previously unbeaten environments, outperforming Fable https://t.co/Dg3uO…
open ↗Opus 5 sets a new state-of-the-art on ARC-AGI-3, at 30%. ARC-AGI-3 measures solving problems with no prior exposure -- the setting where scaling has historically bought the least. Impressive jump!
open ↗Introducing the LLM Inference Handbook: tokenization, KV cache, quantization, speculative decoding, batching, routing, disaggregated inference, and kernel optimization. All the pieces of production LLM inference, explained in one handbook with diagrams and examples. https://t.co/…
open ↗Contrastive SDF gives copies of the same model opposing beliefs about what the grader prefers, then measures how their behavior changes. https://t.co/eXSh5LKJ0j
open ↗The International Math Olympiad (IMO) 2026, the hardest math contest for high schoolers, just ended. I ran Fable (high), Sol (xhigh), K3 (max) and Axiom against it and all got a perfect score of 42/42 (repo below if you want to check their solutions): — Claude Fable 5 was the sol…
open ↗2026-W29 · week of Jul 13
378 high-signal posts from 184 accounts
what is a multimodal LLM thinking as it watches a video? Gemma 4 12B reads raw image patches, as if they were tokens. It was never trained to predict anything at these 'tokens' - but this video shows what it would predict if you did sample from its next token prediction head http…
open ↗GPT‑Red learns through adversarial self-play, where its goal is to prompt inject a variety of challenging defender models. Every successful attack that GPT-Red finds is used to improve these defenders, pushing GPT‑Red to continuously find broader and more complex failures.
open ↗I'm defending my PhD next Tuesday! The talk is titled "The Missing Science of AI Evaluation" and is based on my faculty job talk. I'll talk about my work on AI-based science, agent evaluation, and open model risks. The talk will be livestreamed and is open to all. https://t.co/Bk…
open ↗LingBot-VLA 2.0 shows practicality too. Robbyant open-sourced the post-training code. Inference takes about 130 ms on a single NVIDIA GeForce RTX 4090D using 10 denoising steps. That means you can adapt and test it without a cluster. https://t.co/2TdlEKoEQw
open ↗New in-depth blog post time: "Inside TPU and GPU Clusters: The Anatomy of Collective Communication". If you want to deeply understand the core primitives behind scaling the training / inference for MoEs and dense transformers, going a level below FSDP, expert parallelism, data pa…
open ↗GPT-5.6 Sol dominates our ErdosBench. It's a clear top performer with 78 problems solved (full solutions, counterexamples, citations from literature), compare to 55 by GPT-5.5 xhigh. ErdosBench contains 226 research-level Erdos-like synthetic open math problems with no solutions …
open ↗2026-W28 · week of Jul 06
293 high-signal posts from 137 accounts · partial week (clipped by the window)
@dorukkavcioglu @konstipaulus @diffusionhq Full tutorial is on YouTube 🎉 Install to final render, every prompt I used, every bug: https://t.co/XXlBnbW39T
open ↗The capacity trick is a sparse Mixture-of-Experts video stream. 128 experts, top-8, one shared. About 13B total params, about 1.9B active per token. The action stream stays dense. Capacity for hard visual dynamics, low per-step compute for high-frequency control.
open ↗The result: roughly 20% lower average review cost, while maintaining the same review quality. ✅ The bigger lesson: for an agent, tool instructions are as load-bearing as your API docs. Here's how we traced the regression to the instructions ⬇️ https://t.co/r9KOkDlPMi
open ↗it didn't get there by brute force. Nobody told it crystals are periodic. It worked out that raw atom-to-atom distances misread the chemistry across a cell boundary, then rebuilt them as metal-to-oxygen nearest neighbors. 677 features by the end. 😳 https://t.co/fcAFaJbetK
open ↗What I care about most: this ships as a real foundation, open and reproducible, with a paper, code, and weights you can run today. That is how the whole field moves faster. This is the bigger AI story I track at AlignedNews. Code:https://t.co/8CLaJXopJz Model:https://t.co/xlB9T1W…
open ↗I benchmarked the new models (Sol, Terra, Luna, Fable 5, Meta Muse Spark 1.1, Grok 4.5) on an induction reasoning task. This is an updated table from yesterday, and my benchmark is described as spotlight in ICML '26. How to read the table: The models are given several small graph…
open ↗History
A snapshot is taken automatically on the first sync of each ISO week, recording every recommendation's verdict and the week's feed volume. Neither is recoverable later — the practices file is edited in place, and rescoring moves historical feed numbers.
High-signal posts per week
2 snapshot(s) · 2026-W35 → 2026-W36
Recommendation changes
No changes recorded yet — needs at least two snapshots.
Most-revised recommendations
Every recommendation has held its verdict so far.
Weekly data is computed live from 2,816 scored posts. Practices are curated in data/applied_ai_practices.json.