curious people and people interested in llm evals, benchmarks, high data circles, please wake up. my blog post "SWE-bench Multimodal v2 is not multimodal" is up now. https://t.co/Kqd1oOExIn https://t.co/Vd4nPqjiTx

Evals built by @PrimeIntellect residents, fully open: - MazeBench by @patience_cave - PMPP-Hard by @myainotez - PretrainGym by @HuNtEr_J1324 - OpenTTD-Bench by @thesophiaxu [and even more :) ] Super happy to support such amazing work, check it out!

Evals built by @PrimeIntellect residents, fully open: - MazeBench by @patience_cave - PMPP-Hard by @myainotez - PretrainGym by @HuNtEr_J1324 - OpenTTD-Bench by @thesophiaxu [and even more :) ] Super happy to support such amazing work, check it out!

Today we are launching FrontierCS 2, the most frontier benchmark for AI agents in computer science R&D. We built 48 tasks from published papers across 13 areas of CS, refining each one with experts in its field. Excited to co-launch FrontierCS 2 with @kaggle @GoogleDeepMind https://t.co/rfWyJ9tFsf

New research: We developed JOLT, a post-training method where one model acts as both teacher and student, with the teacher trained to give feedback the student can use. On MATH, it matched GRPO's accuracy with 6.5x fewer training tokens. https://t.co/KFHdZ0tuFn https://t.co/Y4Svd9QK4G

We’re working to build the largest collection of private and public practitioner evals. Today, 31 new partners join the Specialized Intelligence Index. Compare open, closed, and specialized models on real work, weigh quality against cost, and see where training could pay off. https://t.co/Ofq6wHNe8w

Plotting ARC-AGI model scores reveals a strong relationship across all three benchmarks. For example, the power curve v2 = v1^4.53 explains about 98% of the variation in ARC-AGI-2 scores. Models tied on ARC-AGI-1 can still differ by over 20 percentage points on ARC-AGI-2. https://t.co/7uG7hNXAwo

Marin's 535b is now >80% through pretraining! Here’s loss vs. our scaling-ladder forecasts. The sharp drop comes with the 4K→16K context switch, so we're kind of cheating the forecast. (This was planned.) We'll re-eval at 4k too. https://t.co/YPREnmluJY https://t.co/hU9lhL63yx

GitHub has released Copilot Fusion in GitHub Copilot, which orchestrates between Anthropic and OpenAI models. It scores 2 points below Opus 5.5 in Claude Code at half the Cost per Task in the Coding Agent Index GitHub describes Copilot Fusion as a runtime orchestration system, which selects and coordinates models to optimize performance, cost, and latency for each task. Copilot Fusion chooses between a number of models from Anthropic and OpenAI, including Claude Opus 5.5, Claude Sonnet 5.5, GPT-6 Astra, and GPT-6 Sol. We benchmarked its max setting in GitHub Copilot CLI in the Artificial Analysis Coding Agent Index v2.0. Token costs are charged based on the underlying models used to complete a request. Key takeaways: ➤ Cost-efficient for near-frontier performance: Copilot Fusion in GitHub Copilot costs $5.59 per task, versus $11.42 for Claude Code with Opus 5.5 (max). Devin Fusion with Opus 5.5 (max) + GPT-6 Luna (high) scores 62 at $8.00 per task. ➤ Slight lead in Terminal-Bench 4.0: The Coding Agent Index v2.0 weights its three constituent benchmarks equally. Copilot Fusion in GitHub Copilot leads Terminal-Bench 4.0 with a score of 71. It ranks third on FrontierCode in the chart at 51, behind Claude Code with Opus 5.5 (max) at 56 and Devin Fusion with Opus 5.5 (max) + GPT-6 Luna (high) at 54. Its SWE-Atlas-QnA-AA score lags at 57, between GPT-6 Astra in Codex and Gemini 4 Argon in Antigravity CLI. ➤ Half the token use of Opus 5.5 in Claude Code: GitHub Copilot Fusion used 7.2M tokens per task in the Coding Agent Index, approximately half of the tokens used by Claude Opus 5.5 in Claude Code, and 30% less than Devin Fusion with Opus 5.5 (max) + GPT-6 Luna (high).

https://t.co/TcaAfe4JAR Great way to try out MMLU and get a sense for just what, exactly, we are using to evaluate LLMs! I doubt folks would knife fight for a percent on this benchmark if its contents were realized more broadly. https://t.co/TMPlaKRAnf

NVIDIA just released MotionBricks on Hugging Face A real-time generative motion model for humanoid robots and characters, reaching 15,000 FPS with 2ms latency across 350k+ motion clips. https://t.co/IIxgL3ZQgL

Big day for the @PyTorch ecosystem at #PyTorchConf2026 🚀 We’re officially open-sourcing TorchTPU—enabling native, hardware-accelerated PyTorch execution across Cloud TPUs and custom infrastructure. Here’s everything we launched to supercharge your training & inference workloads 🧵👇 https://t.co/1X5A5bhG0z

llama.cpp v0.7.0, 128GB Mac/Spark: llama serve -hf ggml-org/Qwen3.8-Flash-Next-GGUF:IQ4_NL --spec-type draft-mtp --lazy-mode on

Together Post-Training is live Turn open models into custom models you own, with SFT or RL. You define the data, rewards, environments and the training loop itself. We run the hard infrastructure underneath it, tuned for quality and speed.

While QAT helps improve model quality at 2-4 bits, it has a structural optimization problem. Indeed, at every step, the high-precision latent weights receive the gradient updates, but the loss and gradients are computed on their low-bit reconstructions, i.e. the weights the model will use at inference. This mismatch results in suboptimal training dynamics and a higher loss floor. The main existing QAT methods (learning the scales alongside model parameters, soft-to-hard rounding, or better initialization) do not directly address this mismatch. With QUASAR, we introduce a simple and computationally cheap way to address the mismatch. QUASAR consistently achieves lower loss than all QAT baselines and adds no inference overhead. Also, it can be combined with most other QAT methods, since it only changes the weight reconstruction map in the fwd pass.

Can a JEPA world model provably avoid contraction and merging of latent states over long rollouts? Unconstrained autoregressive predictors may contract the latent distribution. Our Symplectic Hamiltonian JEPA (LeHami) replaces the predictor with an action-conditioned Hamiltonian flow over a latent state (q, p), integrated with an exactly symplectic leapfrog step. Volume is preserved at every rollout step, independently of the learned parameters or action sequence. With @randall_balestriero @ylecun Paper: https://t.co/JqJ7rrOXmF Website: https://t.co/I5pmtHM9VB Code: https://t.co/D9EIFGS87U 1/6... 🧵👇

friendly reminder: this is on Monday 🤝 → learn about speculative decoding to speed-up inference → if you're a beginner and not interested, just use https://t.co/eZJZOpPFkl that does this for you automatically 🦙 https://t.co/97svGez4R3

Built for long-horizon agents. Monitor agents against your own criteria. Search and investigate patterns at scale. Benchmark improvements and regressions before rollout. https://t.co/EIFfrfR72q

Our newest decision model is the world's best decision model. Introducing GLiDE v1.2, the #1 model on the @huggingface Decision Index. With an overall score of 71.5, GLiDE v1.2 also has one of the lowest calibration errors on the entire Decision Index, meaning its confidence more closely reflects how often its answers are correct. Beyond improved benchmark performance, it also adds support for multimodal inputs, enabling decision-making using text, JSON, and images. We are excited to release the model weights on Hugging Face. For hosted inference, you can also run GLiDE v1.2 through the Fastino API at $0.04 / million input tokens. Use promo code GLIDEIS1 for $10 in API credits. Model weights: https://t.co/FQ8F2XM971 API: https://t.co/vZ8enLvARJ Blog: https://t.co/7EeErLDqse

+1 more

Frontier performance at a fraction of the cost. In the @ArtificialAnlys Coding Agent Index, Copilot Fusion Max scored within two points of Claude Code with Opus 5.5 Max at about half the API cost per task. Get 30% off through Nov. 30. Try it now. ⬇️ https://t.co/c0k5alLKqD https://t.co/kVBzvq8ph5

An update here: we’re upgrading all Codex for OSS grants to Pro 200! There will still be the 10,000 grants in this round, now at the higher tier. (If you've already received a grant, the upgrade should be reflected automatically.) Applications are still open :)

An update here: we’re upgrading all Codex for OSS grants to Pro 200! There will still be the 10,000 grants in this round, now at the higher tier. (If you've already received a grant, the upgrade should be reflected automatically.) Applications are still open :)

We gave GPT-6 Astra and Claude Fable 5.1 open-ended AI research questions from unpublished papers, with a $10,000 budget each, and had the original authors grade the papers they produced. Compared to our first tests with Claude Opus 4.8 in July, we saw meaningful improvements in experimental design and resource management, but persistent limitations in creative problem solving, judgment, calibration, and communication; both papers were rejected by our experts. https://t.co/n1OMKf4M7T

One thing I like about @paperswithcodee is how it automatically surfaces trending benchmarks for me e.g. right now the top benchmark for image understanding is Chartography, which includes 100 professional chart-understanding tasks from @HelloSurgeAI I had no idea this benchmark existed, but both Anthropic and DeepSeek are reporting evals on it for their new models Find it here: https://t.co/pxwrAzgIj0

Before he logs off, @mitchellh sends Codex agents to compare tools and prototype features he’s still figuring out. He picks up from there the next morning. https://t.co/AXVctw8Es5

I trained a specialist decision model for curating web-scale data. It's small (380M, fine-tuned from UltraBERT-380M), so it runs relatively cheaply at scale: 1M FineWeb pages took 19 minutes and ~$5 on @huggingface Jobs. It thinks a quarter of those pages are product pages. 6% passed a simple "educational, not spam" filter, and 1 page in 1,020,000 got "outstanding". I've shared the model, the training data (with the source, labeller, and licence for every row), and the full recipe. I hope more people share how they prepare data for training open decision models! We've had some cool open model releases. Time for some more data releases! https://t.co/IdHCqnmnP1

too many benchmarks are too sanitized. they test if models can solve well-defined problems when you tell them exactly what problem is. but figuring out the problem is half the job and all the fun infusing the chaos of reality into our benchmarks (and figuring out how to grade them nonetheless) is what our Evals team does every day. ~~introducing DAYJOB: TAX, the next in our economically valuable 9 to 5 series~~ one messy example: Anna asks her tax advisor whether to sell her house in Oct or Jan, to optimize her taxes. a good advisor tells her that both dates qualify for a $250K exclusion in capital gains... ...but an even better advisor proactively warns her that she could lose the exclusion if the sale gets inevitably delayed past next August. (Opus mentioned the August deadline 5 out of 5 times. Astra never did.) (maybe your Opus bill is worth it after all!) knowing tax rules isn't enough. great tax agents should give you *advice* too, and have the professional judgment and experience to handle everything you didn't know to ask. we built DAYJOB: TAX (and our entire DAYJOB Knowledge Work series) full of the messy holes of reality that even the smartest agents still don't know how to fill in.

Interesting finding about reasoning models and CoT. They find that reasoning models can control their chain of thought much better than earlier tests suggested. Earlier evaluations put the constraint in the user prompt and found very low controllability. Researchers put the same constraint inside the chain of thought as a self-instruction, and models followed it far more often. A generic reminder to follow the user's constraint also helped a lot. The self-instruction can be triggered from the user prompt alone, so no access to the reasoning channel is needed. Paper: https://t.co/fyagSbVT8M Chat with Paper: https://t.co/90LdW0qwdW

ASPIRE website: https://t.co/99bZhCeszA Open-source codebase: https://t.co/TX2t1ZFwjR

Today we're launching Mouse-1. A fast decision model built specifically for agent harnesses. https://t.co/zbIMGwkztp

Perplexity Flash takes the crown! ⚡ In our tests, it matched Wald's accuracy at nearly twice the speed on a 16 GB Mac mini. Synthetic agent decisions, real email threads, prompt injections: see why Flash is our new pick for email-completion checks. 👇 https://t.co/w7qK7mrGQN

Whether you're pushing LLM training & serving onto NPUs ⚙️, keeping accelerators in sync with PyTorch's release cycle , taming dynamic shapes in torch.compile , or exploring ultra-low-precision training. This agenda of #PyTorchCon has you covered 👇 See you in San Jose! #PyTorch https://t.co/HiMnrCCjsw

+1 more