@RishiBommasani
We are really excited to build independent evaluations of RSI progress and are actively hiring. We constantly see conflicting evidence about RSI progress. Most hinges on information coming from labs. To illustrate this, let's just look at evidence discussed *this week* that conflicts in whether it suggests RSI progress is fast or slow. https://t.co/F4vi5Z3fch Results from OpenAI and Google evals indicate the time horizons for R&D tasks being automated are around ~15 minutes, which are much slower than METR time horizons for software tasks. This suggests we should have *slower* RSI timelines. https://t.co/RfFobR22bw Results from Mirendil indicate that pretraining at a certain scale can be heavily automated such that it can be managed by a single pretraining researcher. This suggests we should have *faster* RSI timelines. https://t.co/lIYID0eFII Results from Nvidia indicate that frontier models provide significant uplift for CUDA optimization. This suggests we should have *faster* RSI timelines. Each piece of evidence is more complex than what I write here. Interpreting them all together is confusing even if you trusted them all fully. And factoring in the underlying level of credibility is basically impossible: each result is basically just some plots and numbers selectively disclosed by labs in a tweet/talk/report. So deep sense-making about RSI progress is very hard. If you are interested in building independent RSI evals, apply here: https://t.co/w6CfTsvJfI