{
  "type": "tweet",
  "id": "2112648736484806847",
  "url": "https://x.com/sayashk/status/2112648736484806847",
  "twitterUrl": "https://twitter.com/sayashk/status/2112648736484806847",
  "text": "An update on our evaluations of AI agents conducting open-ended AI research.\n\nMost evaluations of agents conducting AI research focus on narrow, verifiable tasks. But AI research is often open ended. In July we tested the ability of frontier AI agents to conduct open-ended AI research, using Claude Opus 4.8 and GPT-5.6 Sol.\n\nWe gave agents research questions from two unpublished papers and thousands of dollars of API credits and compute. The authors of the original papers reviewed the AI-generated papers, following a NeurIPS-style template, and rejected both papers.\n\nSince then, OpenAI has released GPT-6 Astra and Anthropic has released Fable 5.1. Can these new models conduct open-ended research with a $10,000 budget?\n\nIn follow-up experiments on two papers (one with Fable 5.1 and another with GPT-6 Astra), we observed meaningful improvements to the agents' capabilities at experimental design and resource management.\n\nBut we also saw continued limitations in creative problem solving, judgment, calibration, and communication. As a result, the expert reviewers still unambiguously rejected the agents' papers. \n\nWe plan to continue running these evaluations with better models, better scaffolds, and higher budgets. If you are interested in long-running agent evaluations, reach out!",
  "source": "",
  "retweetCount": 10,
  "replyCount": 12,
  "likeCount": 51,
  "quoteCount": 3,
  "viewCount": 3138,
  "createdAt": "Tue Oct 20 20:54:44 +0000 2026",
  "lang": "en",
  "bookmarkCount": 31,
  "isReply": false,
  "inReplyToId": null,
  "conversationId": "2112648736484806847",
  "displayTextRange": [
    0,
    276
  ],
  "inReplyToUserId": null,
  "inReplyToUsername": null,
  "author": {
    "type": "user",
    "userName": "sayashk",
    "url": "https://x.com/sayashk",
    "twitterUrl": "https://twitter.com/sayashk",
    "id": "3084274082",
    "name": "Sayash Kapoor",
    "isVerified": false,
    "isBlueVerified": true,
    "verifiedType": null,
    "profilePicture": "https://pbs.twimg.com/profile_images/1521238232867946496/U_GCI43e_normal.jpg",
    "coverPicture": "https://pbs.twimg.com/profile_banners/3084274082/1712802699",
    "description": "",
    "location": "Princeton",
    "followers": 15989,
    "following": 2497,
    "status": "",
    "canDm": true,
    "canMediaTag": true,
    "createdAt": "Sun Mar 15 09:03:24 +0000 2015",
    "entities": {
      "description": {
        "urls": []
      },
      "url": {}
    },
    "fastFollowersCount": 0,
    "favouritesCount": 5953,
    "hasCustomTimelines": true,
    "isTranslator": false,
    "mediaCount": 248,
    "statusesCount": 1678,
    "withheldInCountries": [],
    "affiliatesHighlightedLabel": {},
    "possiblySensitive": false,
    "pinnedTweetIds": [
      "2108744168222969911"
    ],
    "profile_bio": {
      "description": "Incoming prof @UCBerkeley\nAI agents, policy, evals, AI for science\nEssay/newsletter (AI as Normal Technology): https://t.co/5amOkqLb4A\nBook (AI Snake Oil): https://t.co/DabpkhNZ2k",
      "entities": {
        "description": {
          "urls": [
            {
              "display_url": "bit.ly/ai-nt",
              "expanded_url": "http://bit.ly/ai-nt",
              "indices": [
                111,
                134
              ],
              "url": "https://t.co/5amOkqLb4A"
            },
            {
              "display_url": "bit.ly/ai-so",
              "expanded_url": "http://bit.ly/ai-so",
              "indices": [
                156,
                179
              ],
              "url": "https://t.co/DabpkhNZ2k"
            }
          ],
          "user_mentions": [
            {
              "id_str": "",
              "indices": [
                14,
                25
              ],
              "name": "",
              "screen_name": "UCBerkeley"
            }
          ]
        },
        "url": {
          "urls": [
            {
              "display_url": "cs.princeton.edu/~sayashk",
              "expanded_url": "http://cs.princeton.edu/~sayashk",
              "indices": [
                0,
                23
              ],
              "url": "https://t.co/wO9NAWByQN"
            }
          ]
        }
      }
    },
    "isAutomated": false,
    "automatedBy": null
  },
  "extendedEntities": {
    "media": [
      {
        "allow_download_status": {
          "allow_download": true
        },
        "display_url": "pic.x.com/St0pxGXDul",
        "expanded_url": "https://x.com/sayashk/status/2112648736484806847/photo/1",
        "ext_alt_text": "screenshot of the landing page of the research note introducing these results",
        "ext_master_playlist_only": [],
        "ext_media_availability": {
          "status": "Available"
        },
        "ext_playlists": [],
        "features": {
          "large": {
            "faces": []
          },
          "orig": {
            "faces": []
          }
        },
        "id_str": "2112648536638529536",
        "indices": [
          277,
          300
        ],
        "media_key": "3_2112648536638529536",
        "media_results": {
          "id": "QXBpTWVkaWFSZXN1bHRzOgwAAQoAAR1RoqaZlqAACgACHVGi1SFa0L8AAA==",
          "result": {
            "__typename": "ApiMedia",
            "id": "QXBpTWVkaWE6DAABCgABHVGippmWoAAKAAIdUaLVIVrQvwAA",
            "media_key": "3_2112648536638529536"
          }
        },
        "media_url_https": "https://pbs.twimg.com/media/HVGippmWoAAEWgj.jpg",
        "original_info": {
          "focus_rects": [
            {
              "h": 608,
              "w": 1086,
              "x": 0,
              "y": 840
            },
            {
              "h": 1086,
              "w": 1086,
              "x": 0,
              "y": 362
            },
            {
              "h": 1238,
              "w": 1086,
              "x": 0,
              "y": 210
            },
            {
              "h": 1448,
              "w": 724,
              "x": 181,
              "y": 0
            },
            {
              "h": 1448,
              "w": 1086,
              "x": 0,
              "y": 0
            }
          ],
          "height": 1448,
          "width": 1086
        },
        "sizes": {
          "large": {
            "h": 1448,
            "w": 1086
          }
        },
        "type": "photo",
        "url": "https://t.co/St0pxGXDul"
      }
    ]
  },
  "card": null,
  "place": {},
  "entities": {
    "hashtags": [],
    "symbols": [],
    "urls": [],
    "user_mentions": []
  },
  "quoted_tweet": null,
  "retweeted_tweet": null,
  "isLimitedReply": false,
  "communityInfo": null,
  "article": null
}