{
  "type": "tweet",
  "id": "1957139727322411291",
  "url": "https://x.com/sh_reya/status/1957139727322411291",
  "twitterUrl": "https://twitter.com/sh_reya/status/1957139727322411291",
  "text": "We have done 2 cohorts and ~800 people have meaningfully engaged with the AI evals course. Hamel shared a bunch of testimonials with me yesterday. I was really astounded that they are not just generic testimonials; people mentioned very specific results and concepts. It seems like people have been really unblocked and are able to get a lot further in their AI projects, legitimately. E.g.,\n\n\"I often ran into code quality issues when using AI assistants, but I didn’t have a structured way to make sense of them. Before this course, I would just label outputs as “messy code” without really digging into the underlying problems....After this course, I now analyze them systematically across dimensions—things like hardcoded tests, long methods, poor formatting, bad naming, poor architecture choices, duplication, dead code, or ignoring available quality tools.\" - SWE\n\n\"The axial coding just hit different. Before this course, my approach to failures was more of a “vibe investigation,” poking around without a clear structure.After this course, I now cluster failures systematically and trace them back to their core issues... I finally feel like I have a proper way to identify the root problems in my agent instead of just guessing.\" - Staff SWE\n\n\"Some of my favorite highlights: - Build a custom data annotation app! I was so intimidated by this, but I finally made the leap and vibe-coded something out in an afternoon. It has 10x'd my ability to review conversations.\" - Senior PM\n\n\"It's really hard to build LLM judges so be really thoughtful about what you build them for. Often, the biggest impact comes from talking disagreements out and figuring out why there is a disagreement in the first place: are your goals unclear? This seemingly technical course has made me a better PM.\" - Senior PM\n\n\"I have been following Hamel's and Shreya's work for quite some time and it was really awesome to learn from them all the concepts of error analysis, measurement best practices, LLM as Judge + how to make sure it is reliable with human evaluations, collaborative analysis of errors, evaluation of multiturn chats, creation of datasets for CI/CD etc. The last topic on accuracy and cost optimization is really useful as we are seeing in our applications when scaling.\" - Director of ML\n\n\"I was making the common mistake of jumping straight to LLM-as-a-judge without first figuring out what I actually needed to measure. The sections on calibrating LLM judges with TPR/TNR methods were exactly the technical content I was looking for.\" - ML Engineer\n\n\"We kept testing judge models on the same examples we used to build them; obviously they looked great until they hit production.\" - ML Engineer\n\n\"The cost optimization stuff alone could pay for the course (they showed examples of 60%+ savings)\" - ML Engineer",
  "source": "Twitter for iPhone",
  "retweetCount": 5,
  "replyCount": 3,
  "likeCount": 71,
  "quoteCount": 0,
  "viewCount": 7626,
  "createdAt": "Sun Aug 17 17:57:27 +0000 2025",
  "lang": "en",
  "bookmarkCount": 44,
  "isReply": false,
  "inReplyToId": null,
  "conversationId": "1957139727322411291",
  "displayTextRange": [
    0,
    277
  ],
  "inReplyToUserId": null,
  "inReplyToUsername": null,
  "author": {
    "type": "user",
    "userName": "sh_reya",
    "url": "https://x.com/sh_reya",
    "twitterUrl": "https://twitter.com/sh_reya",
    "id": "2286218053",
    "name": "Shreya Shankar",
    "isVerified": false,
    "isBlueVerified": true,
    "verifiedType": null,
    "profilePicture": "https://pbs.twimg.com/profile_images/1342529111839944705/hzr44mb5_normal.jpg",
    "coverPicture": "",
    "description": "",
    "location": "Berkeley, CA",
    "followers": 47577,
    "following": 675,
    "status": "",
    "canDm": true,
    "canMediaTag": true,
    "createdAt": "Sat Jan 11 06:46:16 +0000 2014",
    "entities": {
      "description": {
        "urls": []
      },
      "url": {}
    },
    "fastFollowersCount": 0,
    "favouritesCount": 14364,
    "hasCustomTimelines": true,
    "isTranslator": false,
    "mediaCount": 387,
    "statusesCount": 5393,
    "withheldInCountries": [],
    "affiliatesHighlightedLabel": {},
    "possiblySensitive": false,
    "pinnedTweetIds": [
      "1838617833393283428"
    ],
    "profile_bio": {
      "description": "doing a PhD @Berkeley_EECS, building https://t.co/PmuOqAYt6q | teaching https://t.co/CTWJ6z0JEg | formerly ML eng & undergrad @Stanford CS",
      "entities": {
        "description": {
          "urls": [
            {
              "display_url": "docetl.org",
              "expanded_url": "http://docetl.org",
              "indices": [
                37,
                60
              ],
              "url": "https://t.co/PmuOqAYt6q"
            },
            {
              "display_url": "bit.ly/evals-ai",
              "expanded_url": "http://bit.ly/evals-ai",
              "indices": [
                72,
                95
              ],
              "url": "https://t.co/CTWJ6z0JEg"
            }
          ],
          "user_mentions": [
            {
              "id_str": "0",
              "indices": [
                12,
                26
              ],
              "name": "",
              "screen_name": "Berkeley_EECS"
            },
            {
              "id_str": "0",
              "indices": [
                126,
                135
              ],
              "name": "",
              "screen_name": "Stanford"
            }
          ]
        },
        "url": {
          "urls": [
            {
              "display_url": "sh-reya.com",
              "expanded_url": "http://www.sh-reya.com",
              "indices": [
                0,
                23
              ],
              "url": "https://t.co/j0wChge195"
            }
          ]
        }
      }
    },
    "isAutomated": false,
    "automatedBy": null
  },
  "extendedEntities": {
    "media": [
      {
        "allow_download_status": {
          "allow_download": true
        },
        "display_url": "pic.twitter.com/IZtKRcMhhU",
        "expanded_url": "https://twitter.com/sh_reya/status/1957139727322411291/photo/1",
        "ext_media_availability": {
          "status": "Available"
        },
        "features": {
          "large": {
            "faces": [
              {
                "h": 135,
                "w": 135,
                "x": 1789,
                "y": 423
              }
            ]
          },
          "orig": {
            "faces": [
              {
                "h": 183,
                "w": 183,
                "x": 2421,
                "y": 573
              }
            ]
          }
        },
        "id_str": "1957134866019168256",
        "indices": [
          278,
          301
        ],
        "media_key": "3_1957134866019168256",
        "media_results": {
          "id": "QXBpTWVkaWFSZXN1bHRzOgwAAQoAARspI8qSmrAACgACGykoNm7bcRsAAA==",
          "result": {
            "__typename": "ApiMedia",
            "id": "QXBpTWVkaWE6DAABCgABGykjypKasAAKAAIbKSg2bttxGwAA",
            "media_key": "3_1957134866019168256"
          }
        },
        "media_url_https": "https://pbs.twimg.com/media/GykjypKasAAkIc3.jpg",
        "original_info": {
          "focus_rects": [
            {
              "h": 1344,
              "w": 2400,
              "x": 0,
              "y": 0
            },
            {
              "h": 1344,
              "w": 1344,
              "x": 0,
              "y": 0
            },
            {
              "h": 1344,
              "w": 1179,
              "x": 0,
              "y": 0
            },
            {
              "h": 1344,
              "w": 672,
              "x": 10,
              "y": 0
            },
            {
              "h": 1344,
              "w": 2770,
              "x": 0,
              "y": 0
            }
          ],
          "height": 1344,
          "width": 2770
        },
        "sizes": {
          "large": {
            "h": 994,
            "w": 2048
          }
        },
        "type": "photo",
        "url": "https://t.co/IZtKRcMhhU"
      }
    ]
  },
  "card": null,
  "place": {},
  "entities": {},
  "quoted_tweet": null,
  "retweeted_tweet": null,
  "article": null
}