@danielrupawalla
i've found defects in ~65% of the TAU-banking tasks. these are tasks where the policy isn't enforced correctly. for example, if there are rules the model can violate and not get penalized for (transferring to human reqs, tasks where the tool call can fail and the model is still rewarded, etc). some grading may be directionally correct, but still contains noticeable defects that needs to be hardened prior to use. high-level defects for people working on benchmarks right now: > models are capable of taking many out of scope actions without being penalized (only tool-call checks, no db) > models can directly lie to the customer with no penalty (no communication checks, which for a customer service benchmark feels important, though i didn't label tasks that only fail this as p0) > many of the tool call checks are naive and if the tool call fails there are no repercussions. i.e. a test of email change may just check if the tool was called, with no field checks or follow through check, so if the tool call fails or if i change the wrong email it may still pass. to expound a bit more: (1) i'm going to try and go through every public benchmark. the reason for these audits is that i've become frustrated with people using defective benchmarks to choose what type of data they want to purchase and data co's making flawed data to help a model hill-climb a benchmark. it is very clear that bad RL data is a contributor to reward hacking, and the benchmarks that inform purchasing decisions lie upstream of that, thus handing a huge power to companies that publish benchmarks. (2) sierra and the TAU series have had tremendous contributions to tool-calling evals. tau-bench is actually what got me into RL data in the first place more than two years ago. this is in no means to discredit sierra and the effort they've put into creating the benchmark. big inspiration! (3) please reach out and correct me if i'm wrong on any of these asap! i want to give sierra credit where it is due.