@OpenBMB
What makes preference data truly effective for LLM alignment? 🤔 Introducing AIR: A systematic framework that deconstructs preference datasets into 3 core components (Annotations, Instructions, Response Pairs) and reveals evidence-based optimization principles. No more trial and error! 🤗 Paper: https://t.co/dFxRHSux2W 📄 arXiv: https://t.co/QJl1T1uGyi Why it matters: 1️⃣ Simplicity Wins in Annotations: Basic point-wise scoring with generative models (like Llama-3.1-70B-Instruct) + greedy decoding outperforms complex methods. Less is more—excessive design introduces noise rather than clarity. 2️⃣ Smart Instruction Filtering: Select instructions with low response variance across LLMs. This forces models to learn fine-grained preferences (like logical rigor) rather than relying on obvious differences. 3️⃣ Balanced Response Pairs: Optimal pairs combine moderate score gaps (Δ=2-3), high absolute quality (≥8), and 1:1 On/Off-Policy mixing—achieving clear contrast without overfitting. The results? +5.3 average gain across 6 benchmarks (WildBench, Arenahard, etc) with just 14k curated pairs from 17 open-source LLMs covering coding, math, and chat tasks. AIR transforms preference learning from "scale blindly" to component-aware design—a blueprint for building smarter, more aligned AI systems. 🚀 #AI #LLM #RLHF #PreferenceLearning #Alignment
