@omarsar0
Improving Mathematical Reasoning in Open LLMs One of my favorite papers this past week was the work of Shao et al. who introduced a 7B model called DeepSeekMath for improving mathematical reasoning of LLMs. TLDR: - continues pretraining a code base model with 120B math-related tokens - introduces GRPO (a variant of PPO) to enhance mathematical reasoning and reduce training resources via a memory usage optimization scheme - DeepSeekMath 7B achieves 51.7% on MATH which approaches the performance level of Gemini-Ultra (53.2%) and GPT-4 (52.9%) -when self-consistency is used the performance improves to 60.9%. What's outstanding about this specialized model is that it can reach a performance that's competitive with general-purpose models (Gemini Ultra and GPT-4) and even large specialized models (Minerva 540B). It focuses on pre-training with math information at scale (120B tokens), so there is an element of optimizing the data selection pipeline that leads to huge gains. Code training is also key to enhancing mathematical reasoning capabilities. Another aspect of this paper that I like is the introduction of an efficient PPO variant that hints at more performant and efficient ways to leverage reinforcement learning for LLMs. Lots of discussions, analysis, and exploration about RL in the paper.