@winglian
If you're using LoRA for RL, it's most likely best to deploy your adapter unmerged. Depending on your learning rate, how many effective train steps, etc, you might be losing much of your training updates to rounding to BF16 when you merge the adapter to the base model. https://t.co/xnqBTHVzOW