@wandb
Shared prefix is the big win, but not the only one. Maxing out packed sequence length and tuning the parallelism setup got us further, up to 35 trajectories/s on 4 GPUs. We published the configs so you can use them as a starting point for your own runs. 🫡 https://t.co/S7jomILSRi