@PyTorch
We’re excited to share In-Kernel Broadcast Optimization (IKBO)—a kernel-model-system co-design approach driving @Meta's large-scale recommendation system (RecSys) inference. It serves as the scalability backbone for the request-centric, inference-efficient framework underlying the Meta Adaptive Ranking Model (serving LLM-scale models in production). The core idea is simple but powerful: broadcast is a data layout concern, not a computational necessity. Instead of materializing broadcast at the service layer, IKBO pushes it all the way down into the kernel layer—precisely where request and candidate features interact. This enables the system, together with model co-design, to capture the full performance upside. By preserving request and candidate inputs in their native batch sizes and fusing broadcast directly into the kernel, IKBO eliminates replicated memory movement and redundant computation across the stack. From Attention in sequence modeling to GEMM in traditional workloads, the same idea delivers broad inference gains. Results from our co-designed deployments: ⚡ Up to a 2/3 reduction in compute-intensive net latency on co-designed models. ⚡ End-to-end deployment across Meta's RecSys stacks on both GPU and MTIA (Meta's in-house accelerator). ⚡ Up to 4x speedups for linear compression GEMM and up to 2.4x/6.4x (kernel only / kernel + broadcasting) throughput for IKBO attention compared to FlashAttention-4 on H100. 🔗 If you’re interested in how rethinking a primitive like broadcast can unlock system-level gains for modern AI inference, take a look: https://t.co/mdl0F1Zj7E By @jiao_jian62798, @liptds_11, Hongtao Yu, Yuanwei Fang, Zhengkai Zhang, Zoran Zhao, Yuxuan Chen, Sijia Chen, Yang Chen #PyTorch #OpenSourceAI #RecSys #AIInfrastructure
