@DrJimFan
Introducing NitroGen, an open-source foundation model trained to play 1000+ games: RPG, platformer, battle royale, racing, 2D, 3D, you name it! We are on a quest for general-purpose embodied agents that master not only the real world physics, but also all possible physics across a multiverse of simulations. We found that our GR00T N1.5 architecture, originally designed for robotics, can be adapted easily to play lots of games with wildly different mechanics. Our recipe is simple and bitter lesson-pilled: (1) a 40K+ hour, high-quality dataset of public in-the-wild gameplay; (2) a highly capable foundation model for continuous motor control; (3) a Gym API that wraps any game binary to run rollouts. Our data curation is a lot of fun: it turns out that gamers love to show off their skills by overlaying real-time gamepad control on a video stream. So we train a segmentation model to detect and extract those gamepad displays and turn them into expert actions. We then mask out that region to prevent the model from exploiting a shortcut. During training, a variant of GR00T N1.5 learns to map from 40K hours of pixels to actions through diffusion transformers. NitroGen is only the beginning, and there's a long way to hill-climb on the capability. We intentionally focus only on the System 1 side: the "gamer instinct" of fast motor control. We open-source *everything* for you to tinker: pretrained model weights, the entire action dataset, code, and a whitepaper with solid details. Today, robotics is a superset of hard AI problems. Tomorrow, it might become a subset, a dot in the much larger latent space of embodied AGI. Then you just prompt and "ask for" a robot controller. That might be the end game (pun intended). NitroGen is co-led by our brilliant minds: Loic Magne, Anas Awadalla, Guanzhi Wang. It's a multi-institutional collaboration. Check out Guanzhi's technical deep dive thread and repo links below!