Learning Generative Maps & the Rewards Guiding Them: Normalizing Flow Maps & Surrogate Reward Mixing

Learning a generative model gives us a sampler for a data distribution. Adapting that sampler to a downstream task requires rewarding preferential outcomes typically in the form of scalar rewards or rankings that the model must internalize. In this talk, I will discuss learning generative maps as well as new methodology to exploit (multiple) rewards at inference used to guide them. I will first introduce Normalizing Flow Maps (NFM), a new generative modeling class which combine few-step generation with exact likelihood evaluation by learning invertible transport maps along a generative probability path. NFMs are centered around two matching principles: Pointwise and distributional matching provide that provide complementary training objectives, while access to sample densities enables direct likelihood-based post-training. In the second part of the talk, I will focus on application domains where the objective of interest requires costly human judgments or experimental measurements such that the true oracle reward is unavailable at inference. However, aiding us in these settings is often cheaper reward surrogates capture only parts of of the desired oracle reward. I will introduce MARTINI, which uses limited oracle scores or rankings to learn a composition of these surrogates. The resulting reward can guide generation without further oracle queries. We characterize what information about the oracle can be recovered from the surrogates and, in the small-tilt regime, relate reward approximation error to the discrepancy between the desired and induced distributions. Examples from image generation and post-training, language steering, and molecular modeling will illustrate how the structure of the generative map and the information available about the reward determine the adaptations we can achieve.

Speaker bio

Joey Bose is an Assistant Professor of Computing at Imperial College London, an ELLIS member, and an Affiliate member Mila. He leads the Bose Lab, which builds generative models for science. Previously, he was a Post-Doctoral Fellow at University of Oxford working with Michael Bronstein. He completed his PhD at McGill/Mila under the supervision of Will Hamilton, Gauthier Gidel, and Prakash Panangaden. His research interests span Generative Modelling, Differential Geometry for Machine Learning with a current emphasis on geometric generative models for scientific applications. Previously, he completed his Bachelors and Master’s degrees from the University of Toronto working on adversarial attacks against face detection and is the President and CEO of FaceShield Inc an educational platform for digital privacy for facial data. His work has been featured in Forbes, New York Times, CBC, VentureBeat and other media outlets and was generously supported by the IVADO PhD Fellowship, and NSERC Post-doc Fellowship.

Series: MSR New England Generative Modeling & Sampling Seminar