Physics-Guided Vision-Language World Models for Agentic 4D Scene Understanding

This project develops a unified framework for physically grounded world modelling that combines video-based temporal prediction with Gaussian Splatting for photorealistic 3D representation. A Physics Vision-Language Model translates natural-language instructions into transformations that respect physical constraints, enabling interpretable and goal-directed control in dynamic scenes. By integrating perception, prediction, and action in a Vision-Language-Action loop, the research aims to advance agentic AI systems capable of transparent, physics-aware reasoning—supporting applications in robotics, simulation, and education.

This research is conducted via The Agentic AI Research and Innovation (AARI) Initiative which focuses on the next frontier of agentic systems through Grand Challenges with the academic community and Microsoft Research.