Computer vision
Teaching machines to interpret and reason about the visual world, from object recognition to medical imaging.
Tool
LLM2CLIP
LLM2CLIP is a novel approach that embraces the power of LLMs to unlock CLIP’s potential. By fine-tuning the LLM in the caption space with contrastive learning, we extract its textual capabilities into the output embeddings,…
Project
Autoregressive Video Models
Driving large video models with next token prediction In-context learning for vision data has been underexplored compared with that in natural language. Previous works studied image in-context learning, urging models to generate a single image…
Publication
Video In-context Learning
Project
A Dynamic Benchmark for Image Understanding
We have created a procedurally generatable, synthetic dataset for testing spatial reasoning, visual prompting, object recognition and detection. A key question for understanding multimodal model performance is how well is can understand images, in particular…