MIT News AI· Lauren Hinkel | MIT-IBM Computing Research Lab·· 1 hr agoAI score28
Q&A: Enabling more human-like interactions and understanding in multimodal foundation models
Q&A: Enabling more human-like interactions and understanding in multimodal foundation models
AI summary
An MIT-IBM Computing Research Lab team co-led by Jim Glass investigates critical failures of state-of-the-art multimodal models and develops benchmarks and data-efficient methods to address them with limited supervision. The work includes IPLoc, which uses pseudonym substitution to force models to attend to visual context, and OmniR1, which found audio-language models can answer multiple-choice benchmarks without listening to the audio. The team plans to shift focus toward full-duplex spoken dialogue systems while continuing joint audio-visual work.
Source: MIT News AI · news.mit.edu