multimodal vision encoding

Why vision encoders still matter even when AI models skip pretraining

Separate pretraining for vision encoders is optional.

3 min readMachine Learning
Why vision encoders still matter even when AI models skip pretraining
[Meta FAIR + NYU] Unbundling Multimodal Vision Encoding: training 200+ MLLMs to find encoder pretraining is optional but vision-only parameters are not [R]

The debate over vision encoders in multimodal AI has been unsettled for too long, with major labs pulling in opposite directions. A new systematic study from the Unbundling team finally cuts through the noise with a clear, evidence-based answer: vision needs its own parameters, but it does not need its own pretraining or its own separate encoder. This is a genuinely useful finding for anyone building or deploying multimodal systems, and it arrives at a moment when the field is hungry for practical clarity rather than another architecture manifesto.

What makes this work stand out is its discipline. The team trained over 200 models from scratch on identical data, counting encoder pretraining against the same compute budget as everything else. That controlled setup lets them isolate three choices that a standard encoder bundles together: separate pretraining, dedicated vision-only parameters, and placement before the LLM. The results are refreshingly direct. Separate pretraining, whether contrastive, self-supervised, or caption-based, at best ties an encoder trained from scratch alongside the LLM. Even free pretraining failed to improve vision QA. Vision-only parameters, however, are non-negotiable. Encoder-free models trail at every size and need roughly three times the compute to catch up. Placement is flexible: an "in-fusion" approach that adds zero-initialized vision-only weights to the first eight LLM layers matches the traditional encoder while cutting time-to-first-token by about 13 percent.

For our readers building practical AI tools, the takeaway is immediate. You do not need to maintain a separate vision encoder pipeline with its own pretraining schedule. You do need to reserve dedicated parameters for vision inside your model. This aligns with recent work on How minimalist AI agents can make spreadsheets smarter, where task-specific parameter allocation proved more important than massive pretrained backbones. It also echoes lessons from Predicting touch beats reconstructing it for robotic skill learning, where the right inductive bias in parameter design outperformed generic reconstruction objectives. The pattern is consistent: architectural choices that seem fundamental often turn out to be negotiable, while the allocation of parameters to the right modality is not.

One specific consequence deserves attention. The in-fusion approach achieves comparable vision performance with lower latency, which matters for any application where users wait for a response. If you are embedding vision into a spreadsheet assistant or a document analysis tool, those milliseconds add up. The study also leaves an open question about scaling: the in-fusion results hold from 0.6B to 6.4B parameters, but what happens at the frontier sizes where pretrained encoders are most heavily invested? That is the next experiment worth watching, and the Unbundling team has already shown they know how to run it cleanly.

From Machine Learning

Paper: https://www.alphaxiv.org/abs/2611.17334

Blog and FAQ: https://unbundling.github.io

Read the original at Machine Learning