Multimodal AI needs a plan for missing inputs
A paper on missing arbitrary modalities argues that multimodal systems should train sensors and data streams to teach each other, not just fuse them, because real deployments often lose inputs at inference time through failures, privacy limits, or workflow gaps.
TL;DR: If your multimodal model only works when every input shows up, you do not have a deployment-ready system, you have a demo with fragile assumptions.
What happens when the camera, sensor, or private field disappears?
The primary source here is “Co-Learning for Missing Arbitrary Modalities in Multi-modal Classification,” posted to arXiv cs.AI and cs.LG, with code at https://github.com/fmenat/Co4Miss. The paper targets a very practical failure mode: training with several modalities, then running inference when some of them are missing.
That can happen for boring reasons. A sensor fails. A hospital system withholds a data field because of privacy rules. A user uploads text but not an image. A fleet device has audio disabled. A customer workflow changes. None of this is exotic.
The paper calls this “missing arbitrary modalities.” That phrase matters. A lot of earlier work, according to the arXiv paper, has focused on bimodal setups or known missing patterns. Think image plus text, with one missing in predictable ways. This work asks for a harder promise: any subset can disappear at inference time, without assuming the missing pattern ahead of time.
That is closer to the mess builders see in production.
The usual instinct is fusion. Take all the available inputs, combine them in a smarter way, and hope the model degrades gracefully. The paper argues for a different center of gravity: co-learning. Instead of treating modalities mainly as ingredients to be fused, train them to collaborate so each one carries more useful signal when its peers are absent.
Is co-learning better than fusion?
The useful distinction is not “fusion bad, co-learning good.” It is more specific.
Fusion assumes the system’s strength comes from combining signals at prediction time. Co-learning tries to make each modality learn from the others during training. In plain English: if you have audio, video, text, and sensor readings during training, the model should use that period to transfer structure across them, so one remaining stream is less helpless later.
“Co-Learning for Missing Arbitrary Modalities in Multi-modal Classification” introduces two approaches using information at the feature level and decision level. The paper reports significant resilience gains across two multimodal classification benchmarks under different missing-modality conditions.
The split in results is the practical bit. The first method behaves better when missingness is mild, such as one modality being absent. The second performs better in extreme cases, where all but one modality are missing.
That is not a universal win claim. It is a design cue. If your production system usually loses one input at a time, your training strategy may differ from a system that must still classify from a single surviving signal. A warehouse robot with occasional camera dropout is not the same problem as a clinical model that often receives only a partial record.
What should builders take from this?
The big mistake is evaluating multimodal models only in the “all inputs present” condition. That benchmark is comfortable. It is also incomplete.
If you are building with multiple data streams, missingness should be part of the eval plan from day one. Test every single-missing case. Test all-but-one cases. Test realistic privacy-driven removal, not just random masking. Track which modality carries the system when everything else falls away.
The paper’s framing also pushes teams to think about training data differently. If you have rich multimodal training data, do not only ask, “How do I combine these signals?” Ask, “What can each signal teach the others before deployment breaks the perfect-input assumption?”
That applies outside academic classification benchmarks. Customer support systems may combine chat text, screenshots, account metadata, and event logs. Manufacturing systems may combine vibration, thermal, image, and operator notes. Healthcare workflows may combine imaging, labs, notes, and demographics, with access varying by institution and consent. In all of these, the model’s worst day is more important than its clean-room score.
Practitioner’s Take: before adding another modality, run a missing-input audit. List every input your model expects, then remove them one at a time and in harsh combinations. If performance collapses, try training regimes where modalities teach each other instead of only meeting at the fusion layer. The catch most teams miss: the best multimodal system is not the one with the most inputs, it is the one that still behaves sensibly when the real world takes some away.