Mage-VL — codec-native streaming multimodal model

microsoft/Mage-VL is a 4B VLM whose visual encoder is trained from scratch to tokenize video the way a codec compresses it: keep every anchor (I) frame patch, keep only the predicted (P) frame patches where the codec spends bits, pack the survivors into canvases. That cuts visual tokens by >75% versus uniform frame sampling. A lightweight cognition gate on top turns the same weights into a proactive streaming model.

Project page · GitHub · Tech report

Preprocessing follows the reference inference.py and inference_streaming.py from the model repo / Mage GitHub. The codec path shells out to cv-preinfer (codec-video-prep) for HEVC bit-cost readiness; the DCVC-RT neural-codec engine is not wired up in this Space.