GMOS: The Trade-offs of Anchoring Moving Object Segmentation with 3D Geometry

by
Table of contents

🌏 閱讀中文版本

When camera motion creates parallax in a 2D image, pixel displacement does not mean an object is truly moving. Likewise, an object that moved somewhere in a sequence is not necessarily in motion in every frame. Traditional methods typically rely on precomputed 2D auxiliary modalities, such as optical flow or point trajectories, to identify pixel motion. But they lack an understanding of scene depth. To address temporal discontinuity, many systems also treat motion as a sequence-level global property, overlooking whether an object is moving or stationary at a particular moment in a single frame.

GMOS (Grounding Moving Object Segmentation in 3D Space and Time) addresses both boundaries. It combines a 3D geometry encoder with segmentation features directly on RGB video. Under the evaluation setup in Tab. 1, the paper reports leading results across the benchmarks it evaluates, while running about three times faster than earlier multi-object MOS methods.

Error Mechanisms: The Limits of 2D Auxiliary Modalities

Traditional MOS designs introduce two concrete sources of error, primarily because of missing geometric information and how they handle time:

The first path comes from missing geometric information. Existing MOS methods commonly rely on precomputed optical flow or point trajectories to determine pixel motion. This 2D information is inherently planar. When a scene contains depth variation, parallax from camera motion can be interpreted as a motion signal even when background objects are stationary. Without a 3D geometric anchor, models can produce false positives more often.

The second path comes from sequence-level handling of time. Many methods treat motion as a sequence-level property. They tend to ask whether an object moves somewhere in the entire video rather than examining its instantaneous state in every frame. This makes start-stop transitions harder to handle precisely, and makes it harder to distinguish brief camera-induced background parallax from genuine object motion.

GMOS attempts to address both bottlenecks. It no longer depends on external 2D auxiliary modalities. Instead, it operates directly on RGB video and anchors motion states with 3D geometric information. It also introduces a finer-grained temporal evaluation standard, MOS-I, which requires the model to identify objects that are moving “right now” in every frame.

Core Mechanism: Combining 3D Geometry and Segmentation Features

GMOS uses a Proposer–Propagator framework. The design breaks the complex task of understanding 3D spatiotemporal motion into two coordinated steps: first, it uses 3D geometry to generate object mask proposals within a single frame; then, it tracks and propagates those proposals over time.

Proposer: Anchoring Space and Semantics

First, the system uses a frozen π3 geometry encoder to process video frames. The encoder takes consecutive frames from a short temporal window, roughly 0.5 seconds, and extracts 3D geometric features and spatial structure from the scene. At the same time, a separate frozen SAM2 segmentation encoder processes the RGB image from the middle frame of that window and extracts high-level semantic segmentation features.

GMOS then uses its Proposer module to fuse these two feature types. Geometry features provide a spatial anchor, while segmentation features provide semantic contours. Through this fusion, the Proposer produces two outputs in one step: object mask proposals and a prediction of each object’s motion state in the current frame, meaning whether it is moving.

Adding geometric information is not simply adding another dimension. It establishes a spatial anchor for segmentation decisions, helping suppress motion predictions caused by parallax. Ablation studies show that removing the π3 geometry encoder substantially reduces the model’s grounding capability and produces many hallucinated false positives.

During training, GMOS uses a frame-level mask loss that combines Focal Loss (α = 0.25, γ = 2) and Dice Loss at a 20:1 weighting. This design handles the class-distribution difference between foreground and background while improving segmentation precision in overlapping regions.

Propagator: Temporal Continuity

After generating single-frame proposals, GMOS enters the Propagator stage. This stage operates on SAM2 state S. It maintains a set of prompts and outputs frame-level masks M̃t and motion labels m̃t. The design aims to maintain continuity over time.

GMOS defines several key thresholds to manage the lifecycle of trajectories:

  • τm = 0.5: When the motion score exceeds this value, the object is considered to be moving.
  • τu = 0.7: When a proposal’s predicted IoU exceeds this value, the mask is trusted.
  • τmatch = 0.95: The threshold used for trajectory matching.
  • τnew = 0.3: When the maximum mask-overlap precision between a new proposal and existing trajectories (maxjPRECISION(M̂t(i),M̃t(j))) is below this value, the system seeds a new trajectory.

GMOS also provides a variant called GMOS-S. For scenarios requiring faster inference, GMOS-S removes the Transformer decoder and learnable object queries. It also does not use the propagator. Instead, it directly predicts a single foreground-background binary mask through an upsampling decoder, enabling faster streaming inference. This design fits workflows that prioritize streaming speed and do not need multi-object distinction, while trading away the ability to distinguish multiple objects.

Evaluation Protocol and Performance

One of GMOS’s contributions is a redefinition of how moving object segmentation is evaluated: MOS-I (Instantaneous), a frame-level protocol for instantaneous motion evaluation.

MOS-I requires models to predict precisely which objects are moving “right now” in every frame. It scores only those frames and penalizes false predictions on stationary backgrounds. This fine-grained evaluation requires higher spatiotemporal resolution instead of relying on sequence-level inertia. MOS-I is demanding because it removes the shortcut of sequence inertia and requires a frame-by-frame output of the current motion state.

To support this evaluation, the research team built the GMOS-2K dataset. It contains 2,210 real-world videos. Its labels come from five existing video object segmentation (VOS) benchmarks, with detailed annotations for every object’s temporal motion state. From 5,001 initial candidate videos, the first filtering stage excluded 2,791, leaving 2,210 videos for the final dataset. The paper separately reports an overall training-data scale of 14,171 videos, 782,128 frame annotations, and 34,506 objects.

For performance, the paper reports that GMOS reaches leading levels in its MOS, MOS-I, and unsupervised VOS (UVOS) evaluations. Its speed result is bounded by the Tab. 1 evaluation setup: compared with the earlier multi-object MOS methods in that table, GMOS is about three times faster. The available material does not provide the names of specific optical-flow baselines, same-condition metrics, or differences. This result therefore should not be generalized into an advantage across all optical-flow methods, hardware, or deployment conditions.

Boundaries and Tradeoffs

Although the paper reports leading results on the benchmarks it evaluates, its technical approach has clear tradeoffs and applicability boundaries:

  1. Geometric information vs. computational complexity: GMOS’s core advantage comes from using 3D geometric information to handle scenes with complex parallax or depth variation. However, introducing the π3 geometry encoder increases computational demand. For severely resource-constrained devices, GMOS-S offers faster inference as an alternative by trading away multi-object distinction.

  2. Limits in spatial resolution: GMOS has room to improve when detecting fine-grained motion. Because the π3 geometry backbone has limited spatial resolution, the model is less reliable at identifying small moving objects in cluttered scenes, such as the missing small background objects shown in Fig. G6. In addition, strong reflections or extreme depth parallax can shift the model’s motion-state predictions.

  3. The demanding evaluation protocol: MOS-I places high requirements on dataset quality because it scores only the frames in which an object is actually moving. Frame-level motion labels increase annotation granularity. It follows that scoring is more sensitive to the accuracy of frame-level motion-state annotations. In scenes where stationary objects shift because of camera motion, the model faces a stricter penalty.

Conclusion

By anchoring 3D space and time directly in RGB video, GMOS provides a transition path for areas where traditional MOS methods can further strengthen geometric information and temporal resolution. While the model still has room to improve in extreme-parallax settings and small-object detection, its Proposer–Propagator architecture and MOS-I protocol demonstrate potential for efficiently handling multiple objects and frame-level motion states under the evaluation conditions described in the paper. This observation remains bounded by the compared methods, hardware, and scene conditions.

GMOS provides a path from 2D optical flow toward 3D geometric anchoring. In resource-constrained edge scenarios, whether the GMOS-S tradeoff is sufficient to replace traditional methods depends on the application’s criteria for multi-object distinction, inference latency, spatial resolution, and conditions such as reflections, extreme parallax, and cluttered backgrounds. This is a direction worth evaluating against your system’s own boundaries.

Sources