SDM: Self-Supervised Separation of Camera and Object Motion in Video

by
Table of contents

🌏 閱讀中文版本

In visual representation, the coupling of camera displacement and object motion has long been a source of ambiguity. When the camera pans, background and foreground move together. When an object rotates, the camera may be following it too. This overlap in visual signals makes it difficult for most vision models to distinguish actual motion in the scene from a change in viewing angle.

SDM, or Structured Dynamics Model, proposes a structured approach to this coupling. Rather than directly predicting the next video frame, it uses self-supervised learning to decompose complex dynamics into a primary token and a residual token. The research team reported stronger results on five ProbeMotion tasks, including Kubric synthetic data and SSv2-110k action prediction.

The value of SDM lies in offering an inspectable mechanism for decomposing dynamics, rather than only a numerical improvement. The primary token captures the dominant motion, and its observed role changes with camera movement: in videos with a moving camera, it mainly corresponds to camera motion; in videos with a nearly static camera, it mainly corresponds to object motion. The residual token captures object dynamics outside that dominant component. This is not only an academic challenge. It also creates practical engineering considerations: for downstream tasks such as action recognition or physical reasoning, the interplay between viewpoint changes and object behavior still needs to be validated task by task.

The Challenge and Current State: Coupled Dynamic Signals

Before looking at SDM, it helps to clarify the core limitation facing conventional vision models. Every frame used to train a model to understand video contains two distinct sources of information: motion from the camera itself, such as translation or rotation, and relative motion from objects in the scene. At the pixel level, these two are often tightly intertwined.

For example, when a camera moves to the right, every object in the frame appears to move left. If a model only learns patterns of pixel change, it has difficulty determining whether that movement comes from the camera moving or from every object moving left at once. This ambiguity creates two limitations. First, the model has difficulty learning object features that remain invariant to camera motion. Second, in tasks requiring physical reasoning, it can mistake a viewpoint change for an object’s actual displacement.

Earlier work has often used global pooling or a CLS token to compress information from an entire frame. This works well in low-cost summarization settings, but it also loses spatial detail. A more substantial challenge appears when the camera and objects move at the same time: a single global representation mixes the two dynamics together, leaving dynamic signals entangled in one vector and making causal attribution or physical reasoning difficult.

SDM starts from a direct question: if dominant dynamics can be separated from the remaining dynamics, can a model obtain more robust visual representations? The research team assumes that temporal change in video can be decomposed into a dominant component and a residual component. The role of the dominant component depends on scene conditions: when the camera moves, it is usually camera motion; when the camera is nearly still, it may be object motion. The residual component captures the remaining object dynamics. Through this structured decomposition, the model can organize different motion signals according to the scene configuration.

Core Mechanism: A Two-Track Decomposition of Primary and Residual Motion

SDM’s core mechanism is how it extracts motion information from frozen features and organizes it into two separate tokens. This does not rely on complex end-to-end training. Instead, it is built on a relatively concise and effective architecture.

First, the model uses frozen DINOv2-B/14 as its feature extractor, or encoder. DINOv2 is a widely used pretrained vision Transformer that has learned rich semantic and structural information from large amounts of unlabeled imagery. The research team extracts patch features from all 12 DINOv2 blocks and uses its built-in registers to strengthen spatial position awareness.

Without retraining the underlying model, SDM organizes dynamics into primary and residual tokens. The key is predicting future features and decomposing their dominant and remaining components.

Next comes SDM’s central design: decomposing temporal change into a primary token and a residual token. This is implemented through future-feature prediction. The model is asked to predict features from the next frame, but the prediction target is not the entire image. It is two separate dynamic components.

More specifically, the primary token captures the largest source of temporal change in a scene. In videos with a moving camera, this usually corresponds to camera motion. In videos with a static camera, it corresponds to object motion. The residual token captures the remaining object-related dynamic detail. This design lets the model adapt to scene configuration: when the camera moves, the primary token focuses on viewpoint change; when the camera is fixed, it shifts toward object behavior.

To make this mechanism work, the research team trained on three scene configurations: static scenes with moving cameras, dynamic scenes with static cameras, and scenes where both move. The purpose of this setup is to let the model learn how responsibility is allocated between primary and residual components in different situations. The effects observed so far are limited to the evaluated data and ProbeMotion tasks. For example, in Kubric synthetic data, objects translate without rotating, allowing the model to focus on learning 3D translation vectors [vx, vy, vz]. In real DL3DV data, it must predict 6D camera motion vectors that include yaw, pitch, and roll.

The advantage of this two-track decomposition is that it introduces an explicit structured inductive bias. Conventional models often treat all pixel changes as equally important. By distinguishing dominant dynamics from residual dynamics, SDM encourages the model to learn a more structured representation. It is like separating a main melody, the primary token, from accompaniment, the residual token, when reading a complex score. Each instrument’s role becomes easier to understand.

Evidence and Evaluation: Performance on ProbeMotion

To evaluate SDM, the research team introduced the ProbeMotion benchmark suite, a comprehensive test platform spanning synthetic and real videos. ProbeMotion includes multiple subtasks designed to evaluate a model’s ability to separate dynamics across different scenarios.

SDM won five of seven ProbeMotion tasks. Compared with the base model DeltaTok, SDM achieved its largest gain on the Kubric object-motion task, reducing MSE by 0.16. Under the ProbeMotion and Kubric comparison setup, this result indicates that SDM’s mechanism captures object dynamics more effectively. It is an observation within that specific benchmark, rather than confirmation of the model’s internal capabilities. In ablation experiments, SDM’s MSE objective reduced Kubric object-motion error from 0.30 to 0.19 and camera-motion prediction error from 0.18 to 0.16, compared with an L1 training objective. These improvements in motion-regression metrics indicate that the model can separate and predict different dynamics more precisely.

SDM also performed strongly on SSv2-110k action prediction. Compared with the DeltaTok base model, which uses a global CLS token or mean-pooled features, SDM improved accuracy by 9.6 percentage points. This suggests that the separated primary token can not only capture camera motion but also effectively encode action-related object dynamics, even when those dynamics are not the largest visual change in a scene.

Notably, SDM’s training cost is relatively low. The main experiments used three random seeds, totaling roughly 132 GPU hours, with each seed taking about 11 hours on four GPUs. Nine ablation experiments required roughly 396 GPU hours. Compared with many approaches that require large-scale retraining of vision models, this training scale is practical to verify in engineering work.

For data processing, the research team used a strict filtering strategy. They retained samples with motion magnitude above the 75th percentile and randomly downsampled the remaining lower-motion samples to maintain balance. For the DL3DV dataset, they used a video-level split of 70% training and 30% testing to avoid data leakage. These details are an important foundation for the credibility of the results.

Trade-offs and Boundaries: When Does SDM Work Best?

Every technical approach has an application boundary, and SDM is no exception. Understanding its trade-offs helps determine whether it fits a particular application.

First, SDM’s strength is its ability to separate structured dynamics. That also means its performance may differ from expectations when handling highly unstructured or random noise. Because the model relies on predicting future features and decomposing them into primary and residual components, its performance may be affected when scene changes lack clear physical regularities, such as extensive cluttered background interference. About 90% of Kubric dynamic-object samples passed the validity checks. That figure only describes the filtering result for that sample set; it cannot establish how data quality affects model performance.

Second, SDM training relies on a mix of synthetic data, Kubric, and real data, SSv2 and DL3DV. This strategy covers both synthetic and real videos, but the domain gap remains a transfer boundary that needs evaluation. Synthetic data provides precise motion labels, but may not fully cover real-world complexity. Real data, meanwhile, lacks precise supervision signals. The research team addresses this by retaining high-motion samples and balancing lower-motion samples, but this remains an area that merits continued attention.

Another important trade-off is between computational efficiency and representational capacity. SDM uses frozen DINOv2 features, avoiding the overhead of retraining a large vision model. However, it still requires additional compute to predict and separate primary and residual tokens. Compared with simple global pooling, SDM has a slightly higher inference cost, but provides richer structured information in return. For latency-sensitive applications, this trade-off needs careful evaluation.

Finally, SDM is still at the research stage. Its current application scope is mainly video-dynamics analysis and physical-reasoning tasks. It is not a general-purpose replacement for vision models. Instead, it is a modular component designed to strengthen existing models. This means it needs to be closely integrated with downstream tasks, such as action recognition and scene understanding, to deliver its greatest value.

Conclusion: The Boundaries and Value of Structured Representation

SDM demonstrates the feasibility of structured decomposition on frozen features. For scenarios that require physical reasoning, this separation provides a clearer signal than global pooling. In high-noise or unstructured environments, however, its gains still need further validation. The research offers a design pattern worth considering: transform complex dynamics into manageable components through structured decomposition, rather than relying on a single black-box model.

The value of this structured thinking is not only improved accuracy. It also provides an inspectable representation of dynamics. Under the ProbeMotion and frozen-encoder setup, the primary token mainly captures camera motion in moving-camera videos and object motion in nearly static-camera videos. When the primary component explains camera-induced change, the residual token captures the remaining object dynamics. Whether this division of responsibility improves interpretability for specific downstream tasks, and whether its engineering cost is appropriate, still needs to be validated separately against data, latency, and task requirements. This approach to disentangling complex signals may also apply to other similar temporal-data settings.

Sources