🌏 閱讀中文版本
Why Can’t Your Video Model Understand Physics?
Open your prompt testing interface.
Drag in a hand-drawn sketch: a ball, a ramp, a few lines representing obstacles.
You think it is a simple physics-simulation test.
Three seconds later, the model returns an answer, but the ball passes through a wall.
That is when you notice it: the representation gap is one contributing factor. The issue is not that the model has no capability at all. Abstract inputs raise the bar for interpretation.
In its paper, Visual prompt engineering for video models, DeepMind highlights a counterintuitive finding: when video models handle visual-reasoning tasks, they show a much stronger preference for photorealistic scenes than abstract sketches.
Note: this paper is an arXiv preprint submitted in July 2026 and has not yet completed peer review.
By automatically converting input images into a photorealistic style through VIPE, Veo 3.1 accuracy on the VPCT benchmark rose from 41.3% to 59.3%, while Omni Flash improved from 56.3% to 67.5%. Under these test conditions, this showed better cost efficiency than simply adding more text prompting or test-time scaling.
The Key Difference: Abstract Sketches Raise the Representation Barrier
We are used to thinking that LLMs need strong text prompts to produce more accurate answers.
For video models, we rely on visual prompts in the same way.
But the conventional approach is to send the original image directly to the model, whether it is a photograph or a simple sketch.
The paper identifies a frequently overlooked variable: when the input is an abstract sketch, scene consistency drops sharply.
In abstract settings, objects may appear, disappear, or deform at random. In photorealistic scenes, physical rules are easier for the model to capture. This suggests that many benchmarks, including VPCT, may be measuring both abstract-image interpretation and physical reasoning. When a benchmark uses a simple-sketch style, it is testing physics, but also a kind of visual “translation” ability.
This points to an unconfirmed hypothesis: a mismatch in input distribution. The model is expected to understand the physical world, yet it receives an abstract diagram. Is this because the model has limited abstract-understanding capability, or simply because of differences in training-data distribution? The paper also notes that the underlying mechanism still needs further ablation experiments to clarify.
The Core Mechanism: How Does VIPE Work?
VIPE is a task-oriented visual-input editing framework. It does not modify model weights or alter text prompts. Instead, it automatically optimizes the task image before inference.
The flow is straightforward:
Take a sketch → send it to an image editor for photorealistic conversion → feed the resulting image to the video model.
VIPE includes multiple editing strategies, such as free-form style transfer and Atomic Concept Editing (ACE). Photorealization is the most common category, but not the only one. It generates multiple candidate edits, for example with different editing strengths or random seeds, then uses a task-oriented selection step to form the visual input that is ultimately sent to the video model. The evidence cited here does not establish how candidates are scored, selected, or combined, so this step should not be interpreted as a publicly specified fixed-ranking mechanism.
Where It Intervenes: Not the Prompt, the Input
One point needs to be clear:
VIPE does not tell the model, in text, “please imagine this is the real world.”
It directly replaces abstract pixel structures with photorealistic pixel structures at the visual-input layer. This is input preprocessing outside the model.
It may look like one extra filter. But the key is that it changes the data distribution the model sees, not the instruction.
Key Insight: VIPE’s test results offer an observation: rather than spending effort explaining in text that “this is the real world,” it may be more effective to make the image look like the real world directly.
Back to the Betting Table: VIPE’s Cost Efficiency
Let’s look at the numbers. These results come from DeepMind’s tests on VPCT, the Visual Physics Comprehension Test. VPCT primarily evaluates a model’s physical intuition around object motion, collisions, and gravity.
1. A Clear Increase in Accuracy
| Model | Baseline Accuracy | Accuracy with VIPE | Improvement |
|---|---|---|---|
| Veo 3.1 | 41.3% | 59.3% | +18.0 percentage points (about 43.6% relative improvement) |
| Omni Flash | 56.3% | 67.5% | +11.2 percentage points (about 19.9% relative improvement) |
Both results suggest that even stronger-performing models still have substantial potential to unlock when working with abstract inputs.
Note: data from Table 1 of the paper, based on mean accuracy on the VPCT benchmark.
2. Cost Efficiency: Image Editing vs. Video Generation
In this test, VIPE appears to be a cost-efficient test-time scaling strategy.
- Baseline video-generation cost: Veo 3.1 costs approximately USD 0.40 per second, or about USD 3.20 for an 8-second video. Pricing is based on Gemini API pricing at the time of publication.
- VIPE preprocessing cost: Five image edits cost approximately USD 0.40 in total.
- Full VIPE inference cost: Under the specific test conditions, including five preprocessing runs and one 8-second video generation, the total was approximately USD 3.60.
This means that, under these test conditions, optimizing the visual input first may be more cost-efficient than increasing video sampling. Increasing sampling and optimizing inputs are two strategies that can work in sequence. Which one to emphasize depends on your latency budget and compute constraints.
3. An Advantage Over Text Prompting
The paper compares VIPE with conventional text prompt engineering. The results show that for visual-reasoning tasks, VIPE generally performs better than optimizing text prompts alone. This supports a hypothesis: for the models and tasks in this test, visual-input quality matters more than fine-grained adjustments to text instructions.
Your Decision: Fit Boundaries and Trade-offs
VIPE works around the representation gap through external preprocessing. It does not directly extend the model’s native ceiling for physical reasoning. Its fit boundaries and trade-offs are clear.
Who Is It For?
- Applications that process abstract or sketch-like scenes: such as educational software, physics teaching tools, and prototype testing in game development. This is a direction worth testing, provided semantic fidelity and latency costs are evaluated first.
- Teams seeking high inference accuracy with limited budgets: VIPE provides a lower-cost path to improved performance.
- Developers building on existing video models: no retraining is needed; add an image-editing step before inference.
Which Situations Are Less Suitable?
- Native Image Generation Models: The paper indicates that VIPE did not systematically improve reasoning performance for native image-generation models. The tested image models did not show consistent gains, and differences between model categories still need broader experiments to confirm.
- Scenarios requiring extremely low latency: Image editing has a low cost, but it still adds a compute step. If latency is the primary consideration, assess the bottlenecks across the entire pipeline.
The Trade-offs: What Does It Cost?
Dependence on image-editor quality: VIPE’s results depend on whether the underlying image-editing model can accurately generate a photorealistic scene.
Potential bias introduced by added detail: A photorealistic scene may introduce details absent from the original sketch, such as lighting or texture. Could these details lead the model toward new misjudgments? Current research shows an overall benefit, but this needs continued monitoring.
Key Insight: VIPE’s test results reveal a new optimization interface that can be verified: visual prompts are no longer fixed inputs, but an optimizable space. This resembles the evolution of prompt engineering for LLMs, from handwritten prompts toward automated optimization.
Conclusion: Decisions Within the Evidence Boundary
If your workflow relies heavily on sketches or abstract inputs, VIPE offers a direction for improving physical-reasoning accuracy through preprocessing, without retraining the model.
But this direction has evidence boundaries. Current test data is concentrated on the VPCT benchmark and specific video-generation models, including Veo 3.1 and Omni Flash. Before using it in production, validate semantic fidelity, added latency, and fit across different model architectures for your own use case. Increasing sampling and optimizing inputs are two strategies that can work in sequence; their relative weight depends on the system’s latency budget and compute constraints.
Sources
- Visual prompt engineering for video models — Defines the VIPE mechanism and reports VPCT benchmark results
- Gemini API Pricing — Official Veo 3.1 price of USD 0.40/s
- VPCT dataset — VPCT fields and scale