Inspect before proposing
Identify the frozen checkpoint, permitted observations, action interface, benchmark, and compute budget. Retrieve relevant mechanisms together with their assumptions and prior failures.
Under review at CVPR
RoboMetaHarnessAutomated Harness Discovery
for Frozen Robot Policies
University of California, San Diego
Can a research agent improve how a robot uses a policy it cannot retrain?
RoboMetaHarness closes the loop from failure diagnosis to executable harnesses, controlled experiments, and reusable research memory. Every model weight stays frozen.
0model-training steps
12harness capability families
3 armscandidate · baseline · null
4 stagesaudit → subset → validation → full
Abstract
Frozen robot policies can fail because of how observations are presented, how action proposals are selected, or how execution is coordinated. Improving these interfaces offers a route to better behavior without updating the policy’s parameters. Discovering a useful intervention, however, requires more than generating plausible wrapper code: it requires compatible interfaces, reproducible comparisons, and evidence that survives independent evaluation.
RoboMetaHarness is an automated research pipeline for discovering training-free harnesses around a user-selected robot policy. It retrieves mechanisms from a research memory, diagnoses recorded failures, proposes falsifiable hypotheses, implements and freezes candidates, and evaluates them against both a baseline and an overhead-matched null control. A staged protocol separates development from fresh validation and full evaluation; negative findings and infrastructure failures return to memory with their costs and limitations.
The project currently contains development screens and reproducibility audits for robot-policy harnesses. These records illustrate why proposal selection, inference precision, and simulator replay must be tested together. The evidence shown here does not establish a validated full-benchmark harness improvement.
How it works
From a robot failure to a testable intervention
A research loop grounded in the policy interface, measured trajectories, and a fixed evaluation contract.
Make the mechanism falsifiable
A proposal specifies a failure, an intervention, a predicted effect, a null control, and a cost. Independent review checks whether the mechanism fits the original policy.
Freeze the experiment
Implement a wrapper and pin its source, configuration, cases, and checkpoints before evaluation. Keep policy and auxiliary weights frozen, with official horizons and control timing.
Test the comparison itself
Check initial observations and replay traces before interpreting outcomes. An overhead null performs extra work while preserving the baseline action rule.
Earn each larger experiment
Start with a small, predeclared development subset. Only complete, valid improvements against both controls can advance to fresh validation and then full evaluation.
Retain what failed, and why
Analyze gains, regressions, intervention activations, and inference cost. Preserve negative results and distinguish robot-policy failures from invalid experiments.
Experimental evidence
Measured outcomes. Explicit limits.
A snapshot of the recorded development screens and audits, reviewed October 20, 2026.
Current evidence status
No validated full-benchmark harness improvement is established in these records. The development screens below were not promoted. Historical screens inform the research memory; they are not evidence that a new pipeline candidate passed validation.
Reproduced baseline · Awomo-0.5I
43.96%
211 / 480 episodes · RoboCasa365 Composite-Unseen
The cited published reference is 217/480 (45.21%). It is a separate reference, not a matched control or a SOTA claim.
Reproduced baseline · GR00T N1.5
4.25%
34 / 800 episodes · native Composite-Unseen audit
Human300 checkpoint 120000. The session records a separate official 2.7% reference; this local result is not an exact paper replication claim.
Historical Awomo discovery screens
Each screen contains 32 development episodes per arm. These measurements were cached as starting evidence for harness discovery. Hatching marks an initial-observation mismatch that invalidated comparison with the baseline.
| Screen / intervention | Baseline | Null | Candidate | Interpretation |
|---|---|---|---|---|
| D · consensus medoid (K = 3) | 10/32 | 11/32 | 14/32 | Invalid comparisonInitial input mismatch against baseline. About 2.30× aggregate inference time; no promotion. |
| D · local action mean | 10/32 | 11/32 | 11/32 | Invalid comparisonInitial input mismatch against baseline; tied its null. |
| E · FP32 flow accumulation | 10/32 | 11/32 | 12/32 | Below gain gateNet gains of +2 and +1 did not meet the declared +3 gate against both controls. |
| E · second-order Heun | 10/32 | 11/32 | 7/32 | Invalid comparisonInitial input mismatch; the observed count was also below both controls. |
Why the null control matters
The medoid harness samples three native action chunks and executes the one closest to the others. A sampling null also generates extra proposals but returns the original first proposal and restores RNG state. This separates the action-selection mechanism from the effects of additional inference. Three consistently wrong proposals can still produce the wrong action.
Reproducibility is an experiment
Initial fixture restores passed
Independent task/process restores matched the original camera, state, and instruction bytes. This establishes initial identity for those checks, not long-trajectory determinism.
Steps per native preflight arm
On one GR00T CuttingToolSelection scene, baseline and a copy/hash null matched public inputs and executed actions throughout two complete 1,200-step arms. This was a one-sample control.
Required K = 3 replay pairs completed
A separate replay matched BreadSelection and CuttingToolSelection. GarnishPancake stopped at chunk index 31 after public-input drift. The audit remained incomplete; candidate quality was not measured.
Download the evidence snapshot (JSON) ↓Includes source-relative provenance and SHA-256 hashes.
Real simulator recordings
Watch a matched replay pair
GR00T N1.5 · BreadSelection · 1,950 steps per arm · recorded at 20 fps
Frozen-policy baseline
Sampling null · K = 3
These are control recordings from a completed replay pair, not candidate success demonstrations. The two video files have identical SHA-256 hashes, consistent with the recorded input/action parity for this case. The broader three-case replay audit stopped on another task.
Project materials
Figures & evidence
Architecture diagramIllustrated research and robot execution loopsPNG ↗Vector method overviewOriginal schematic for reading or presentationPDF ↗Vector method artworkOriginal editable schematicSVG ↗Development resultsHistorical screens with comparison validity markedPDF ↗Evidence snapshotCounts, limitations, and source hashesJSON ↓
RoboMetaHarness · Under review at CVPR