← Selected work

Bimanual Garment Folding

Run a camera-driven VLA in Isaac Sim and promote fine-tunes only after paired fold evaluations.

Checker: success
Checker: no fold
FIG. 08 — Historical policy clips: a transient checker pass and a failure; separate from the latest settled-fold study.

Result & current limits

October 6 runtime screen: unchanged baseline weights settled 16/48 folds at H10 and 8/48 at H50 across 48 matched pairs. Exact paired p = 0.0577 missed the 0.05 gate, so the baseline and H50 remain. Eight reused development poses; simulated cloth.

Problem

A cloth-folding policy can appear successful during a demonstration replay or a transient fold. Evaluation needs actual policy actions, held-out poses, and a checker that scores the garment after it settles.

Solution

Ported SmolVLA to PhysX cloth using three camera views and direct joint targets. Added recovery supervision and paired candidate/baseline evaluations with pinned source. The challenge checker scores after 600 actions and a settling period. Compared replanning every 10 actions with the full 50-action deployment chunk.

What I built

I built the simulator port, observation path, failure diagnostics, recovery-training adaptations, and paired evaluation drivers. Robot assets, garment assets, the challenge checker, and the SmolVLA backbone are upstream components.

Technologies

More demos, engineering decisions, and evaluation notes

What both arms see

The policy reads three camera views and joint state, then emits joint targets. Both wrist views show whether the arms coordinate. Demonstration replay only proves the scene is solvable; reported results use closed-loop policy actions and the challenge checker. Failures stay visible alongside successes.

FIG. 08A — Left and right wrist cameras during the scored fold.

More frequent replanning, inconclusive gain

The October screen tests how often the same model replans. All 96 episodes completed 600 actions plus a 60-step settle on A40 GPUs. Fourteen pairs favored H10 and six favored H50, giving exact paired p = 0.0577. The preregistered gate fails: baseline retained. Clustered repeats and reused poses limit generalization.

The earlier fine-tune also fails promotion

Changing the behavior-cloning anchor removed an early H50 action shift, but did not pass the September v8 release gate. H10 improved from 2/16 to 9/16; H50 remained 6/16 versus the paired baseline's 8/16. The historical 2/8 checker-pass sweep and its video use an earlier metric and checkpoint; they are not settled-fold scores.

New adaptation, recorded failure

Gate 21367715 completed 600 actions and 601 validated render calls after an asset-root repair. The adapted seed-0 policy did not fold this short top. This is a development pose and a different checkpoint from the historical success above, not a matched success-rate comparison.

FIG. 08B — 20 September: new policy failure, with both wrist views and the overhead camera.

Why the original policy ignored the cloth

The policy was trained on path-traced frames, but the simulator rollouts feed it rasterized ones. On identical replayed states, the converged checkpoint’s action-prediction skill (1 is perfect; 0 is no better than always predicting the mean action) was +0.976 on path-traced frames and −0.038 on rasterized frames. Fine-tuning on 18,200 rasterized frames raised it to +0.652. This is a single-step, teacher-forced score, not a fold rate. Job ledger, rows 21155266, 21155267, 21192053 and 21193835.

A silent optimizer defect, found in the checkpoint bytes

The trainer ran AdamW directly on bf16 action-expert weights with no FP32 master copy, so most updates rounded to zero. A byte-level comparison of the stored weights showed that only 11.5% of the 96.6M expert weights had changed. A later fine-tune with FP32 master weights changed 59.4%. The two checkpoints come from different runs and steps, so this is not an isolated ablation, and weight movement is not fold success. Report · Byte audit.

Links