Language-Guided Manipulation
A humanoid follows an instruction to pick a tumbler, load a tray, and carry it to the requested table.
Result & current limits
Retrained SmolVLA completed 75/100 tasks with original instructions and 76/100 with a held-out phrasing. These are simulation results from one checkpoint, with the base held in place while standing. The domain contains two tumbler colours, two destination tables, and six instruction templates.
Problem
The same scene can require a different object or destination depending on the instruction. A successful task needs language grounding, vision, grasping, and carrying to work together.
Solution
Built a simulated waiter task for a 24-joint humanoid with grippers. Fine-tuned SmolVLA on scripted demonstrations to turn head-camera images, joint state, and text into base, arm, and gripper commands. Evaluated unseen layouts and a held-out instruction phrasing, then tested DSRL noise steering against the frozen policy and constant-noise controls.
What I built
Built the task environment, demonstration pipeline, policy integration, and evaluation gates. Traced object-choice errors to sparse early-episode samples and retrained with more demonstrations and targeted frame sampling.
Technologies
More demos, engineering decisions, and evaluation notes
Learn the choice as well as the motion
The instruction selects the tumbler colour and destination table. SmolVLA receives a rendered head-camera image, text, and proprioception, then predicts chunks of base-velocity, arm, and gripper commands. A learned leg controller executes walking while the arms and grippers track their commanded targets. Task and interface contract.
The first fine-tune scored 28/100 with each layout’s instruction. Its features held both the instruction’s colour and which tumbler was which, but it picked between the two at about chance. The choice is made in an episode’s first ten frames, 1.6% of the training data. On development layouts, 3.4× the demonstrations alone left the choice at chance (95 of 189 pick-ups). With the same demonstrations, drawing 30% of each training batch from the first 12 frames made it right in 192 of 192. Phase 3b, trained that way, scored 75/100 on fresh layouts and 76/100 with the held-out template. Program and results.
Keep the stronger baseline
DSRL noise steering on the retrained policy reached 80/100 versus 74/100 for its normal noise, failing the improvement gate (p = 0.20). Constant zero noise reached 82/100. The learned noise policy therefore did not justify replacing the baseline. Phase 4b verdict.
What the demonstration establishes
This project tests a small, simulated instruction domain: two colours, two tables, and six phrasing templates. The scripted expert reads oracle object positions from the simulator. The learned policy uses rendered vision, text, and joint state. The demonstration clip is a development-layout run, separate from the scored evaluation.
A declared stance lock holds the base while the robot stands and releases it for walking. The scripted expert scored 86/100 with this assistance and 1/100 with a free base. Removing standing assistance remains unfinished; these results do not demonstrate unassisted whole-body manipulation or physical deployment.
Related work: Locomotion & Control and Perception & Navigation.