← Selected work

Vision-Guided Pruning

Follow a branch with wrist RGB-D, verify contact and geometry, then release a simulated spur.

FIG. 07 — October 2: targeted evening pass with depth-aware appearance checks; one of four low-sun trials.

Result & current limits

October 2: depth-aware appearance checks complete 4/4 targeted low-sun trials while rejecting genuine occlusion controls. A separate valid tree1 population passes 2/7. ROS 2 replay reproduces 199/199 decision states from one capture. Simulator ground-truth depth and rigid-piece release.

Problem

A pruning arm must keep a thin branch in view and stop when tool occlusion or lost tracking makes release unreliable. A convincing recording alone cannot establish that the cut conditions were met.

Solution

Built a wrist RGB-D controller in Isaac Sim using Lucas–Kanade tracking and bounded Cartesian commands. Dual ToF, contact, and geometry checks gate a rigid-spur release; a separate grader checks the saved sequence. Depth is simulator ground truth, and branch identity comes from mesh metadata. This is a control study, not a learned detector or a wood-fracture simulation.

What I built

I built the capture loop, RGB-D controller, release gates, independent grader, and success/failure recordings. The robot, orchard meshes, and sensor frames come from pinned upstream sources.

Technologies

More demos, engineering decisions, and evaluation notes

Keep the failure next to the success

Tracking confidence fell below 0.15 at 7.7 s on job 21317409. Closure stopped at 2/3; no release. The public recording is the failed episode, not a cleaned retake.

FIG. 07A — Closure failure at 7.7 s; motion stops without release.

What drives the arm

Live control uses seeded LK features and RTX depth. Farneback flow is an offline diagnostic. Depth is simulator ground truth. The jaw is a visual proxy; there is no blade CAD or wood fracture model.

Generalization stays an open problem

The first registered sweep completed 0/40 planned trials; 20 had invalid tree1 registrations. Later populations passed 2/7 listed tree1 targets and 1/30 seeded targets. These are separate protocols. The selected 17/17 demo is one episode, not a population success rate.

Separate shadow loss from occlusion

A depth-aware surface check completed 4/4 targeted morning/evening trials on target 15004; true jaw and wire occlusion controls still stop. A separate known-map jaw-hold variant passes 3/3 on target 530 with freshness checks waived on held frames. Neither variant is pooled with baseline trials.

FIG. 07B — Same target and evening lighting: earlier appearance-based tracking stops at the shadow.

Links