Vision-Guided Pruning
Follow a branch with wrist RGB-D, verify contact and geometry, then release a simulated spur.
Result & current limits
October 2: depth-aware appearance checks complete 4/4 targeted low-sun trials while rejecting genuine occlusion controls. A separate valid tree1 population passes 2/7. ROS 2 replay reproduces 199/199 decision states from one capture. Simulator ground-truth depth and rigid-piece release.
Problem
A pruning arm must keep a thin branch in view and stop when tool occlusion or lost tracking makes release unreliable. A convincing recording alone cannot establish that the cut conditions were met.
Solution
Built a wrist RGB-D controller in Isaac Sim using Lucas–Kanade tracking and bounded Cartesian commands. Dual ToF, contact, and geometry checks gate a rigid-spur release; a separate grader checks the saved sequence. Depth is simulator ground truth, and branch identity comes from mesh metadata. This is a control study, not a learned detector or a wood-fracture simulation.
What I built
I built the capture loop, RGB-D controller, release gates, independent grader, and success/failure recordings. The robot, orchard meshes, and sensor frames come from pinned upstream sources.
Technologies
More demos, engineering decisions, and evaluation notes
Keep the failure next to the success
Tracking confidence fell below 0.15 at 7.7 s on job 21317409. Closure stopped at 2/3; no release. The public recording is the failed episode, not a cleaned retake.
What drives the arm
Live control uses seeded LK features and RTX depth. Farneback flow is an offline diagnostic. Depth is simulator ground truth. The jaw is a visual proxy; there is no blade CAD or wood fracture model.
Generalization stays an open problem
The first registered sweep completed 0/40 planned trials; 20 had invalid tree1 registrations. Later populations passed 2/7 listed tree1 targets and 1/30 seeded targets. These are separate protocols. The selected 17/17 demo is one episode, not a population success rate.
Separate shadow loss from occlusion
A depth-aware surface check completed 4/4 targeted morning/evening trials on target 15004; true jaw and wire occlusion controls still stop. A separate known-map jaw-hold variant passes 3/3 on target 530 with freshness checks waived on held frames. Neither variant is pooled with baseline trials.