SPUR: Metric Depth for Robotics
Estimate orchard depth and reconstruct branch geometry with RGB-D models, ONNX export, and an inference API.
Result & current limits
Five saved best-validation scores: 0.0445 ± 0.0057 m trunk RMSE with three-pair DINOv2 RGB+D refinement. A seed-1 re-render scores 0.0467 m. V100 fp16 refiner-only p50: 156 ms, excluding DA2 and HTTP. Synthetic evaluation; real-orchard accuracy remains unverified.
Problem
Thin branches make metric-depth estimation difficult: small geometric errors can move a proposed cut away from the target. The project uses a synthetic orchard with known depth and camera geometry to compare models before attempting field validation.
Solution
Rendered orchard RGB, depth, masks, and camera poses in Blender. Compared monocular depth and multi-view refiners under a fixed synthetic protocol. A DINOv2 RGB-D model refines depth across three stereo pairs. Packaged inference in FastAPI with split ONNX export and calibrated 3D reconstruction. A separate BranchNet pipeline predicts 3D branch graphs from RGB-D; real-orchard generalization remains unverified.
What I built
Built the synthetic orchard, the evaluation protocol, DINOv2 RGB-D refinement, and the FastAPI/ONNX service. Real orchard accuracy is not claimed.
Technologies
More demos, engineering decisions, and evaluation notes
Why a synthetic orchard
Real depth sensors lose thin branches in noise, and hand-measuring metric depth in an orchard is not practical at scale. Blender renders the scene and its exact per-pixel depth together — 100 trees, 6 shots, 4 bark types — with segmentation masks and camera poses for free.
Off-the-shelf depth broke on the branches
Initial Depth Anything V3 maps dropped thin branches and had unstable scale. A streaming least-squares fit estimates scale and shift over trunk pixels in constant memory. Restored-data evaluations use 100 trees and one bark type, separate from the full research dataset. Affine calibration aligns scale; segmentation and branch-boundary errors remain separate constraints.
Two refinement architectures
Both take predicted depth plus RGB from several nearby views and fuse them at the bottleneck. The CNN U-Net trains from scratch; the DINOv2 variant freezes a pretrained ViT-L on the RGB path and trains a CNN side branch for depth. The depth input mattered more than the architecture — fine-tuning DA2 set the floor, and three stereo pairs took it the rest of the way. A fourth pair added nothing.
From paper to service
The research now runs as a service. One endpoint takes a single RGB frame through DA2-ft; another takes six views through the DINOv2 refiner. The ONNX graph is split — encoder 1.2 GB, fuse and decode 26 MB — so six views do not unroll 144 ViT-L blocks, and Torch-versus-ONNX agreement is 1.5e-5. Reconstruction back-projects those metres through the cameras already logged. No second network.
Paper
From depth to branch geometry
October 2–5 studies extend the service to branch graphs and cut directions. BranchNet raises skeleton F1 at 2 cm from 0.50 to 0.60 on two held-out synthetic trees, with 120 correlated frames. Fitted axes raise strict cut recall from 0.12 to 0.17 on that same set. Rendering whole orchard rows (the target row plus three behind it) instead of a single tree raises pixel mIoU from 0.23 to 0.76 and skeleton F1 from 0.01 to 0.36 on 10 held-out synthetic trees with rendered depth; with fused depth, every method collapses. On real MFO cherry images, the same whole-orchard training reduces background classified as wood from 31% to 8%, but class mIoU remains 0.12. These are perception results; physical cutting and real-orchard metric-depth accuracy remain unvalidated. Protocol and comparisons.
A placeholder camera matrix
Every annotation stored a hard-coded placeholder intrinsics matrix instead of the render camera’s. With the correct matrix, ground-truth depth lies 0.09 mm (median) from the rendered branch surfaces instead of 7.2 cm, and four-view reconstruction error falls from 7.0 cm to 7.4 mm. This is one tree and one shot. Per-pixel depth RMSE does not use the matrix and is unchanged, and the six-view refiner has not been retrained. Correction and report.
What sim-to-real actually costs
Reported model accuracy is synthetic; real-orchard accuracy has not been evaluated here. A hold-out affine fit across the 100-tree, one-bark restore puts DA2-ft at 0.0325 m on 78 valid ground-truth-mask frames from two held-out trees. A TinyUNet trunk segmenter reaches 0.930 IoU. But running the same depth through predicted masks instead of ground-truth ones moves error from 0.031 m to 0.112 m. That gap, not the headline, is the number to beat.
Stereo fusion is not shipped: naive flow with guessed variance made things worse, and epipolar gating left too few valid trunk pixels to score. A TensorRT fuse/decode engine exists for the RTX 8000, but its timing covers only that stage; it is not an end-to-end refiner result.