← Selected work

SPUR: Metric Depth for Robotics

Estimate orchard depth and reconstruct branch geometry with RGB-D models, ONNX export, and an inference API.

FIG. 06 — Application context: orchard camera rig. Model accuracy below is measured on synthetic data.

Result & current limits

Five saved best-validation scores: 0.0445 ± 0.0057 m trunk RMSE with three-pair DINOv2 RGB+D refinement. A seed-1 re-render scores 0.0467 m. V100 fp16 refiner-only p50: 156 ms, excluding DA2 and HTTP. Synthetic evaluation; real-orchard accuracy remains unverified.

Problem

Thin branches make metric-depth estimation difficult: small geometric errors can move a proposed cut away from the target. The project uses a synthetic orchard with known depth and camera geometry to compare models before attempting field validation.

Solution

Rendered orchard RGB, depth, masks, and camera poses in Blender. Compared monocular depth and multi-view refiners under a fixed synthetic protocol. A DINOv2 RGB-D model refines depth across three stereo pairs. Packaged inference in FastAPI with split ONNX export and calibrated 3D reconstruction. A separate BranchNet pipeline predicts 3D branch graphs from RGB-D; real-orchard generalization remains unverified.

What I built

Built the synthetic orchard, the evaluation protocol, DINOv2 RGB-D refinement, and the FastAPI/ONNX service. Real orchard accuracy is not claimed.

Technologies

More demos, engineering decisions, and evaluation notes

Why a synthetic orchard

Real depth sensors lose thin branches in noise, and hand-measuring metric depth in an orchard is not practical at scale. Blender renders the scene and its exact per-pixel depth together — 100 trees, 6 shots, 4 bark types — with segmentation masks and camera poses for free.

Real orchard photo beside a noisy sensor depth map where branch structure is lost in blocky colour.
FIG. 06A — The starting point: real sensor depth loses the branches entirely.
Procedurally generated dormant apple tree on a trellis, rendered in Blender.
FIG. 06B — A procedural dormant tree on trellis wire, rendered in Blender.
Ground-truth depth render of the same tree, every spur and wire resolved in colour-coded distance.
FIG. 06C — Its exact ground-truth depth — every spur and wire resolved.

Off-the-shelf depth broke on the branches

Initial Depth Anything V3 maps dropped thin branches and had unstable scale. A streaming least-squares fit estimates scale and shift over trunk pixels in constant memory. Restored-data evaluations use 100 trees and one bark type, separate from the full research dataset. Affine calibration aligns scale; segmentation and branch-boundary errors remain separate constraints.

Four panels: orchard RGB, DA3 metric depth, DA3 relative depth, and tiled PRO depth on the same scene.
FIG. 06D — Same scene, four models. Only the tiled PRO pass keeps the thin branches.
Bar chart of RMSE per model at three stages: uncalibrated, calibration only, and calibration with preprocessing.
FIG. 06E — Log axis: DA3 metric falls 18.99 m to 0.138 m through calibration and preprocessing.

Two refinement architectures

Both take predicted depth plus RGB from several nearby views and fuse them at the bottleneck. The CNN U-Net trains from scratch; the DINOv2 variant freezes a pretrained ViT-L on the RGB path and trains a CNN side branch for depth. The depth input mattered more than the architecture — fine-tuning DA2 set the floor, and three stereo pairs took it the rest of the way. A fourth pair added nothing.

Shared CNN U-Net encoder diagram: two views encoded with tied weights, bottlenecks concatenated and fused.
FIG. 06F — Baseline — shared U-Net encoders, concat plus 1x1 conv fusion.
Diagram of a frozen DINOv2 ViT-L RGB path fused with a trainable CNN depth side branch before a decoder.
FIG. 06G — Frozen DINOv2 ViT-L on RGB, trainable CNN side branch on depth.
Grouped bar chart of validation RMSE for DINO and CNN refiners at one, two, three and four stereo pairs.
FIG. 06H — Historical paper figure; headline uses five best-validation records, not a verified matched-baseline gain.

From paper to service

The research now runs as a service. One endpoint takes a single RGB frame through DA2-ft; another takes six views through the DINOv2 refiner. The ONNX graph is split — encoder 1.2 GB, fuse and decode 26 MB — so six views do not unroll 144 ViT-L blocks, and Torch-versus-ONNX agreement is 1.5e-5. Reconstruction back-projects those metres through the cameras already logged. No second network.

FIG. 06I — Back-projected metres, four cameras, Blender intrinsics and extrinsics.
FIG. 06J — Flow and detection cycling across neighbouring rigs — sensors add, they do not vote.
Bar chart of p50 inference latency by GPU and precision, comparing Quadro RTX 8000 and Tesla V100 at fp32 and fp16.
FIG. 06K — Refiner only: V100 fp16 p50 156 ms, fp32 393 ms; excludes DA2 and HTTP.

Paper

From depth to branch geometry

October 2–5 studies extend the service to branch graphs and cut directions. BranchNet raises skeleton F1 at 2 cm from 0.50 to 0.60 on two held-out synthetic trees, with 120 correlated frames. Fitted axes raise strict cut recall from 0.12 to 0.17 on that same set. Rendering whole orchard rows (the target row plus three behind it) instead of a single tree raises pixel mIoU from 0.23 to 0.76 and skeleton F1 from 0.01 to 0.36 on 10 held-out synthetic trees with rendered depth; with fused depth, every method collapses. On real MFO cherry images, the same whole-orchard training reduces background classified as wood from 31% to 8%, but class mIoU remains 0.12. These are perception results; physical cutting and real-orchard metric-depth accuracy remain unvalidated. Protocol and comparisons.

A placeholder camera matrix

Every annotation stored a hard-coded placeholder intrinsics matrix instead of the render camera’s. With the correct matrix, ground-truth depth lies 0.09 mm (median) from the rendered branch surfaces instead of 7.2 cm, and four-view reconstruction error falls from 7.0 cm to 7.4 mm. This is one tree and one shot. Per-pixel depth RMSE does not use the matrix and is unchanged, and the six-view refiner has not been retrained. Correction and report.

What sim-to-real actually costs

Reported model accuracy is synthetic; real-orchard accuracy has not been evaluated here. A hold-out affine fit across the 100-tree, one-bark restore puts DA2-ft at 0.0325 m on 78 valid ground-truth-mask frames from two held-out trees. A TinyUNet trunk segmenter reaches 0.930 IoU. But running the same depth through predicted masks instead of ground-truth ones moves error from 0.031 m to 0.112 m. That gap, not the headline, is the number to beat.

Stereo fusion is not shipped: naive flow with guessed variance made things worse, and epipolar gating left too few valid trunk pixels to score. A TensorRT fuse/decode engine exists for the RTX 8000, but its timing covers only that stage; it is not an end-to-end refiner result.

Links