Long Range Navigator (LRN): Extending robot planning horizons beyond metric maps

Problem Setup

The robot operates in a static but initially unknown environment. It receives GPS localization and a distant goal waypoint, but no prior map. A local policy operates under a finite horizon because sensing and computation are bounded. The policy observes the robot state, plans, executes an action, and replans in a model-predictive fashion. The resulting path should minimize expected cost over unknown environment instances:

Here, $\xi^{\pi}$ is a path induced by the policy, $\phi$ is one unknown environment realization, and $C$ is path cost. The objective is an expectation because the robot does not know the terrain in advance. LRN does not solve this global problem directly. It constructs an intermediate long-range direction estimate that can be consumed by the local policy.

Let $f$ denote a frontier state on the edge of the local horizon $H$. If the optimal frontier $f^{*}$ lies on the optimal route and the local policy is reliable within $H$, then planning to $f^{*}$ decomposes the global problem into local subproblems. This is the optimal-substructure argument. In practice, $f^{*}$ depends on terrain that no current sensor has observed, so the robot can only estimate which frontier is worth trying.

The value $V(s,g,f)$ combines two quantities. $A(s,f)$ measures whether the frontier is affordable from state $s$. $D(f,g_t)$ measures the estimated cost from the frontier to the goal $g_t$. The first term asks whether the robot can enter the opening and continue. The second asks whether the opening points toward the current task. Neither quantity alone is sufficient.

The affordability score contains two existence probabilities. The first requires a path from the robot to the frontier. The second requires some possible continuation from the frontier toward a goal beyond the local horizon. A surface-level traversability model captures mostly the first term. LRN's key addition is the second term, which rewards openings that can plausibly lead farther into the environment rather than merely being free of nearby obstacles.

Once the frontier values are known, the selected frontier and the resulting policy are:

$F$ is a discrete set of frontier candidates and $\pi_{\lambda}$ is the existing local policy. LRN therefore produces a direction, not motor commands. The local system can reject unsafe trajectories and continue to perform obstacle avoidance. This separation is central to the paper's engineering argument.

Method

Predicting image-space frontiers instead of a distant map

Projecting a frontier into 3D requires depth. In outdoor monocular images, the sky, distant trees, and ground can have ambiguous appearance, and a small depth error can place a pixel in empty space or on a lethal obstacle. LRN avoids this intermediate 3D reconstruction. It predicts an image-space heatmap representing a visual path to an affordable opening, then projects heatmap responses into angular directions using camera intrinsics.

The approximation still has a geometric assumption. A two-dimensional hotspot is not a measured 3D point. LRN extends the corresponding ray to the local horizon and assumes that there is a clear line of sight from that projected point to the real opening represented by the image. The mapping from images to heatmaps is learned from walking trajectories and human labels rather than enforced by a perfect geometric constraint. The output is therefore a direction worth investigating, not a collision-free far-field path.

Turning walking trajectories into labels

To train the heatmap model, the system needs to know which image locations correspond to affordable frontiers. Human annotators can mark gaps between trees, hill crests, and forks in a road, but annotation is slow. The authors instead collect egocentric walking videos and track a grid of image points with CoTracker. They run the video backward and select points near the camera at the later time, so successful future ground locations can be traced back to the earlier observation.

When a tracked point becomes occluded after returning to the start, it was not visible ground in the starting image even though the walker later reached that area. The point is marked as an affordable-frontier hotspot with score 1. Other points along the trajectory receive score 0, and image regions not covered by the trajectory remain unlabeled. This avoids requiring robot pose, metric depth, or semantic segmentation.

The authors add one important correction. They label the vertical column around each hotspot as negative, especially to reduce false positives in the sky. Without that constraint, a network may exploit the vertical alignment of ground and sky and put high response above the actual opening. This detail shows that weak labels must encode not only positive locations but also how projection ambiguity should be suppressed.

A frozen visual encoder and a small decoder

The affordance backbone encodes RGB images with a frozen visual foundation model and trains a small deconvolution decoder to predict spatial heatmaps. Spot uses MobileSAM to meet the compute budget of a Jetson Orin AGX, while the heavy tracked vehicle uses SAM2. The design reuses large-scale visual pretraining and concentrates task-specific learning in a relatively small output network.

The supervised loss is mean squared error on the valid heatmap pixels, with L2 regularization on the learned weights. For training sample $(I_i,Y_i)$ and model $A_{\theta}$, the objective can be written as:

$Y_i$ contains positive hotspots, negative trajectory samples, and regions omitted from supervision. Freezing the encoder reduces training cost and preserves the foundation model's robustness to appearance changes. The paper does not isolate every source of generalization, however. The two platforms use different encoders, different label sources, and different thresholds, so the deployed systems still contain substantial platform-specific tuning.

Projection, thresholding, and temporal filtering

At runtime, every camera produces a heatmap. Responses are projected into $k$ angular bins with camera intrinsics, summed within each bin, thresholded at $h_{\mathrm{thresh}}$, and merged across overlapping cameras by taking the maximum. The resulting vector $\mathbf{b}_t$ is normalized:

Normalization removes frame-level scale changes, preventing an overall stronger heatmap from dominating every historical observation. An exponential moving average then smooths the angular scores:

The parameter $\alpha$ trades responsiveness against stability. A small value suppresses noise but delays the reaction to a newly revealed opening. Both real-world systems use $\alpha=0.1$, so a new frame contributes only ten percent of the updated score. This makes the direction sticky, but it also means a rapid change in the environment cannot be reflected immediately.

Combining affordability, goal alignment, and consistency

Affordability alone can produce many candidates in open terrain. The goal-conditioned head first forms a Gaussian over angular bins centered on the goal direction:

It also forms a consistency Gaussian centered on the previously selected direction:

The three vectors are multiplied element by element, and the maximum component becomes the next direction:

Multiplicative fusion is intentionally conservative. A direction is penalized if it is visually affordable but far from the goal, or goal-aligned but poorly connected. The previous-direction term reduces switching, but it also introduces inertia. The resulting value is a heuristic ranking, not a calibrated estimate of the true long-term cost.

Spot adds two distance-based safeguards. Inside 30 meters, the goal Gaussian narrows linearly with distance, and inside 12 meters the robot heads directly toward the goal. The tracked vehicle switches to direct goal heading inside 75 meters. These rules acknowledge an important boundary: LRN is useful when local sensing cannot see enough route structure. When the goal is already within the local planning range, continued exploration pressure may cause unnecessary detours.

Integration with two local navigation stacks

Spot maintains a 16-meter square elevation map from depth cameras and lidar, converts slope to cost, and uses ARA* over a lattice. A carrot point three meters along the planned path produces a body-frame velocity for Spot's internal locomotion controller. LRN processes front and rear fisheye images at 4 Hz and places the selected direction as a goal just outside the local costmap.

The 12-ton tracked vehicle uses a circular costmap with a 50-meter radius, three front cameras, and one rear camera. Its search-based planner stops at the costmap edge when it reaches a frontier close to the requested heading. LRN runs on all four synchronized images at 4 Hz. In both systems, LRN chooses a long-range subgoal while the existing planner retains local path search and safety checks.

flowchart LR
    A[Egocentric camera] --> B[Frozen SAM2 or MobileSAM encoder]
    B --> C[Small deconvolution decoder]
    C --> D[Affordance heatmap]
    D --> E[Project to angular bins]
    E --> F[Threshold and EMA filter]
    G[Goal heading] --> H[Goal Gaussian]
    I[Previous heading] --> J[Consistency Gaussian]
    F --> K[Multiply score vectors]
    H --> K
    J --> K
    K --> L[Select maximum heading]
    L --> M[Local costmap and planner]
    M --> A

The feedback loop denotes repeated visual inference, not maintenance of a complete spatial map. LRN stores only short-term angular scores and the previous decision. The platform-specific local stack continues to maintain metric geometry. This boundary also explains why the paper later identifies missing history and possible backtracking as open problems.

Training Data and System Settings

For Spot, the authors recorded 54 minutes of egocentric video while walking through two semi-urban environments with an Insta360 camera. Long trajectories can accumulate point drift as the tracked ground points move out of view, so the videos were split into two-minute segments. The automatic pipeline produced 92,711 heatmap labels. The resulting MobileSAM model uses front and rear fisheye images during deployment.

The tracked-vehicle experiment used an earlier LRN trained only on human labels. The authors collected 1,901 images in a California oak savanna and asked annotators to mark all affordable frontiers. Positive labels received a small Gaussian blur, and augmentation with color jitter, sharpness changes, rotations, and blur expanded the set to 11,406 images. The labels came from a different Racer Heavy vehicle with front-facing cameras, while deployment used three front and one rear camera, making the demonstration a cross-vehicle and cross-sensor transfer test.

The two systems share the same conceptual pipeline but require different thresholds and Gaussian widths. These settings should not be treated as universal defaults. They depend on camera geometry, local map range, planner behavior, and vehicle dynamics.

Experiments and Results

Spot was evaluated on three semi-urban courses. Dump placed the goal 60 meters behind a large wall. Night required crossing a bridge to a point 57 meters away while a wall of bushes and a river blocked the straight line. Helipad required reaching the other side of an 82-meter open field with a hedge across the direct route. Each method ran five trials per course, giving 60 main trials across five approaches. The threshold ablation added 30 trials at Dump, five for each of six heatmap thresholds.

The baselines test different hypotheses. Goal Heuristic uses LRN's goal-conditioned head but replaces the heatmap scores with a uniform distribution, isolating the effect of learned affordances. NoMaD is a diffusion policy that predicts trajectories from a current and goal image. Traversability plus Depth Anything V2 multiplies visual traversability with monocular depth to identify distant traversable points.

Performance is measured by total distance, total time, and human interventions. An intervention is counted when the robot stops making progress or enters a dangerous situation. The authors explicitly exclude interventions caused only by failures of local obstacle perception, because the study targets long-range direction selection rather than the low-level perception stack.

On the Spot courses, LRN outperformed Goal Heuristic on all reported metrics for Dump and Night and was comparable on Helipad. It was competitive with NoMaD and the traversability baseline overall, but it did not minimize every metric on every course. On Helipad, several predictive methods produced longer paths because changing heatmap decisions caused wandering. LRN, traversability plus depth, and NoMaD all showed this behavior to some degree.

The stronger system-level result is that LRN required zero human interventions across the three Spot courses, while the other methods required at least some intervention. This supports the claim that seeing distant obstacles earlier can reduce rescues, but it does not mean every LRN trajectory is shorter. The Helipad result is an explicit counterexample to equating better long-range perception with immediate path efficiency.

The tracked-vehicle test followed a 660-meter route over three hills, two containing dense tree clusters. Because of limited field time, each method received one run. Goal Heuristic drove into the treeline on the second hill, escaped, and then became trapped on the next hill, requiring 60 meters of teleoperation. LRN selected a clear opening, avoided the dense trees, and completed the route without intervention after temporarily veering south near the end.

Table I reports the tracked-vehicle run. LRN had higher average and maximum speed and required no intervention, but it traveled farther and took more time. The longer route is consistent with avoiding dense forest. The demonstration supports safer route choice, not shortest-path optimality.

Table I: Racer Heavy run on the 660 m course. LRN drove farther and longer, but faster and without human intervention.

The offline evaluation used 330 unseen Spot images and 315 unseen Racer Heavy images from the same environments, with human heatmap labels. Targets and predictions were binarized at a heatmap threshold of 0.15. LRN was compared with the traversability-plus-depth baseline using AUROC, F1, precision, recall, false positive rate, and false negative rate. The Spot automatic-label model achieved a high AUROC of 0.93, but its F1 was only 0.10 and precision was 0.06. Ranking quality and pointwise localization accuracy must therefore be interpreted separately.

Table II: Offline heatmap metrics. Human labels perform better overall; the authors attribute the automatic-label gap to tracking errors.

The threshold ablation illustrates the shape of the design space. A threshold near zero marks almost everything affordable, causing direct motion into local minima similar to Goal Heuristic. A threshold near one leaves too few choices and produces switching between direct goal motion and occasional hotspots. On Dump, an intermediate value of 0.7 performed best. The authors fit a line from threshold 0 to 0.7 and report a correlation with traversal distance at $p < 0.05$. This is useful evidence, but it comes from one course and should not be read as a universal calibration.

The generalization evidence has three parts. A model trained on human walking video ran on Spot, a model trained from another Racer Heavy's front camera ran on the deployed vehicle, and a Spot model trained on daytime video performed in some nighttime conditions. The authors credit the frozen visual embeddings and the goal-agnostic intermediate representation, while noting that supervised models remain strongest near the training distribution.

Limitations and Failure Modes

The first author-stated limitation is the absence of explicit depth. When two hotspots have similar angular distance to the goal direction, LRN cannot know which is actually closer in 3D. The angular difference is used as an implicit distance proxy. Open courses such as Helipad and the Racer Heavy route can contain many hotspots, producing occasional wandering. The authors suggest sparse depth or a detector that falls back to Goal Heuristic when goal-distance changes become erratic.

The second limitation is switching caused by heatmap score fluctuations. Normalization, EMA, and the previous-heading Gaussian reduce rapid changes but do not eliminate them. Both deployed systems use fixed Gaussian widths, and the tracked vehicle retains a wide goal Gaussian even near the goal, which contributed to a missed opening in the demonstration. Learning a goal-conditioned head with temporal history is proposed as a stronger alternative.

The third limitation comes from the automatic labels. CoTracker can drift, and small tracking errors near occlusion boundaries can place heat on the edge of an obstacle. The Spot model is consequently optimistic around some openings. Human labels outperform automatic labels in the offline study, showing that improving track quality or filtering bad labels could directly improve the learned affordance model.

Finally, LRN remains a heuristic for exploring unknown space. A frontier that looks good from one viewpoint may lead to a dead end or impassable terrain after it is reached. Without memory, the robot cannot reliably remember a previously attractive hotspot or backtrack to it. The authors explicitly list sparse depth and frontier memory as future work. The paper therefore extends local perception rather than solving fully autonomous long-range navigation.

An independent review should also note the scope of the evaluation. Spot contributes 60 main trials and 30 ablation trials, but only three courses. The tracked vehicle offers a compelling cross-platform test on a 660-meter route, yet each method has only one trial, so the result provides no variance estimate or confidence interval. The platforms also differ in label source, encoder, camera layout, and tuning. Cross-platform success is evidence of engineering feasibility, not a statistical guarantee of generalization.

The offline metrics need careful interpretation. Spot's automatic model has a high AUROC but low F1 and precision at a fixed threshold. Most of its hottest pixels are not exactly on the human label. The system still works because heatmaps are aggregated into angular bins and then filtered over time. The downstream consumer needs a useful relative ordering of directions, not pixel-perfect segmentation. Future evaluations should therefore pair classification metrics with closed-loop interventions, path stability, and completion rate.

Discussion

LRN offers a useful lesson about matching model abstractions to downstream consumers. A local planner needs a boundary target and a direction, not a potentially inaccurate distant 3D reconstruction. By learning whether an opening is worth approaching, the visual model can reduce myopia while leaving collision avoidance in the mature local stack. This design is simpler to integrate than an end-to-end replacement and makes failure modes easier to separate.

The data strategy is equally important. Egocentric walking video can reveal where a person later reached, but it does not reveal whether that route is safe for a specific vehicle, whether the opening continues indefinitely, or whether a large tracked platform will fit. Weak supervision is suitable for learning visual affordance cues, not for replacing dynamics, terrain safety, or local risk estimation. Keeping those functions in the navigation stack is one reason LRN can cross embodiments.

The most valuable next step may be memory rather than a larger image model. Sparse depth can resolve same-angle distance ambiguity. A lightweight topological memory of attempted and successful frontiers could reduce oscillation and support backtracking. Such memory would turn the current frame-level heuristic into a longer-lived route hypothesis while retaining the ability to react quickly in unknown terrain.

Conclusion

The paper asks what long-range navigation must preserve, rather than how to enlarge a metric map indefinitely. Its answer is a small set of affordable frontier directions. LRN learns those directions from unlabeled egocentric video, re-ranks them with goal and temporal context, and integrates the selected direction with existing planners on Spot and a 12-ton tracked vehicle. The experiments show earlier avoidance of dangerous terrain, fewer interventions, and transfer from human walking data to a robot.

The remaining problems are concrete: noisy automatic labels, missing depth, unstable direction scores, and no memory of failed routes. The work is persuasive because it defines a tractable intermediate question and demonstrates it on two very different physical systems. It does not establish universal outdoor autonomy. Because code and data were not yet public, independent reproduction remains an important next test.