WayTrace: A Map-to-Language-to-Trace Hierarchy for Embodied Urban Navigation

System Formulation

Let $c_t$ denote the atomic command selected at time $t$, $\boldsymbol{\tau}_t$ the predicted image-space trace, and $\mathbf{a}_t$ the low-level action. The three levels observe at different delays and frequencies. The orchestrator sees a satellite crop, recent camera frames, and the active map direction. The onboard predictor sees the current frame, five historical frames, a walked-path map, and the command. The controller sees depth, robot state, and a local carrot point.

$$c_t=\mathcal{O}(\mathbf{x}^{o}_{t-\Delta_o}),\qquad \boldsymbol{\tau}_t=f_\theta(\mathbf{x}_{t-\Delta_\tau},c_t),\qquad \mathbf{a}_t=\pi(D_t,\mathbf{q}_t,\mathbf{v}_t)$$

Here $\mathcal{O}$ is the orchestrator, $f_\theta$ is the trace predictor, and $\pi$ is the reinforcement learning policy. $\mathbf{x}^{o}$ contains the route context available to the remote model, while $\mathbf{x}$ contains the local visual history available to the robot. The offsets $\Delta_o$ and $\Delta_\tau$ make the asynchrony explicit instead of pretending that every module receives a synchronized world state.

The practical timing differs sharply. The orchestrator emits roughly one command every two seconds and takes about 1.3 seconds per decision. The Jetson-based trace predictor takes about 1.5 seconds per request. The controller runs at 5 Hz. This arrangement lets the robot continue following a valid trace while the slower semantic layers are still computing.

flowchart TB
    R[Web route and turn directions] --> S[GNSS selects active route step]
    R --> M[50 m satellite crop]
    C[Recent 15 m camera frames] --> O[Remote orchestrator VLM]
    S --> O
    M --> O
    O -->|one atomic command| T[Onboard trace predictor]
    F[Current frame] --> T
    H[Five frames about 1.8 m apart] --> T
    P[Walked-path map over 16 m] --> T
    T --> G[Goal plus 10 image points]
    G --> D[Depth lift to ground]
    D --> K[Carrot point 4.5 m ahead]
    K --> A[RL policy 5 Hz]
    A --> B[Unitree B2W motion]
    A --> Q{Refresh trace}
    Q -->|new command or 3 m or 40 percent or 5 s| T

The Mermaid diagram mirrors the paper's actual dependency graph rather than a generic perception-planning-control stack. GNSS appears only on the route-selection branch. It does not feed the trace pixel by pixel. The local policy receives a carrot point, not the entire web-map polyline. The refresh loop is essential: without it, an initially correct trace would become stale after the robot moved or the orchestrator changed its mind.

Route Construction and the Orchestrator

WayTrace first obtains a walking route and turn-by-turn directions from Google Routes, then replans the route over the OpenStreetMap pedestrian graph. The replanning reduces the chance that a route intended for vehicles will be followed across an unsafe or non-walkable connection. During execution, GNSS selects the current route step and an approximate remaining distance. The final stop condition is based on the robot reaching within 1.5 m of the destination.

The orchestrator is Qwen3.8-Flash-Next with FP8 weights and no task-specific fine-tuning. It runs on four H100 GPUs. Its input combines the active map instruction, a heading-up satellite crop extending about 50 m ahead, and camera images from the previous 15 m. Its output begins with a brief visual reasoning trace and ends with one command. Examples include “turn left at the intersection,” “continue along the sidewalk,” and “go right onto the stairs.”

The authors add robot-side rules that reject commands issued too early. This matters because a web map may say “turn left in 4 m” while the current camera view shows a wall on the left and the actual crossing farther ahead. The orchestrator is asked to decide whether the robot has reached the trigger location, but the final module still checks obvious timing violations. The hierarchy therefore combines learned semantic judgment with deterministic safeguards.

Language is retained as the interface for several reasons. A person can audit the command, a field operator can replace the web map with spoken directions, and a failed route can be revised without retraining the onboard trace model. The language layer is not merely an explanation generated after control. It is the representation that decouples a cloud-scale route planner from an onboard visual controller.

Figure 2: One decision in the Snipes mission. The orchestrator sees the route, satellite crop, and recent frames; the trace model receives the command, visual history, and walked-path map.

Language-Conditioned Trace Prediction

The onboard predictor starts from RynnBrain-9B and is adapted with LoRA for one epoch on approximately 1.19 million samples. It receives a 448-pixel current image, five historical images spaced by about 1.8 m, and a 448 by 448 walked-path map spanning plus or minus 16 m. The output is a goal point followed by ten image-space points. The model is deployed with FP8 weights on a Jetson AGX Thor.

The paper factors the trace distribution into a goal decision and a path decision conditional on that goal.

$$p_\theta(\boldsymbol{\tau}\mid\mathbf{x},c)=p_\theta(\mathbf{g}\mid\mathbf{x},c)\,p_\theta(\boldsymbol{\tau}\mid\mathbf{g},\mathbf{x},c)$$

This factorization has a concrete purpose at intersections. Two commands such as “take the left fork” and “take the right fork” share much of the same visual context but require different terminal regions. A model that directly regresses a curve may average the two branches or choose a locally smooth path toward the wrong side. Factoring out the goal makes the side decision explicit while allowing the second factor to concentrate on visible walkable ground.

Training uses a weighted objective over all predicted points, the final point, and the language model's next-token likelihood. The endpoint term is weighted by $\lambda_g=2$, and the cross-entropy term by $\lambda_{\mathrm{CE}}=0.5$.

$$\mathcal{L}=\frac{1}{N}\sum_{i=1}^{N}\lVert\hat{\mathbf{u}}_i-\mathbf{u}_i\rVert_1+\lambda_g\lVert\hat{\mathbf{u}}_N-\mathbf{g}\rVert_1+\lambda_{\mathrm{CE}}\mathcal{L}_{\mathrm{CE}}$$

The first term keeps intermediate points close to the reference trace, the second prevents the curve from ending on the wrong side of a junction, and the third preserves the language model as a conditional feature extractor. This is not a generic auxiliary-loss recipe: the endpoint emphasis follows from the fact that image-space traces are often similar near the robot but diverge sharply at the decision point.

For efficient deployment, the predictor makes one language-model call and then applies a lightweight multilayer perceptron head. The head maps the normalized 4096-dimensional embedding to twenty coordinates, representing ten image points. LayerNorm, a 1024-unit hidden layer, GELU, and a residual structure are used.

$$\hat{\boldsymbol{\tau}}=(\hat{\mathbf{u}}_1,\ldots,\hat{\mathbf{u}}_N)=\sigma(h_\phi(\mathbf{e})),\qquad h_\phi:\mathbb{R}^{4096}\rightarrow\mathbb{R}^{20}$$

The model does not directly output motor commands. Its coordinate predictions remain in image space until stereo depth and camera geometry lift them to a local ground representation. This separation is useful because an image-space trace can be learned from street-level visual data without robot-specific dynamics, while the policy can adapt to the B2W's legs, suspension, and control limits.

Building Intent-Conditioned Traces at Scale

The central data problem is that one street image does not have one correct trajectory. A visible fork can support a command to continue straight, take the left branch, or take the right branch. Collecting only the trajectory that a particular human happened to walk teaches the model to reproduce data-collection bias. WayTrace therefore constructs several intent-conditioned alternatives around selected anchor images.

Figure 3: Data generation. Mapillary walks are reconstructed, scaled by GNSS arc length, and converted into historical traces; a VLM drafts commands and a single-image teacher renders intent-conditioned curves.

The source pool contains 51,907 Mapillary sequences and 15.7 million images. About 39,931 sequences satisfy the walking-speed filter. The team reconstructs walks with VGGT-Omega and scales each reconstruction using GNSS arc length. Past paths are projected into every frame, producing a visual history of how the camera reached its current pose. The final reconstructed set covers 26,740 episodes across 864 locations and contributes 288,000 samples.

A single-image teacher is first trained on 1,917 human traces over 605 images. Qwen3.8-Flash-Next then drafts multiple natural-language commands for each anchor image, about 3.4 on average. A VLM judge checks whether each generated curve is consistent with the corresponding command and retains approximately 94.5% of the proposals. The intent-conditioned branch produces 863,000 samples from 106,000 anchor images.

The hand-labelled or NaviTrace branch contributes 3,950 samples and is repeated ten times, giving 39,500 emphasized examples. The final mixture contains 1,190,594 samples from 182 cities or regions. The repetition is not evidence that the hand-labelled set dominates the dataset; it is a deliberate way to preserve scarce high-quality supervision while most coverage comes from automatically reconstructed and intent-expanded walks.

Figure 3 also exposes a limitation of the data strategy. The teacher and judge are themselves models, so their errors can enter the trace labels even if the consistency filter removes obvious mismatches. The paper reports the retention rate but not a detailed human audit of the retained intent-conditioned traces. The data-scale claim is therefore credible, while the exact label-noise profile remains uncertain.

From Image Trace to Robot Motion

The control layer uses ZED X stereo depth to lift the selected image points onto a local ground plane. Appendix A expresses the projected point as the camera or ray center minus a height vector, using a plane height of 1.5 m.

$$\mathbf{X}_j=\mathbf{C}_j-h\mathbf{n},\qquad h=1.5$$

The lifted points form a short 3D path. A carrot point is placed approximately 4.5 m ahead along that path. The reinforcement learning policy observes depth, robot state, path progress, and the carrot point, then produces base velocity commands at 5 Hz. The policy is not asked to select the next street or interpret the store name.

$$\Pi(K_t,T_t,\mathbf{X}_j)\rightarrow\mathbf{a}_t$$

Here $K_t$ is the robot state, $T_t$ is the current trace/progress context, and $\mathbf{X}_j$ is a lifted local waypoint. The trace is refreshed when a new command arrives, the robot has traveled about 3 m, roughly 40% of the current trace has been completed, or five seconds have elapsed. These conditions cover both semantic changes and geometric drift.

The physical platform is a Unitree B2W with a ZED X camera, lidar-inertial odometry, and GNSS. Trace prediction runs on a Jetson AGX Thor and the RL policy on an AGX Orin. The reported speed cap is 1.3 m/s. This is a reasonable urban sidewalk speed from a safety perspective, but it also means the system cannot rely on fast reactions to traffic signals or a tram entering the street.

Closed-Loop Evaluation

The simulation is a 0.91 square kilometer reconstruction of Zurich's old town. It contains 123 zebra crossings, 880 shops and restaurants, narrow streets, and stairs. Ten long-range missions are each executed three times. With perfect GNSS, WayTrace succeeds on 83.3% of trials. No baseline exceeds 20%.

The comparison should be read as a system-level test, not as proof that one base model is intrinsically stronger. The baselines were supplied with the goal representations their authors designed them to use, but they were not redesigned around the hierarchical interface. WayTrace's advantage comes from matching route, language, and trace to different spatial and temporal scales. A point-goal policy can follow a coordinate when localization is accurate, yet it has no mechanism for deciding that the coordinate belongs on the far side of a crosswalk.

Figure 4 shows that the ten simulated missions are spatially diverse. Some stay on one long sidewalk, while others cross bridges, climb stairs, or approach shops along narrow streets. This matters because a route-following system can look strong on a long straight corridor and fail as soon as it must choose between visually similar branches.

Figure 4: The ten simulation missions cover different routes, intersections, tram stops, crosswalks, and stairs in the reconstructed old town.

GNSS Degradation and Failure Modes

WayTrace does not eliminate GNSS dependence. As radial RMS error increases from 2 m to 5 m and 7 m, success falls from 83.3% to 63.3%. The important comparison is with point-goal methods using the same RL policy: at 7 m error those baselines fall to 23.3% and 10.0%. The hierarchy degrades, but its error does not propagate directly into every steering command.

Safety violations show the same pattern. WayTrace keeps crosswalk violations below 6% and road violations below 1% across the tested GNSS levels. Point-goal baselines become approximately ten times more likely to violate the intended walking geometry. A noisy route-step estimate can send the orchestrator to the wrong command, but the trace predictor still tends to place the local curve on visible walkable ground.

An oracle-orchestrator experiment raises WayTrace to 90.0%, which isolates the language layer as a major remaining bottleneck. Conversely, replacing the orchestrator with a worse model can make the trace predictor follow the wrong command very accurately. The system's interpretability helps a human diagnose this failure, but the interface itself does not guarantee semantic correctness. The authors report about a 5% critical error rate for the selected orchestrator.

The failure taxonomy includes stale GNSS distance, map omissions, and commands triggered at the wrong time. A route may remain topologically connected after a missed turn, so the robot can continue along a plausible sidewalk while moving away from the intended destination. Long missions compound these mistakes because a single early semantic error changes the visual context for every later decision.

Real-World Deployment

The system completed four city missions of increasing length. The first three required no intervention; the 1,368 m Mendokoro mission required two. Table II reports the exact distances, speeds, command counts, and success outcomes.

The Snipes mission illustrates the benefit of retaining a map but not tracking it geometrically. The route asks for a left turn in roughly 4 m. The satellite view suggests a turn at the intersection, while the robot camera shows a narrow pedestrian path. The orchestrator issues “turn left at the intersection,” the trace predictor places the endpoint across the near kerb, and the robot follows the curve across the zebra crossing. The map still provides the global objective, but the camera decides what a safe local turn looks like.

Figure 5: Three real-world routes. Magenta curves show predicted traces, orange markers show new commands, and the labels summarize each executed maneuver.

These deployments are valuable evidence that the complete system can run outdoors for hundreds of meters and interact with real curbs, crossings, narrow sidewalks, and storefronts. They are not a controlled comparison against the simulation baselines. Weather, pedestrian density, route familiarity, and operator assistance are not reported as a matched experimental design. The 97.2% result for the longest mission is therefore a deployment outcome rather than a generalizable real-world success rate.

Open-Loop and Component Analysis

On Human-Test, WayTrace obtains a NaviTrace score of 83.2, average displacement error of 5.0, and side accuracy of 97.9%. The best baseline reaches a NaviTrace score of 57.0. On the harder Walks-Test, WayTrace obtains 70.4, an error of 8.6, and side accuracy of 92.6%. The ranking is consistent, although the absolute performance gap shrinks when the visual domain is more varied.

The temporal-history ablation is one of the clearest component results. With five past frames and the walked-path map, three-way junction accuracy is 94.7%. With only the current frame, it falls to 74.7%. Figure 8 provides three examples in which the current image is ambiguous because the robot is already close to the junction or looking along a wall, while the history reveals which street the robot arrived from.

Figure 8: Orange traces use only the current frame; pink traces use temporal history. The historical path disambiguates left, right, and crosswalk decisions.

Intent data also produces monotonic gains. Adding intent-conditioned examples improves the NaviTrace score from 79.3 to 83.3 as the amount increases to 863,000 samples. This is evidence that the model benefits from controlled variation in language intent, not just more images of streets. The same visual fork appears with different commands, forcing the representation to retain the language rather than relying only on scene appearance.

The authors report that supervised fine-tuning outperformed GRPO when the training set reached its final scale. This result should be read narrowly. It says that their reward formulation and data mixture did not turn GRPO into a reliable improvement at this stage. It does not establish that reinforcement learning is unnecessary for trace prediction in general, especially when safety constraints or human preferences are directly optimized.

Orchestrator Benchmark and Language as an Interface

The authors evaluate five vision-language orchestrators over 957 decisions. Qwen3.8-Flash-Next makes critical errors on 5.1% of decisions, takes about 1.4 seconds, and runs locally. Gemini 3.5 Flash-Lite has a 5.9% critical-error rate, takes 5.1 seconds, and costs roughly $2.9 per thousand calls. Gemini 3.8 Flash and GPT-6 Sol both reach 3.2%, but take about 6.5 and 4.5 seconds and cost roughly $7.4 and $7.1 per thousand calls.

The choice of the local model is therefore an operational tradeoff rather than a claim of best accuracy. The two proprietary models with lower error rates are slower and more expensive, while the system needs repeated commands over long routes. A 5.1% error rate also compounds: if one bad turn changes the next observation, later commands may be internally consistent with the wrong place. The evaluation reports per-decision errors but does not provide a full closed-loop error-propagation model.

Figure 7 demonstrates why the language interface has value beyond web-map playback. In the closed-street case, the trace avoids a fence and the orchestrator recomputes the route. In the missed-turn case, an injected command loss makes the robot continue straight for about 10 m before the next command and route correction arrive. In the human-directions case, a person guides the robot through a 127 m mission with no web map, route, or GNSS.

Figure 7: Three applications of the language interface: rerouting around a closed street, recovery after a dropped turn command, and following spoken human directions without a map.

The human-directions case is particularly important because it changes the source of high-level intent while leaving the onboard trace model unchanged. This modularity would be difficult to obtain if the route planner directly emitted an image goal for the predictor. The cost is that the command vocabulary and timing logic must remain precise enough for the trace model to interpret consistently.

What the Work Establishes

WayTrace establishes a credible systems argument for explicit interfaces in outdoor embodied navigation. The route layer answers where the robot should go at city scale. The language layer answers what maneuver is appropriate from the current view. The trace layer answers where the maneuver lies in the image. The control layer answers how the B2W should move over the next few seconds.

The empirical evidence supports three narrower claims. First, the hierarchy is substantially better than the tested baselines on long simulated missions. Second, it degrades more gracefully than point-goal control when GNSS noise increases. Third, the language-conditioned trace predictor benefits from intent diversity and temporal history. The four real deployments add feasibility evidence, but not a statistically strong real-world comparison.

The paper also reframes interpretability. A natural-language command is useful not because a generated sentence sounds plausible, but because a human can audit the semantic decision independently from the local trajectory. If the command is wrong, the downstream trace may still look smooth and visually reasonable. If the trace is wrong, the controller may still execute it precisely. The hierarchy makes this failure chain visible instead of hiding it inside one model.

Limitations

The authors identify command-timing and junction errors as major failure sources. Stale GNSS distances and map omissions can make an otherwise correct command occur too early or too late. History covers only about 9 m, and the traces can cross low obstacles. The system uses only a front-facing camera, so it cannot inspect hazards to the side or behind. End-to-end command latency is around three seconds, which is too slow for reacting to red lights, trams, or a pedestrian stepping into the path. The robot speed is capped at 1.3 m/s.

The authors also state that approximately 5% critical orchestrator errors can compound over long routes. Their system improves auditability but does not fully solve long-horizon intent verification. A command can be locally sensible and still move the robot away from the user's destination. The paper does not demonstrate a robust automatic method for detecting that the active route has become semantically inconsistent with the original request.

Independent limitations remain. Four real missions cannot establish cross-city generality, and the longest route required two interventions. The simulation includes streets and stairs but not the full complexity of dynamic traffic, dense crowds, construction, or unusual lighting. The work is concentrated in Zurich-like urban environments. The code, data, trained models, and full controller configuration were not public at the time of analysis, which prevents external reproduction of the reported pipeline.

The architecture also depends on a remote model running on four H100 GPUs. The onboard components remain functional if the remote service disappears, but only for commands already issued. A network outage prevents new semantic route decisions. This is a meaningful deployment constraint for a robot expected to operate continuously in a city, even if the local trace and control layers are fully onboard.

Conclusion

WayTrace is best understood as a representation hierarchy rather than a single navigation model. It preserves the global reach of web maps, uses language to express the current maneuver, converts the maneuver into a trace that a local policy can execute, and avoids forcing any one module to solve all scales at once. The results show a large simulation advantage, meaningful robustness to GNSS error, and successful outdoor deployments over routes from 248 m to 1,368 m.

The next research step is not simply a larger vision-language model. It is a stronger contract between layers: better timing checks, longer visual memory, wider perception, explicit destination consistency, and a recovery policy that can detect when a locally valid command no longer serves the global goal. WayTrace provides a practical starting architecture for that agenda.