HA-VLN 2.0: An Open Benchmark and Leaderboard for Human-Aware Navigation in Discrete and Continuous Environments with Dynamic Multi-Human Interactions

Paper: HA-VLN 2.0: An Open Benchmark and Leaderboard for Human-Aware Navigation in Discrete and Continuous Environments with Dynamic Multi-Human Interactions

Authors: Yifei Dong, Fengyi Wu, Qi He, Lingdong Kong, Heng Li, Minghan Li, Zebang Cheng, Yuxuan Zhou, Jingdong Sun, Qi Dai, Alexander G. Hauptmann, and Zhi-Qi Cheng

Full text: https://arxiv.org/abs/2503.14229

Official code: https://github.com/UWMILab/HA-VLN. The inspected repository uses the MIT license, and this article was checked against commit 4e1d1fe4ceebee7253d3ba53c04468a8bc2c16c8.

One-Sentence Summary

HA-VLN 2.0 turns dynamic people from incidental scenery into a first-class part of vision-and-language navigation by combining a unified discrete/continuous task, a 486-motion human activity corpus, 16,844 socially grounded instructions, collision-aware metrics, reference agents, a public leaderboard, and real-robot validation.

Why Existing VLN Benchmarks Are Not Enough

Vision-and-language navigation asks an embodied agent to follow a natural-language route while using egocentric observations. The classic formulation is already difficult: language is under-specified, views are partially occluded, and small heading errors can accumulate. Most benchmarks nevertheless make one simplifying assumption that removes a central real-world difficulty: the environment is treated as a static collection of rooms, furniture, and viewpoints. If people appear at all, they tend to be fixed meshes or generic obstacles.

That assumption changes the problem in a way that aggregate success rate does not reveal. A robot in a corridor is not merely avoiding geometry. It is interpreting whether a person is walking toward it, standing in conversation, waiting by a door, or about to turn. The appropriate action may be to slow down, pass on a particular side, wait, or choose a longer route. A path that reaches the goal while forcing a person to step aside should not be considered equivalent to a path that preserves comfortable spacing and predictable movement.

The paper identifies three connected gaps. The first is social awareness. Conventional VLN scoring rewards arrival and path efficiency but has no explicit penalty for invading personal space. The second is linguistic granularity. Instructions such as “turn left and pass the chair” describe geometry, whereas a deployable robot may need to understand “go past the person taking a phone call on the left.” The third is dynamics. A crowd can close a passage after the agent has committed to a route, so the planner must reason under partial observability and revise its plan online.

HA-VLN 1.0 made an important early attempt by inserting humans into navigation, but its motion library contained language-motion alignment errors, its evaluation leaned heavily toward viewpoint jumps, and most scenes did not support rich interaction among several people. HA-VLN 2.0 broadens the intervention: it rebuilds the motion data, connects activities to scene regions, supplies both discrete and continuous simulators, rewrites instructions around human behavior, introduces safety-aware metrics, and checks whether policies survive on a physical robot.

The motivating claim should therefore be read literally: adding moving meshes is insufficient. In the paper’s ablation, replacing people with cylinders makes the benchmark easier, and removing interaction or movement enrichment also raises success. Human motion changes visibility, traversability, and the meaning of language. It creates a task that cannot be solved by geometric collision avoidance alone.

A Shared Interface for Discrete and Continuous Navigation

The benchmark does not claim that discrete viewpoint graphs and continuous control are physically identical. Instead, it gives them a common semantic interface so that task definitions, observations, scenarios, and safety metrics can be compared without hiding their execution differences. At time step $t$, the agent state is represented as

$$s_t=\langle \mathbf p_t,o_t,\Theta_t^{\mathrm{FOV}}\rangle,$$

where $\mathbf p_t$ is the three-dimensional position, $o_t$ is orientation, and $\Theta_t^{\mathrm{FOV}}$ is the egocentric visual observation. The state does not expose every human trajectory. People can be outside the field of view, hidden by furniture, or change direction between observations, making the task a partially observable decision process rather than a fully observed path-following exercise.

Both regimes use the same action vocabulary:

$$\mathcal A=\{a_{\mathrm{forward}},a_{\mathrm{left}},a_{\mathrm{right}},a_{\mathrm{up}},a_{\mathrm{down}},a_{\mathrm{stop}}\}.$$

In the continuous simulator, forward movement and rotation have explicit physical increments, normally $0.25\,\mathrm m$ and $15^\circ$. In the discrete simulator, the agent moves between navigable viewpoints. The shared vocabulary allows a policy to express the same high-level intention, but evaluation must still report how often collision is checked and how large each movement is. The paper’s step-size experiment shows that an apparently stronger result can be produced by allowing a long action to jump through a moving human and checking only the endpoint.

The human-aware constraint is also stated per regime rather than treated as one universal distance. Discrete evaluation uses a three-meter proximity threshold, while continuous evaluation detects overlap through the robot and human bounding radii. These are operational proxies designed to expose unsafe behavior in both regimes. They should not be mistaken for a complete theory of social distance, since acceptable spacing varies with culture, context, relative motion, and the social relationship between people.

HAPS 2.0: Building a More Trustworthy Human Activity Corpus

HAPS 2.0 contains 486 validated human motion sequences, 172 activity classes, roughly 58,000 frames, and more than 430 hours of annotation effort. It spans 26 region types, 90 building scans, 910 human models, and 428 annotated regions. The corpus includes 111 outdoor human models, 72 two-person interactions, 59 three-person groups, 15 four-person groups, and 268 complex motions such as stair climbing. These numbers matter less as a size contest than as evidence that the benchmark covers object use, social grouping, and movement patterns that a single-person walking cycle cannot provide.

An activity is represented by a sequence of SMPL parameters over time:

$$\mathcal H=\langle h_1,h_2,\ldots,h_{120}\rangle,\qquad h_k\in\mathbb R^{P}.$$

Here, $h_k$ describes body pose, shape, and global displacement at frame $k$, and the released tensor also contains mesh vertices. The simulator cycles through a 120-frame sequence while the navigation episode continues. A asynchronous producer-consumer design sends refresh signals from one thread and updates rendered human meshes in the other. Outdated meshes are discarded, and the paper reports retrieval delays below 50 ms. On a 24 GB GPU, scenes with up to about ten people run at 30 to 60 frames per second.

The upgrade from HAPS 1.0 is not just additive. In the reported quality study, the action count rises from 435 to 486, action accuracy from 6.3 to 8.5, and action-environment compatibility from 5.9 to 8.1. Failed actions fall from 120 to zero. The authors also remove region-action pairs that do not make sense, then rebuild the mapping between activity descriptions and spatial context. This is important because a plausible walking animation in the wrong room teaches the agent a false relation between language, object, and behavior.

HAPS version	Actions	Accuracy	Compatibility	Failed actions	Annotation hours
HAPS 1.0	435	6.3	5.9	120	320
HAPS 2.0	486	8.5	8.1	0	430+

From Region Semantics to Physically Valid Placement

A motion becomes useful only when it is placed coherently in a scene. The annotation pipeline therefore begins with a coarse stage. A region $R$ has a label, bounding coordinates, and an object set $\mathbf O=\{j_1,j_2,\ldots,j_n\}$ with known object positions. Motions whose semantics match the region are retained. Each remaining motion $h_i$ is paired with an object $j_i$, and particle swarm optimization searches for a position that keeps the activity near the relevant object without crossing walls, floating above the floor, or intersecting other geometry.

The objective can be summarized as a semantic-distance term plus constraint penalties:

$$f(\mathbf p)=d(\mathbf p^h,\mathbf p^{j_i})+P_{\mathrm{constraints}}(\mathbf p),$$ $$\mathbf p^{h_i}_{\mathrm{opt}}=\arg\min_{\mathbf p\in\mathcal R_i}f(\mathbf p).$$

The paper uses a one-meter minimum clearance from other objects as one concrete constraint. This stage is not presented as a substitute for human judgment. It eliminates obviously invalid placements so that scarce annotation time is spent on ambiguous cases. The distinction is methodologically important: automated optimization supplies candidates, while human verification determines whether the semantics and visual contact are credible.

The fine stage uses nine cameras around each human model. For the eight lateral cameras, the horizontal angle is

$$\theta_{\mathrm{lr}}^i=\frac{\pi i}{8},\qquad i=1,\ldots,8,$$

and the vertical angle alternates by camera index:

$$\tan\theta_{\mathrm{ud}}^i=\begin{cases}0,&\text{if }i\text{ is odd},\\[4pt]\dfrac{\Delta_z}{\sqrt2\,\epsilon},&\text{if }i\text{ is even}.\end{cases}$$

The ninth camera is overhead, with zero horizontal angle and a $90^\circ$ vertical view. Annotators can therefore inspect foot-floor contact, side silhouettes, and top-down overlap at the same time. The procedure targets failures that are easy to miss in a single render: a model hovering above the floor, a hand passing through a chair, or a body leaning into a wall.

Multi-Human Enrichment and Real-Time Simulation

The first annotation pass produces valid individual activities. A second, human-in-the-loop pass adds characters and interactions around an existing activity. The system provides a language model with regional context, including objects within six meters, and asks for plausible social scenarios. Four rounds of manual review then remove physically impossible or semantically weak combinations. This creates groups with conversations, shared meals, queuing, and other activities that cannot be represented by independently walking avatars.

The resulting scene is dynamic but bounded. The paper describes up to about ten concurrent humans in a scene, with a larger pool of 910 models distributed across 428 regions. This distinction matters for interpretation. The benchmark supports many models overall and local multi-human interaction during an episode, but it does not simulate an entire open public space with continuous arrival and departure over hours. Its contribution is controllable local density, not unrestricted crowd simulation.

Collision handling closes the loop. The simulator exposes human state, updates the scene when a human advances to another frame, and checks whether a proposed robot move causes bounding volumes to overlap. A collision triggers a revert in the execution loop. In continuous mode, the agent can use small actions to correct heading and lateral clearance. In discrete mode, people continue moving while the agent jumps between viewpoints, so the task remains dynamic even though the control interface is coarse.

The official implementation reflects these claims. HASimulator/environments.py manages the signal queue and frame index, including the 120-frame loop. HASimulator/measures.py records collision counts and step details, while HASimulator/metric.py converts raw counts into benchmark-level metrics after subtracting the collisions expected from the annotated route. The paper-to-code correspondence is direct rather than a conceptual analogy.

HA-R2R: Instructions That Refer to What People Are Doing

HA-R2R extends the continuous R2R-CE routes with socially grounded language. It contains 16,844 instructions, split into 10,819 training, 778 validation-seen, 1,839 validation-unseen, and 3,408 test examples. The unseen test set uses 18 buildings absent from training and deliberately increases the proportion of routes involving multiple people. The instruction does not merely append “watch out for pedestrians.” It may identify an activity, locate it along the route, and constrain how the agent should pass or where it should stop.

The language distribution changes substantially. Average instruction length rises from 27 words in R2R-CE to 112 words in HA-R2R, while vocabulary grows from 2,616 to 5,032. Most instructions contain between 20 and 60 percent human-related content. Longer text is not automatically better, but here the extra length carries task-relevant distinctions: who is present, what they are doing, whether they form a group, and where the robot can move without disrupting them.

Generation uses ChatGPT-4o and LLaMA-3-8B-Instruct for few-shot expansion, followed by iterative human editing. The prompts request neutral descriptions that can be checked against the scene. Human reviewers remove invented intentions and unsupported mental states. The pipeline should therefore be read as model-assisted authoring with manual control, not as an unvalidated dump of synthetic instructions. Even so, the paper leaves room for more analysis of inter-annotator agreement and cultural variation in social norms.

The dataset also quantifies how strongly people affect the route. About 35.5 percent of paths intersect a human motion trajectory directly, and 49.7 percent of viewpoints are indirectly affected by visible people. Movement-distance statistics show a mixture of small adjustments and larger relocations: 22.4 percent move no more than 0.5 m, 37.3 percent move between 0.5 and 1 m, 25.0 percent between 1 and 1.5 m, 11.6 percent between 1.5 and 2 m, and 3.7 percent more than 2 m. Motion is distributed throughout the episode rather than concentrated at the start or goal.

Two Reference Agents with Different Hypotheses

HA-VLN-VL tests how far stronger visual-language alignment can go with a comparatively streamlined architecture. It is adapted from recurrent VLN-BERT and uses imitation learning instead of actor-critic training. At each step, the previous state and action are combined, then a Transformer with a dedicated state token attends to visual and language tokens:

$$s_t=[s_t';a_t]W_s,$$ $$A_{s,v}^{t}=\operatorname{Softmax}\left(\frac{Q_sK_v^{\top}}{\sqrt{d_h}}\right).$$

Pooling over the attended representation produces an action distribution. The design isolates the contribution of multimodal grounding: if performance improves, it is more likely to come from aligning the current visual options with long, activity-rich instructions than from a more elaborate recurrent control stack.

HA-VLN-CMA makes the opposite bet. It emphasizes cross-modal attention, history, and adaptation under partial observability. RGB and depth streams are encoded separately and merged with a directional representation:

$$v_i=[v_i^{\mathrm{rgb}}W_{\mathrm{rgb}};v_i^{\mathrm{depth}}W_{\mathrm{depth}};d_i]W_{\mathrm{merge}}.$$

Language and visual tokens then interact through multi-head attention, and the fused state is updated recurrently:

$$\hat f_t=\operatorname{MultiHeadAttn}(v_t,l),$$ $$f_t=\operatorname{GRU}([(v_t,l),a_{t-1}],f_{t-1}),$$ $$a_t=\operatorname{softmax}(\operatorname{MLP}(f_t)).$$

The repository implements the two attention directions explicitly. In agent/VLN-CE/vlnce_baselines/models/cma_policy.py, a state query attends over text memory, and the resulting text embedding queries RGB and depth memories. The variable used in the source is text_state_k. This matters because a superficial diagram can name a nonexistent tensor and create a misleading connection between method and implementation.

text_state_q = self.state_q(state)
text_state_k = self.text_k(instruction_embedding)
text_embedding = self._attn(text_state_q, text_state_k, instruction_embedding)
rgb_embedding = self._attn(self.text_q(text_embedding), rgb_k, rgb_v)
depth_embedding = self._attn(self.text_q(text_embedding), depth_k, depth_v)

The discrete planner shows how physical goals are translated into the shared action vocabulary. It uses a 15-degree turn, a 0.25-meter forward step, a 0.13-meter goal radius, and a 200-step limit. This does not mean discrete and continuous control are interchangeable. It reveals that the implementation deliberately retains fine-grained physical actions while changing how the next observation is selected.

The Navigation and Evaluation Loop

flowchart LR
    A[HA-R2R instruction] --> B[RGB-D observation]
    B --> C[Visual-language grounding]
    C --> D[Action selection]
    D --> E[Advance human motion]
    E --> F[Bounding-volume collision check]
    F -->|safe| B
    F -->|collision| G[Stop or replan]
    F --> H[Count TCR and CR]
    D --> I[Goal-distance check]
    I --> J[Compute NE and socially safe SR]
    

This loop contains three time scales. Human motion advances on its own frame schedule, the agent selects an action at its control period, and collision is checked after the proposed movement. A continuous controller can inspect a short displacement, while a discrete viewpoint hop may skip over space that would matter physically. The paper explicitly tests this discrepancy instead of assuming that a common action label guarantees equal execution.

The Total Collision Rate subtracts collisions that are expected from the annotated route before aggregating excess safety violations:

$$\operatorname{TCR}=\frac{\sum_{i=1}^{L}(c_i-|A_i^c|)}{L}.$$

Here, $c_i$ is the set of time steps in episode segment $i$ that enter the collision range around a person, and $A_i^c$ is the set of human activities that the reference route is expected to encounter. The subtraction prevents an agent from being punished merely because a valid route must pass near people. TCR therefore asks whether the agent creates collisions beyond those required by the task.

The Collision Rate is an episode-level indicator normalized by the proportion of human-affected instructions:

$$\operatorname{CR}=\frac{\sum_{i=1}^{L}\min(c_i-|A_i^c|,1)}{\beta L}.$$

Navigation error and success are defined as

$$\operatorname{NE}=\frac{\sum_{i=1}^{L}d_i}{L},$$ $$\operatorname{SR}=\frac{\sum_{i=1}^{L}\mathbb I(c_i-|A_i^c|=0)}{L}.$$

where $d_i$ is final distance to the goal and $\beta$ controls the proportion of human-affected cases used in the CR normalization. The important design choice is that SR is not pure arrival. An episode must reach the goal and avoid excess collisions. This prevents a fast but socially unsafe policy from looking successful. The implementation in HASimulator/metric.py follows the same logic: TCR is clipped at zero after subtracting annotated collisions, CR is a zero-or-one indicator, and SR multiplies task success by the no-excess-collision condition.

Continuous Navigation Results

The continuous unseen split separates navigation accuracy from social compliance. HA-VLN-VL reaches NE 5.35, TCR 6.63, CR 0.59, and SR 0.14. The fully adapted HA-VLN-CMA variant reaches NE 6.23, TCR 8.10, CR 0.69, and SR 0.10. BEVBert is stronger overall in this table with NE 5.51, TCR 4.71, CR 0.55, and SR 0.21. ETPNav reaches SR 0.17 but incurs TCR 6.94 and CR 0.58.

Agent	NE (m) lower	TCR lower	CR lower	SR higher
HA-VLN-CMA-Base	7.34	47.06	0.77	0.07
HA-VLN-CMA-DA	7.00	27.25	0.69	0.09
HA-VLN-CMA*	6.23	8.10	0.69	0.10
HA-VLN-VL	5.35	6.63	0.59	0.14
BEVBert	5.51	4.71	0.55	0.21
ETPNav	5.43	6.94	0.58	0.17

These results do not support a simple claim that the newly proposed baselines dominate established models. BEVBert is the strongest all-around result on the listed unseen metrics. The value of the comparison is to show that mature navigation policies degrade sharply when human dynamics and safety constraints are made explicit, and that stronger visual-language grounding does not automatically solve social compliance. It also gives future methods a reproducible target rather than an isolated demonstration video.

Discrete Results and the Hidden Cost of Arrival

The discrete results expose a failure mode that standard SR can hide. Speaker-Follower drops from SR 0.35 on conventional VLN to 0.21 on HA-VLN-DE, with CR 0.72. Airbert reaches SR 0.36 but CR 0.83. Rec with PREVALENT and OSCAR remains between SR 0.33 and 0.34, while CR stays above 0.78. Many episodes end near the goal, yet the route includes socially costly interactions.

Discrete agent	Standard NE	Standard SR	HA-VLN-DE NE	TCR	CR	HA-VLN-DE SR
Speaker-Follower	6.62	0.35	7.44	0.32	0.72	0.21
Rec, PREVALENT	3.93	0.63	6.12	0.29	0.81	0.33
Rec, OSCAR	4.29	0.59	6.12	0.28	0.78	0.34
Airbert	4.01	0.62	5.54	0.30	0.83	0.36
NavCoT	6.26	0.40	6.83	0.36	0.85	0.23

The difference between discrete and continuous collision magnitude has a methodological explanation. A discrete hop can cover a larger spatial interval, so endpoint checking can undercount interactions that would occur during physical motion. The continuous setting uses smaller steps and overlapping bounding volumes, making local avoidance more visible. This does not prove that continuous agents are universally safer, but it shows why raw leaderboard entries need regime labels and comparable collision protocols.

Ablations: People, Interaction, and Step Size Change the Task

The human-presence ablation is the clearest evidence that the benchmark measures more than static obstacle avoidance. With complete human models and enrichment, BEVBert records NE 6.10, TCR 5.72, CR 0.56, and SR 0.15. Keeping people but removing interaction enrichment changes the numbers to NE 6.32, TCR 5.11, CR 0.46, and SR 0.17. Replacing humans with cylinders and removing enrichment lowers TCR to 3.25 and raises SR to 0.19. ETPNav shows the same direction, moving from SR 0.08 in the full condition to 0.10 and then 0.14 as human structure is simplified.

Condition	Environment	Agent	NE	TCR	CR	SR
Humans + enrichment	CE	BEVBert	6.10	5.72	0.56	0.15
Humans, no enrichment	CE	BEVBert	6.32	5.11	0.46	0.17
Cylinders, no enrichment	CE	BEVBert	6.13	3.25	0.35	0.19
Humans + enrichment	CE	ETPNav	7.40	7.94	0.71	0.08
Humans, no enrichment	CE	ETPNav	7.35	6.12	0.63	0.10
Cylinders, no enrichment	CE	ETPNav	7.75	4.47	0.53	0.14

Step size produces an even sharper warning. BEVBert with the default 0.25 m action has SR 0.21 on unseen data. A 1.0 m action reports SR 0.26 when only the endpoint is checked, but splitting that same 1.0 m displacement into four 0.25 m sub-steps and checking each one drives success close to zero. The larger action is not better navigation; it can pass through a moving person between collisions checks. This experiment is one of the strongest reasons the paper insists on regime disclosure.

Sensor ablations show that both modalities provide complementary information. BEVBert falls from SR 0.21 with RGB-D to 0.15 without depth, while removing RGB leaves SR 0.20. ETPNav drops from 0.17 to 0.13 without depth and to 0.16 without RGB. Depth is especially useful for estimating distance and detecting occluding furniture, but RGB contributes activity semantics and visual cues that help choose a socially plausible route.

Cross-Domain Transfer and Real-Robot Validation

The benchmark also tests whether human-aware training changes behavior outside its own database. BEVBert trained on VLN-CE achieves SR 0.37 on its usual validation setting but 0.21 after transfer to HA-VLN-CE. In the reverse direction, training with the social corpus reaches SR 0.27 on standard R2R-CE, close to the 0.29 result from training directly on R2R-CE, while improving relative performance on the new benchmark. The result is not a claim of solved generalization, but it suggests that social training retains useful navigation ability rather than memorizing one scene distribution.

The real experiment uses a Unitree Go2-EDU with an Intel RealSense D435i, a MID360 3D LiDAR, an IMU, a Jetson NX, and a Raspberry Pi 4B. Jetson receives camera images and runs policy inference; the Raspberry Pi receives high-level commands such as forward and turn; the IMU calibrates orientation. The robot rotates its camera to obtain panoramic context and executes 0.25 m forward movements with 15-degree turns. The test covers offices, living rooms, hallways, and lobbies, with three instances of each space type and 30 episodes per instance. The human condition uses two to four freely moving volunteers.

Agent and training data	No-human SR	With-human SR
HA-VLN-CMA-Base, VLN-CE trained	0.26	0.09
HA-VLN-VL, VLN-CE trained	0.40	0.12
HA-VLN-CMA-Base, HA-VLN trained	0.25	0.15
HA-VLN-VL, HA-VLN trained	0.44	0.18

Every policy becomes worse when people are present. The more important result is the training transfer: HA-VLN-VL rises from 0.12 to 0.18 with human-aware training, and HA-VLN-CMA-Base rises from 0.09 to 0.15. The absolute rates remain low, especially in social conditions, but the direction is consistent with the simulation ablations and shows that HAPS 2.0 and HA-R2R affect physical robot behavior rather than only virtual leaderboard entries.

What the Code Adds to the Paper

The repository is a complete research stack rather than a landing page with videos. The simulator, HAPS data tools, HA-R2R utilities, two baseline agents, and evaluation scripts are present. The relevant code paths are HASimulator/environments.py for human update and rendering, HASimulator/measures.py for collision bookkeeping, HASimulator/metric.py for TCR, CR, and SR, and agent/VLN-CE/vlnce_baselines/models/ for instruction encoding and cross-modal attention.

The source also clarifies an implementation boundary. The discrete planner’s local geometry assumes free space when proposing candidate steps; the learned policy and simulator-level collision system perform the actual obstacle handling. A reader should not interpret the planner’s shortest-distance candidates as evidence that social avoidance is solved by geometry alone. The method depends on the combined loop of dynamic people, RGB-D observation, learned action selection, and collision checking.

Another useful implementation detail is that the reported metrics are not generic episode averages. metric.py loads the annotated collision count for each episode, subtracts it from the actual collision count, clips TCR to zero, defines CR as a collision indicator, and makes SR conditional on zero excess collisions. This is a small piece of code with a large conceptual consequence: “success” means socially safe arrival, not arrival at any cost.

Limitations and Independent Assessment

The first limitation is acknowledged by the authors in the real-robot discussion. Narrow passages, intersections, and sudden crowding still produce collisions. Volunteers sometimes stand up or change direction without enough warning for the policy to replan. The system is therefore weakest in exactly the situations where human motion is least predictable and local clearance is smallest. A low average safety score would not prove readiness for shared public space.

The second limitation is scale. Four indoor space types, three instances each, 30 episodes per instance, and two to four volunteers provide useful evidence but do not reproduce a busy station, hospital corridor, school, or crowded exit. The maximum real-robot SR in the human condition is 0.18. This is a directional signal, not a deployment result. Larger studies should include children, wheelchair users, older adults, opposing flows, and groups that move together.

Third, the simulator’s local crowd is bounded. Roughly ten concurrent humans and 120-frame motion cycles support controlled interaction, but not long-duration arrival and departure or open-world crowd density. The human models are realistic enough to expose navigation failures, yet they remain sampled motion sequences rather than agents who respond causally to the robot. A person may continue walking through a space even when a real pedestrian would slow down or yield. This can change whether a route is considered socially appropriate.

Fourth, one-meter proximity and overlapping volumes are useful reproducible proxies, but personal space is not equivalent to all social norms. People communicate intention with gaze, body orientation, hesitation, gestures, and verbal cues. The benchmark does not yet score whether the robot’s motion is legible, whether it yields at the right time, or whether a human felt comfortable. A collision-free trajectory can still be awkward, surprising, or unnecessarily dominant.

Fifth, the social instruction pipeline combines language models with human review. The paper reports accuracy, compatibility, and annotation effort, but does not resolve every question about annotator agreement, cultural variation, or subtle template effects. A socially grounded benchmark should eventually measure whether different communities interpret the same distance and yielding behavior in the same way. Otherwise the leaderboard may encode one local convention as a universal norm.

Finally, comparison across agents must respect training budgets. HA-VLN-CMA, HA-VLN-VL, BEVBert, ETPNav, and vision-language models are not trained under identical resource or pretraining conditions. Some results are retrained on HA-R2R, while others are zero-shot or adapted from another benchmark. The paper separates these settings, but readers should keep the condition attached to every number instead of ranking all SR values as if they measured the same capability.

What Comes Next

The benchmark suggests several concrete research directions. A natural next step is to combine trajectory prediction or learned pedestrian priors with the language-conditioned policy, then test whether lower TCR is possible without sacrificing SR. Another is to make uncertainty over occlusion and sudden appearance explicit, so the agent learns when to slow down, wait, or choose a longer route. A third is to expand evaluation beyond geometric personal space toward legible yielding and culturally variable norms.

For robotics teams, HA-VLN 2.0 provides a staged debugging method. Discrete navigation can screen language grounding and coarse route choice, continuous navigation can expose local collision and control errors, and a small physical test can reveal sensor timing, actuator mismatch, and human unpredictability. Every stage should report NE, TCR, CR, and SR together, along with step size and collision-check frequency. Otherwise a leaderboard gain may be an artifact of a coarser action interface.

Conclusion

HA-VLN 2.0’s central contribution is to make the person part of the task specification. HAPS 2.0 supplies more reliable and varied multi-human motion, HA-R2R encodes activities and avoidance requirements in long instructions, the unified task supports both discrete and continuous control, and the metrics separate arrival from social compliance. The real-robot study adds an important but sobering check: training on human-aware data helps, yet success remains low when people are actually moving around the robot.

The benchmark therefore does not claim that human-aware navigation is solved. Its value is to make the remaining problem measurable: whether an embodied agent can interpret a route that depends on people, predict their effect on traversability, and reach the goal without turning arrival into a collision. That is a stronger target than static-map navigation, and a useful foundation for the next generation of safe embodied systems.

Golden line: Human-aware navigation succeeds only when reaching the goal still leaves room for the people along the way.