EgoWitness: Benchmarking Instance‑Level Spatial Memory and Reasoning in Egocentric Video Streams

Annotated egocentric clips with object masks and room labels
Landmark and movable-object masks across seven days. Select a tile to play its clip.
Zhengru Fang1,*Zehao Liu2,*Mingyu Tsoi1Yaoyu Zhu3Junlong Li4Fengmao Lv5Sijie Cheng2,6Yuanliang Sun6Fang Yang2Jun Zhang1
1The Hong Kong University of Science and Technology2Tsinghua University3Southeast University4The Hong Kong Polytechnic University5Southwest Jiaotong University6RayNeo

* Equal contribution

8,453human-verified questions
~324 hweek-long and short-horizon video
1,034,075masks of 34 landmarks and 21 movable-object labels
17,269room intervals over 17 named rooms and zones
7abilities, from object identity to room visits

Abstract

Recalling where and when an object was last seen and who saw it requires recognizing it across days and people. Existing long-video benchmarks focus on events, object categories, or relative locations, leaving instance-level spatial memory largely untested in multimodal models. To address this gap, we present EgoWitness, a benchmark for instance-level spatial memory and reasoning in egocentric video. Its 8,453 human-verified questions span approximately 324 hours of week-long and short-horizon recordings and test seven abilities, from recognizing an object or room to recalling who saw it and reasoning about room visits. Over one million reviewed masks of 34 registered landmarks and 21 movable-object labels are linked through room timelines. Among twelve direct vision-language models (VLMs), open-weight models reach at most 46.97% accuracy, and none exceeds 35% on identifying who saw an object last. We therefore propose EgoClue, a spatial-memory agent that retrieves candidate intervals, verifies instance identity, and computes spatial and temporal relations with five tools. Two-stage training with supervised fine-tuning (SFT) and Group Relative Policy Optimization (GRPO) raises its 4B controller from 23.64% to 72.65% accuracy on 1,320 held-out questions. Evidence-trace synthesis matches the best direct VLM at 74.85% on week-long questions and achieves 80.56% in environments excluded from training.

Overview figure: instance-reference comparison, historical observations with wearer, time and room context, and seven QA abilities
Instance-level spatial memory across egocentric streams. (A) Instance-reference comparison. (B) Historical observations with wearer, time, and room context. (C) Seven QA abilities in two groups: object identity and memory, and room identity and visit reasoning.

Annotations

A week of six wearers, annotated room by room and object by object

The week-long component builds on EgoLife: six wearers share one house for seven days in 32,001 source videos. We add 17,269 room intervals over 17 named rooms and zones, covering all recorded wearer-hours, and 1,034,075 masks of 34 registered landmark instances and 21 movable-object labels.

Room timelines: VLM proposals and human review

A VLM first proposes the wearer’s room from dense captions of the 17 shared spaces. Annotators then review and correct the proposed intervals in the Online Room Curator System, yielding 17,269 room intervals that cover all recorded wearer-hours. Each tour replays a stretch of the week at high speed against both timelines.

Object masks across the week

recorded footage, 09:00 to 24:00annotated clip, click to play

Clips show the 976×976 center crop used for detection, with landmark masks at 1 Hz, movable-object masks at 5 Hz, and the wearer’s room from the room timeline.

Layered annotation and review

Annotation pipeline from room timelines and landmark locations to SAM 3.1 mask tracks, identity review, and evidence-linked questions
From reviewed observations to evidence-linked questions. (a) Room timelines and landmark locations supply spatial context. (b) SAM 3.1 tracks Gemini- and human-checked keyframes. (c) Annotators resolve instance identity using reference views and room context. (d) Accepted observations fix question facts before wording.
Annotation workspaces: the Online Room Curator System and the instance association workspace
Annotation workspaces. (A) The Online Room Curator System supports boundary editing, room-instance assignment, landmark registration, and visual verification. (B) Annotators compare landmark and movable-object instances, inspect logical conflicts, and correct associations before export. Room-boundary conflicts return to A for interval correction.

Room observations across wearers and days

Room observations for six wearers over seven days on a 09:00 to 24:00 clock, colored by 17 named rooms and zones
Each row is one wearer and each column one day, on a common 09:00 to 24:00 clock. Colors identify 17 named rooms and zones, and white gaps mark times without recordings. Click any region to enlarge it.

Benchmark

Seven abilities in two groups

Each question names an object or room, public references, and a wearer and time scope. Answers use observations within that scope and, for historical questions, before the query time.

Object identity and memory

Q01
Object identityCompare reference images to tell the same instance, different instances of one category, and different categories.
Q02
Witness presenceWhich of two wearers saw the target in a shared 6-s window.
Q03
Latest witnessWho saw the target last before the query time.
Q04
Object historyWhich time windows contain the target.

Room identity and visit reasoning

Q05
Spatial pathFind a shortest path between rooms identified from images.
Q06
Room identityApply the Q01 distinctions to room references, using room type as the category.
Q07
Visit reasoningOrder visits, time the current or a completed stay, count visits, and recall the preceding room.
Room references, object annotations, and question coverage
Room references, object annotations, and question coverage. (a) The external room catalog spans 81 scopes and 18 room types, separate from the 17 named zones in the shared home. (b) The week-long component contains 34 registered landmark instances and 21 movable-object labels. (c) Questions cover both components, and Q07 is split into five subtypes.

EgoClue

Finding, verifying, and linking evidence with five tools

EgoClue's 4B controller decides where to look and how to combine observations. It trains in two stages: supervised fine-tuning on teacher trajectories, then GRPO with answer and evidence rewards on its own rollouts.

EgoClue agent: the trained 4B controller selects tools and video intervals and refines its search from returned observations
EgoClue. The trained 4B controller selects tools and video intervals, then refines its search from returned observations. The optional EgoClue + Sol stage synthesizes an answer from the trace without new visual calls.
Memory

Retrieves candidate intervals from text summaries of captions, transcripts, and coarse categories.

Video

Inspects intervals for targets, rooms, and events, starting at 1 FPS with local refinement.

Image

Compares public references with individual frames.

Topology

Computes shortest paths between identified rooms.

Timeline

Computes durations from observed boundaries.

Worked examples: direct answers versus EgoClue’s search

Direct VLMs answer from up to 32 sampled frames. Replay EgoClue’s recorded tool calls and their summarized observations for the same question.

Results

Open-weight VLMs trail, and EgoClue matches the best direct VLM

46.97%best open-weight direct VLM on 1,320 week-long questions
23.64% → 72.65%EgoClue's 4B controller from prompting to SFT+GRPO
74.85% / 80.56%EgoClue + Sol on week-long and cross-environment questions
Metric

Accuracy in percent from Table 3 of the paper: 1,320 week-long and 787 cross-environment questions. Direct VLMs use public references and up to 32 frames, and all agents share the same execution budget.

EgoClue training dynamics: reward, policy entropy, response tokens, and dialogue turns
EgoClue training dynamics. Thin traces show raw scalars and thick traces show an EMA with coefficient 0.2.

Citation

BibTeX

@misc{fang2026egowitness,
  title={EgoWitness: Benchmarking Instance-Level Spatial Memory and Reasoning in Egocentric Video Streams},
  author={Zhengru Fang and Zehao Liu and Mingyu Tsoi and Yaoyu Zhu and Junlong Li and Fengmao Lv and Sijie Cheng and Yuanliang Sun and Fang Yang and Jun Zhang},
  year={2026},
  eprint={2611.25849},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2611.25849}
}

Acknowledgments

Anyone who has gone back through a video to check which person saw which object will appreciate the patience this work took. Zhengru Fang led the project. We owe a special thank you to Grace Zhang, Yaoyu Zhu, Jialiang Lu, and the engineers at RayNeo who put that patience into annotating and correcting the data by hand. RayNeo also shared its expertise and provided the AI glasses and API credits we used. Zehao Liu, Mingyu Tsoi, and Junlong Li helped write the questions, check the answers, and train the agent. We’re grateful to Prof. Jun Zhang and Yuanliang Sun for guiding us throughout the project, and to Sijie Cheng, Prof. Fengmao Lv, and Prof. Fang Yang for reading our work so carefully and supporting us along the way.

Code, dataset, and annotations are coming soon. Release updates and access instructions will be shared in the project repository.