Research question
Can a model tell which physical object it saw, when and where it saw it, and who else saw it, across a week of video from several wearers?
Existing long-video benchmarks mainly test events, object categories, or relative locations. They rarely test whether today’s object is the instance seen days earlier, or who saw it last.
An instance-level spatial-memory benchmark
8,453 human-verified questions over about 324 hours test seven abilities, from object identity to room visits.
Twelve direct VLMs struggle most when identifying who saw an object last. 2Room-guided, human-reviewed annotations
1,034,075 masks of 34 registered landmark instances at 1 Hz and 21 movable-object labels at 5 Hz.
Linked through 17,269 room intervals and reviewed identities. 3EgoClue, a spatial-memory agent
Five tools retrieve candidate intervals, verify instance identity, and compute room paths and visit durations.
SFT and GRPO raise its 4B controller from 23.64% to 72.65%, and trace synthesis reaches 74.85%.Abstract
Recalling where and when an object was last seen and who saw it requires recognizing it across days and people. Existing long-video benchmarks focus on events, object categories, or relative locations, leaving instance-level spatial memory largely untested in multimodal models. To address this gap, we present EgoWitness, a benchmark for instance-level spatial memory and reasoning in egocentric video. Its 8,453 human-verified questions span approximately 324 hours of week-long and short-horizon recordings and test seven abilities, from recognizing an object or room to recalling who saw it and reasoning about room visits. Over one million reviewed masks of 34 registered landmarks and 21 movable-object labels are linked through room timelines. Among twelve direct vision-language models (VLMs), open-weight models reach at most 46.97% accuracy, and none exceeds 35% on identifying who saw an object last. We therefore propose EgoClue, a spatial-memory agent that retrieves candidate intervals, verifies instance identity, and computes spatial and temporal relations with five tools. Two-stage training with supervised fine-tuning (SFT) and Group Relative Policy Optimization (GRPO) raises its 4B controller from 23.64% to 72.65% accuracy on 1,320 held-out questions. Evidence-trace synthesis matches the best direct VLM at 74.85% on week-long questions and achieves 80.56% in environments excluded from training.
Annotations
A week of six wearers, annotated room by room and object by object
The week-long component builds on EgoLife: six wearers share one house for seven days in 32,001 source videos. We add 17,269 room intervals over 17 named rooms and zones, covering all recorded wearer-hours, and 1,034,075 masks of 34 registered landmark instances and 21 movable-object labels.
Room timelines: VLM proposals and human review
A VLM first proposes the wearer’s room from dense captions of the 17 shared spaces. Annotators then review and correct the proposed intervals in the Online Room Curator System, yielding 17,269 room intervals that cover all recorded wearer-hours. Each tour replays a stretch of the week at high speed against both timelines.
Object masks across the week
Clips show the 976×976 center crop used for detection, with landmark masks at 1 Hz, movable-object masks at 5 Hz, and the wearer’s room from the room timeline.
Layered annotation and review
Room observations across wearers and days
Benchmark
Seven abilities in two groups
Each question names an object or room, public references, and a wearer and time scope. Answers use observations within that scope and, for historical questions, before the query time.
Object identity and memory
Q01Q02Q03Q04Room identity and visit reasoning
Q05Q06Q07
EgoClue
Finding, verifying, and linking evidence with five tools
EgoClue's 4B controller decides where to look and how to combine observations. It trains in two stages: supervised fine-tuning on teacher trajectories, then GRPO with answer and evidence rewards on its own rollouts.
Retrieves candidate intervals from text summaries of captions, transcripts, and coarse categories.
Inspects intervals for targets, rooms, and events, starting at 1 FPS with local refinement.
Compares public references with individual frames.
Computes shortest paths between identified rooms.
Computes durations from observed boundaries.
Worked examples: direct answers versus EgoClue’s search
Direct VLMs answer from up to 32 sampled frames. Replay EgoClue’s recorded tool calls and their summarized observations for the same question.
Results
Open-weight VLMs trail, and EgoClue matches the best direct VLM
Accuracy in percent from Table 3 of the paper: 1,320 week-long and 787 cross-environment questions. Direct VLMs use public references and up to 32 frames, and all agents share the same execution budget.
Citation
BibTeX
@misc{fang2026egowitness,
title={EgoWitness: Benchmarking Instance-Level Spatial Memory and Reasoning in Egocentric Video Streams},
author={Zhengru Fang and Zehao Liu and Mingyu Tsoi and Yaoyu Zhu and Junlong Li and Fengmao Lv and Sijie Cheng and Yuanliang Sun and Fang Yang and Jun Zhang},
year={2026},
eprint={2611.25849},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2611.25849}
}Acknowledgments
Anyone who has gone back through a video to check which person saw which object will appreciate the patience this work took. Zhengru Fang led the project. We owe a special thank you to Grace Zhang, Yaoyu Zhu, Jialiang Lu, and the engineers at RayNeo who put that patience into annotating and correcting the data by hand. RayNeo also shared its expertise and provided the AI glasses and API credits we used. Zehao Liu, Mingyu Tsoi, and Junlong Li helped write the questions, check the answers, and train the agent. We’re grateful to Prof. Jun Zhang and Yuanliang Sun for guiding us throughout the project, and to Sijie Cheng, Prof. Fengmao Lv, and Prof. Fang Yang for reading our work so carefully and supporting us along the way.

