CapMem Benchmark Tests AI's Episodic Memory on 33.7 Hours of Wearable Camera Video
Summarized by AI from reporting by ArXiv cs.AI, published under our editorial policy.
Researchers from the University of Washington and Carnegie Mellon University released CapMem, a benchmark with 75 videos and 1,000 questions to test whether AI can use textual captions as episodic memory for long-duration wearable camera footage.
Key takeaways
- CapMem is a benchmark with 75 egocentric videos totaling 33.7 hours and 1,000 multiple-choice questions across 16 scenarios.
- The benchmark tests whether AI can use textual captions as episodic memory for long-duration wearable camera footage.
- CapMem addresses the long-context retrieval failures of current vision-language models, which are a bottleneck for practical wearable AI assistants.
Researchers from the University of Washington and Carnegie Mellon University released CapMem, a new benchmark designed to test whether AI models can use textual captions as a form of episodic memory for long-duration video captured by wearable cameras. The benchmark addresses a critical gap: current vision-language models struggle with long-context retrieval due to bounded frame budgets and growing visual-token costs, making it difficult for wearable AI assistants to recall past events.
CapMem's Design: 75 Videos, 1,000 Questions, 16 Scenarios
CapMem includes 75 videos totaling 33.7 hours of egocentric footage, paired with 1,000 human-annotated multiple-choice questions across 16 different scenarios. The benchmark defines a new task called Episodic Memory Video Caption QA, which evaluates whether AI can retrieve specific details from long videos by relying on textual captions rather than full visual processing.
Why Long-Context Retrieval Is a Bottleneck for Wearable AI
Current vision-language models face practical constraints when processing hours of video: they have limited frame budgets, visual-token costs grow with video length, and long-context retrieval often fails. CapMem tests whether captions can serve as a reusable, efficient memory store that overcomes these limitations. On videos longer than 20 minutes, the benchmark specifically evaluates full-coverage caption-based question answering.
Implications for Everyday AI Assistants
If AI models can reliably use captions as episodic memory, wearable assistants could one day recall where you left your keys, summarize a meeting from hours earlier, or retrieve specific moments from your day. CapMem provides a standardized way to measure progress toward this capability, making it a practical tool for researchers developing memory-enabled AI systems.
Accessing the CapMem Benchmark
The CapMem benchmark and full details are available on ArXiv. While the benchmark is primarily intended for researchers, the paper explains the methodology and goals in detail, offering insight into how AI memory capabilities are being advanced.
Frequently asked
- What is the Episodic Memory Video Caption QA task defined by CapMem?
- It is a new task that evaluates whether AI models can retrieve specific details from long videos by relying on textual captions rather than full visual processing.
- How long are the videos in the CapMem benchmark?
- The 75 videos total 33.7 hours, with a specific focus on videos longer than 20 minutes for full-coverage caption-based question answering.
- Who created the CapMem benchmark?
- Researchers from the University of Washington and Carnegie Mellon University created CapMem.