EarthRecord: AI Agents Analyze 40 Years of Satellite Data to Track Land Changes

Researchers from an unnamed institution released EarthRecord, a new benchmark to evaluate AI agents' capabilities in analyzing raw satellite imagery for land-change detection. EarthRecord is designed to test how well multimodal agents can turn decades of satellite data into reliable land-change analysis, a crucial task for Earth observation science. This benchmark addresses a significant gap in existing evaluations, which often rely on curated images or preprocessed inputs, leaving out the raw multi-sensor archives where such analysis begins.

The EarthRecord Benchmark

EarthRecord is built on the raw record of 56 sites, spanning four decades of data from Landsat, MODIS, VIIRS, and Sentinel-2 satellites. The benchmark includes 238 land-change questions across ten analysis tasks. These questions are scored against withheld references from global products, agreement among methods, and visual interpretation. The benchmark also includes two controls: a 'closed book' test that removes the data record and an 'instrument pairs' test that alters one satellite's data while keeping the ground truth unchanged. This rigorous approach ensures that the agents' performance is thoroughly evaluated, highlighting their strengths and weaknesses in processing raw satellite data.

The data used in EarthRecord is not preprocessed, meaning agents must handle tasks like cloud masking and cross-sensor harmonization themselves. This adds a layer of complexity and realism to the benchmark, as it mirrors the actual challenges faced in Earth observation science. The inclusion of multiple satellite sources also ensures that the agents are tested on a diverse range of data types and quality levels.

Performance of Multimodal Agents

The study evaluated twelve multimodal agents on the EarthRecord benchmark. While leading agents could complete most analyses end-to-end, they faltered when it came to mapping from imagery to real-world changes. Further analysis revealed that agents' answers were often influenced by prior knowledge rather than the actual pixel data. When one satellite's data were altered, half of the correct answers followed the alteration, indicating a reliance on the data rather than ground truth. This finding suggests that while agents are proficient in processing data, they still struggle with accurately interpreting and mapping the data to real-world scenarios.

The evaluations also highlighted that leading agents could complete most analyses end-to-end but struggled with tasks requiring mapping from imagery. This indicates a significant gap in the agents' ability to translate raw data into actionable insights. The 'closed book' test, which removes the data record, showed that agents relied heavily on the provided data, further emphasizing the need for improvements in their ability to interpret and map data accurately.

Why This Matters for Everyday People

Understanding land changes is crucial for environmental monitoring, urban planning, and disaster management. Accurate analysis of satellite imagery can help predict and mitigate the impacts of climate change, deforestation, and urban expansion. For everyday people, this means better tools for understanding and addressing environmental issues that affect their communities. For example, accurate land-change analysis can help communities prepare for natural disasters, plan sustainable urban development, and monitor environmental health.

The insights gained from EarthRecord can also inform policy decisions and resource allocation. Governments and organizations can use this data to make informed decisions about land use, conservation efforts, and disaster response. This can lead to more effective and efficient use of resources, ultimately benefiting the broader community.

What You Can Do Today

While EarthRecord is a research benchmark and not directly available for public use, you can explore existing satellite imagery tools like Google Earth Engine or NASA's Earthdata. These platforms provide access to satellite data and tools for analyzing land changes. You can start by visiting Google Earth Engine and exploring the available datasets and analysis tools. These tools can help you understand the changes in your local environment and contribute to community efforts in environmental monitoring and planning.

For those interested in delving deeper into the technical aspects, you can also explore the EarthRecord paper on arXiv. This will provide a more detailed understanding of the benchmark and the evaluations conducted on the multimodal agents. Additionally, you can follow updates from the researchers and participate in discussions on platforms like arXivLabs, which allows collaborators to develop and share new features directly on the arXiv website.