OpenFrontiers: A New Benchmark for AI to Solve Scientific Problems

Researchers have introduced OpenFrontiers, a cross-disciplinary resource that combines a structured corpus of open research problems with a retrospective benchmark of recently resolved problems in mathematics. This new tool aims to evaluate the capabilities of large language models (LLMs) in solving research-level problems, an area where a comprehensive collection of such problems has been lacking.

What OpenFrontiers Actually Does

OpenFrontiers is a comprehensive resource that includes a corpus of 12,151 open research problems across various disciplines. Each problem statement is accompanied by context, progress, and other relevant information. Additionally, the benchmark contains 200 problems, with 110 resolved in 2026 and 90 still open. These problems are presented in Lean~4, a formal proof language, and are checked by automated probes and two independent reviews. A subset of 50 high-quality problems, known as HQ-50, is further filtered based on quality scores and other criteria.

Lean~4 is a theorem prover and programming language that is widely used in formal verification and mathematical proofs. It allows researchers to express mathematical statements and proofs in a precise and unambiguous manner, making it an ideal tool for evaluating AI's ability to solve complex problems.

How the Benchmark Works

The evaluation protocol for OpenFrontiers grades formal proofs using two independent checkers in an isolated sandbox. The evaluations are conducted under three conditions: closed-book, frozen-retrieval, and live-web. Closed-book rediscovery is rare, with AI agents solving at most 7 of 40 resolved problems. In contrast, live-web success mostly reflects the retrieval of published solutions, indicating that AI agents are more likely to find solutions when they can access external information.

The closed-book condition evaluates AI agents without any external resources, testing their ability to rediscover solutions independently. The frozen-retrieval condition allows AI agents to access a static set of resources, simulating a scenario where the AI has a fixed knowledge base. The live-web condition permits AI agents to access the internet in real-time, enabling them to retrieve the latest information and solutions.

The benchmark also includes a subset of 50 high-quality problems, known as HQ-50, which are further filtered based on quality scores and other criteria. This subset is used to evaluate the performance of AI agents on a more challenging set of problems, providing a more rigorous assessment of their capabilities.

Why It Matters for Everyday People

OpenFrontiers provides a foundation for measuring AI's ability to rediscover and solve currently open scientific problems. This resource could accelerate research in various fields by allowing AI to assist in solving complex problems. For everyday people, this means that AI could potentially help scientists make breakthroughs in areas like medicine, environmental science, and technology, ultimately improving our quality of life.

For example, AI could help researchers develop new drugs by solving complex biological problems, or it could assist in finding solutions to environmental challenges by analyzing large datasets and identifying patterns. By accelerating scientific discovery, AI could lead to faster advancements in technology and improvements in healthcare, education, and other areas that impact our daily lives.

What You Can Do Today

If you are interested in exploring OpenFrontiers, you can visit the dataset and benchmark on the provided URL. While the resource is primarily aimed at researchers and AI developers, anyone curious about the intersection of AI and scientific research can explore the problems and solutions presented. This could be a great way to understand how AI is being used to push the boundaries of scientific discovery.

For those who are not familiar with formal proof languages like Lean~4, there are many online resources and tutorials available that can help you get started. Additionally, you can explore the problems and solutions presented in OpenFrontiers to gain a better understanding of how AI is being used to solve complex scientific problems.