Deep research systems must navigate multi-step workflows like web searching and cross-referencing information, yet evaluating these capabilities often requires prohibitive human effort. To address this bottleneck, researchers at Infinity Lab developed DeepResearchEval, a framework designed to automate both task generation and agent-led evaluation. Lead author Yibo Wang, a researcher at Infinity Lab, spearheaded the project to create more realistic testing environments for advanced language models. The system ensures benchmarks remain challenging by filtering for tasks that necessitate deep evidence integration rather than simple data retrieval.

The framework operates through a persona-driven pipeline that crafts research prompts based on diverse user profiles to simulate real-world demands. Once a research agent produces a report, DeepResearchEval employs an Adaptive Point-wise Quality Evaluation method to generate unique grading criteria for each specific task. This approach moves beyond static rubrics, allowing the evaluation to scale alongside the complexity of the research topics. By automating the creation of these high-fidelity benchmarks, the system provides a more efficient path for developers to refine autonomous agentic behaviors.

A key innovation is the Active Fact-Checking component, which operates independently of the report’s provided citations. This agentic tool searches the web to verify claims autonomously, ensuring that AI-generated content is accurate even when references are missing. By dynamically adjusting evaluation standards and verifying facts in real-time, DeepResearchEval offers a scalable solution for benchmarking research-oriented models. This framework effectively reduces the need for constant human oversight while maintaining rigorous standards for information integrity and reasoning in AI development.

Deep research systems must navigate multi-step workflows like web searching and cross-referencing information, yet evaluating these capabilities often requires prohibitive human effort. To address this bottleneck, researchers at Infinity Lab developed DeepResearchEval, a framework designed to automate both task generation and agent-led evaluation. Lead author Yibo Wang, a researcher at Infinity Lab, spearheaded the project to create more realistic testing environments for advanced language models. The system ensures benchmarks remain challenging by filtering for tasks that necessitate deep evidence integration rather than simple data retrieval.

The framework operates through a persona-driven pipeline that crafts research prompts based on diverse user profiles to simulate real-world demands. Once a research agent produces a report, DeepResearchEval employs an Adaptive Point-wise Quality Evaluation method to generate unique grading criteria for each specific task. This approach moves beyond static rubrics, allowing the evaluation to scale alongside the complexity of the research topics. By automating the creation of these high-fidelity benchmarks, the system provides a more efficient path for developers to refine autonomous agentic behaviors.

A key innovation is the Active Fact-Checking component, which operates independently of the report’s provided citations. This agentic tool searches the web to verify claims autonomously, ensuring that AI-generated content is accurate even when references are missing. By dynamically adjusting evaluation standards and verifying facts in real-time, DeepResearchEval offers a scalable solution for benchmarking research-oriented models. This framework effectively reduces the need for constant human oversight while maintaining rigorous standards for information integrity and reasoning in AI development.