Ian's Noesis Wiki

Properties1
tagsai, scientific-method, modeling, translational, evidence

4 min read

DeepEvidence (Biomedical Deep Research Agent)

DeepEvidence is a deep research agent system for evidence exploration and synthesis across heterogeneous biomedical knowledge sources. Developed by Zifeng Wang, Jimeng Sun (Keiji AI / UIUC), and collaborators at NIH/NLM, Osaka University, Kyoto University, and Weill Cornell. Published in Nature Machine Intelligence (2026).

Unlike generic deep research agents (OpenAI DeepResearch, Gemini Deep Research, Perplexity) that operate over unstructured web text, DeepEvidence is designed for biomedical research where critical evidence is distributed across structured databases, ontologies, knowledge graphs, and literature. 1

Architecture

Multi-Agent Orchestrator–Subagent Design

  • Orchestrator receives user queries, plans the investigation strategy, manages search budgets, and maintains the evidence graph. Built with LangGraph state-machine graphs.
  • BFRS (Breadth-First Research Strategy) subagents perform multigraph queries to rapidly survey first-hop neighbourhoods — e.g., identifying all drugs linked to a disease across disease–drug and gene–disease graphs.
  • DFRS (Depth-First Research Strategy) subagents build on entities surfaced by BFRS for deeper multihop exploration — e.g., recursively tracing citation chains across publications to build comprehensive evidence graphs for systematic review.

BFRS/DFRS are not fixed BFS/DFS graph traversal. The AI agents choose traversal steps autonomously, and can bridge across heterogeneous knowledge graphs by recognizing shared entities or inferring cross-graph relations (“bridge entities”).

Code Execution Sandbox

All agents use code execution as their primary action mechanism. Rather than returning static outputs, tools generate executable Python code run in a sandbox. This enables multistep API queries, data transformation, and statistical analysis. Execution outputs (including errors) are returned to the agent for iterative self-correction via ReAct loops.

Evidence Graph (First-Class Memory)

  • Incrementally constructed throughout the research process
  • Stores normalized biomedical entities: genes, diseases, drugs, pathways, papers, findings
  • Every node and relation grounded with explicit provenance
  • Agent queries existing graph before adding new content; merges near-duplicates; records conflicting findings separately
  • Quality audit (316 items across 10 graphs): 100% provenance validity, 99.7% normalization accuracy, 0.6% duplication rate, ≥99% relation correctness, 93.3% claim alignment

Unified Knowledge Source Tools (17 databases)

BioThings, ChEMBL, ClinicalTrials.gov, Gene Ontology, HPO, KEGG, NCBI Datasets, OpenFDA, Open Genes, ProteinAtlas, PubChem, PubTator, PubMed, Reactome, UMLS, UniProt. Unified modality-wise search tools aggregate multiple sources concurrently (e.g., search_genes_unified queries BioThings, KEGG, OpenTargets in parallel).

Benchmark Results

Open Benchmarks (all systems using GPT-5)

BenchmarkDeepEvidenceBiomniToolUniverseSonnet-4.5GPT-5
HLE-Medicine40.0%20.0%10.0%3.3%3.3%
LabBench-LitQA264.7%48.9%25.0%——
SuperGPQA-Medicine-Hard47.1%40.7%—43.6%—
TrialPanorama-EvidenceQA96.0%84.0%—88.0%—

New Biomedical Discovery Benchmarks (7 tasks across 4 stages)

  1. Target Identification (drug discovery): 68% vs Biomni 56%, ToolUniverse 40%
  2. Target Validation (conflicting evidence reconciliation): strongest at resolving evidence hierarchy, context, and temporal conflicts
  3. Mechanism of Action & Pathway Reasoning (preclinical): 72% vs Biomni/ToolUniverse 44%
  4. In vivo Metabolic Flux Response (preclinical): 80% vs Biomni 68%
  5. Sample Size Estimation (clinical trials): 68% vs Biomni 20%
  6. Drug Regimen Design (clinical trials): 52% vs Biomni 36%
  7. Surrogate Endpoint Discovery (clinical trials): F1 73.3% vs Biomni 60.9%
  8. Evidence Gap Discovery (EBM): gap detection 90.0%, recall@30 44.1% vs Biomni 50.0%/30.2%

Open-Ended Research Challenges (human expert rubrics)

  • Target Prioritization: 1.40 vs Biomni 1.15 (P<0.001), winning 5/8 dimensions
  • Translational Validation: 1.64 vs Biomni 1.46 (P<0.05), winning 6/8 dimensions

Key Ablation Findings

  1. Tool confusion: Exposing many tools via MCP is suboptimal; code execution substantially mitigates this
  2. Multi-agent ≠ better: Vanilla Orchestrator+ReAct underperforms single-agent ReAct+MCP+CodeExec — subagent design is critical
  3. BFRS + DFRS complementary: Combining both yields best results; either alone is insufficient

Relationship to Other Paradigms

  • Not GraphRAG — does not preprocess heterogeneous data into a unified knowledge graph before retrieval
  • Not traditional BFS/DFS over knowledge graphs — agents are LLM-driven deep researchers operating under a graph-inspired exploration paradigm
  • Extends the deep research paradigm (OpenAI DeepResearch, Gemini Deep Research, Perplexity) to biomedical domains with heterogeneous structured + unstructured sources

Limitations

  • Incomplete knowledge source coverage; current benchmarks reflect only a subset of biomedical tasks
  • Works with well-curated KBs (KEGG, GO) — needs adaptation for proprietary/incomplete sources
  • No explicit probabilistic modelling or uncertainty-aware mechanisms for conflicting evidence (relies on LLM parametric reasoning)
  • BFRS/DFRS may not be optimal strategies; future work on self-directed strategy discovery
  • Not yet expert-level on many tasks
  • Text-only; no multimodal data (imaging, structural biology, proteomics)
  • ai-scientist — hub page for autonomous AI research agents; DeepEvidence extends this paradigm to biomedicine
  • andrej-karpathy — creator of autoresearch, another autonomous AI research approach
  • what-is-intelligence-book — broader framework for understanding prediction-based AI

Footnotes

  1. raw/papers/wang-2026-deepevidence.md ↩

Graph View

Backlinks