More Works from Our Lab
Homepage
Overview of all our projectsLLM-ACES: Closed-Loop Discovery of Dynamical Systems with LLM-Guided Adaptive Search
LLM-induced operator priors plus predictive-divergence trajectory acquisition for recovering governing ODEs.
NeurIPS 2026LLEMA: Evolutionary Search with LLMs for Multi-Objective Materials Discovery
Couples the scientific knowledge in LLMs with chemistry-informed evolutionary rules and memory-based refinement for materials discovery.
ICLR 2026LLM-FE: Automated Feature Engineering with Large Language Models
A framework that casts feature engineering as evolutionary program search, with LLMs generating and refining executable feature transformations.
TMLR 2026
NeurIPS 2026
Abstract
Scientific discovery is a closed-loop process where hypotheses guide data acquisition and observations refine the hypothesis space. Yet most approaches reduce discovery to supervised learning over fixed datasets, where limited observations can support multiple plausible mechanisms that fit locally but fail to generalize. Thus, the key challenge is selecting informative observations to resolve uncertainty, shifting the focus from static inference to adaptive data acquisition. To address this, we propose LLM-AutoSciLab, a closed-loop framework that couples hypothesis generation with hypothesis-conditioned experiment selection and mechanism refinement. Rather than fitting models to passively collected data, LLM-AutoSciLab iteratively proposes plausible hypotheses, selects informative experiments to distinguish or refine them, and updates its state using the resulting evidence. To evaluate dynamic, closed-loop scientific discovery with active data acquisition, we introduce ActiveSciBench, comprising two datasets: (i) ActiveSciBench-Chem (57 enzyme-kinetics tasks) and (ii) ActiveSciBench-GRN (45 gene-regulatory-network tasks), that model discovery as a budget-constrained process requiring adaptive experiment design, variable selection, and recovery of true mechanisms. Across NewtonBench, ActiveSciBench-Chem, and ActiveSciBench-GRN, LLM-AutoSciLab outperforms prior methods, achieving 67.6% and 35.1% symbolic accuracy on NewtonBench and ActiveSciBench-Chem, respectively, and 31.1% exact graph recovery on ActiveSciBench-GRN. Moreover, hypothesis-guided experimentation is 2–5× more sample-efficient than the strongest competing baselines.
Method Overview
LLM-AutoSciLab treats discovery as active, hypothesis-conditioned experiment design. The LLMs define a space of competing mechanisms and where they disagree; experiments are chosen to separate, refine or falsify them, and the evidence flows back into memory.
Hypothesis generation
A small LLM (Qwen2.5-7B) is sampled in batches and its proposals are grouped into mechanism families until the distribution stabilizes. A larger LLM (GPT-4o-mini) turns that set into a primary hypothesis, alternatives, and diagnostic search regions.
Hypothesis-conditioned experiment selection
While confidence is low, the next experiment goes where competing mechanisms predict most differently inside the LLM-proposed regions (disambiguate). Once a mechanism is stable, experiments improve its fit (refine).
Optimization and confidence feedback
Hypotheses are refit on the new data (PySR or direct skeleton fits for equations, BFGS for graphs). A bootstrap gate refits on resampled data: stable mechanisms gain confidence, brittle ones are flagged, and the verdict is written to memory.
ActiveSciBench
Existing benchmarks hand the learner a fixed dataset. ActiveSciBench gives it a hidden experimental system, a query budget, and inputs of which only some matter: the learner has to design experiments, find the relevant variables, and recover the true mechanism.
ActiveSciBench-Chem
Enzyme-catalyzed reactions with a hidden rate law. Every experiment sets a shared 7-D assay interface (substrate, inhibitor, second substrate, product, enzyme, temperature, pH) and returns the initial rate. Tiers run from textbook families to cooperative and allosteric kinetics.
ActiveSciBench-GRN
Hidden regulatory systems built from 5 motif families × 3 topological variants × 3 difficulty levels. Experiments knock nodes up or down and observe every downstream change; the target is the signed directed graph, among 320 ≈ 3.5×109 possibilities.
NewtonBench
Counterfactual physics laws across 12 domains, from gravitation and Snell’s law to the Bose–Einstein distribution. Variables are known, so it isolates active symbolic law discovery under a query budget.
Main Results
All methods use GPT-4o-mini where an LLM is involved and the same fixed oracle budgets. Pick a benchmark and a metric; bars show each difficulty tier, and the overall panel adds the spread over three seeds.
Recovery by difficulty
LLM-AutoSciLab in blue, baselines in gray, grouped by whether they design experiments. Hover a row for exact values.
Show the full results table
Fitting the data is not finding the law
Exact accuracy only asks whether predictions match the oracle; symbolic accuracy asks whether the recovered expression is the true mechanism. Fit-driven baselines land far below the diagonal: PySR and Bayesian optimization match NewtonBench data on about 70% of tasks but recover the law on only 24%.
NewtonBench
Overall exact vs symbolic accuracy per method.
ActiveSciBench-Chem
Both LLM baselines sit at the origin: plausible textbook rate laws that are never exactly right.
Sample Efficiency
Every oracle call counts. Conditioning each query on competing hypotheses makes it far more informative than uncertainty-driven or passive sampling.
Queries needed to match LLM-AutoSciLab
How many oracle queries each method needs to reach LLM-AutoSciLab’s fixed-budget performance, as a multiple of LLM-AutoSciLab’s budget (B = 20 for NewtonBench and GRN, B = 60 for Chem). The blue rule at 1× is LLM-AutoSciLab itself.
Show data table
Recovery as the query budget grows
On NewtonBench and GRN the second-best method does not reach LLM-AutoSciLab’s 20-query performance even with 100 queries. On Chem, Bayesian experimental design needs 60 queries to catch up with LLM-AutoSciLab at 20.
What Each Component Contributes
Removing any single component hurts, but each benchmark leans on a different one, a sign that the three benchmarks probe complementary discovery skills.
Ablation: remove one component at a time
Stratified subsets of each benchmark. Dropping hypothesis-conditioned acquisition hurts on all three. Beyond that, Chem depends most on memory (finding the relevant variables), NewtonBench on the adaptive ensemble and confidence gate, and GRN on memory and intervention selection.
Show data table
LLM backbones
Replacing GPT-4o-mini with open-weight Qwen3 models (4B, 14B, 32B) as the main LLM. Among the Qwen3 models, larger backbones raise symbolic accuracy on Chem and edge F1 and exact-graph accuracy on GRN; on NewtonBench exact accuracy rises with size while symbolic accuracy dips at 14B. GPT-4o-mini still gives the best symbolic and graph recovery.
Show data table (including RMSLE)
Qualitative Results
What the recovered mechanisms look like, for one symbolic law and one regulatory graph.
A NewtonBench law with two components
The hidden law adds a restoring term and a nonlinear damping term. Only LLM-AutoSciLab recovers both, with a fitted coefficient of 0.4915 ≈ 1/2 and a damping exponent within about 1% of the truth.
Fit-driven baselines add spurious terms
PySR inserts a sinusoidal component and Bayesian optimization finds an entangled numerical fit: low local error, wrong mechanism.
LLM baselines collapse to textbook forms
LLM-only and code-assisted LLMs return a simplified harmonic law and miss the nonlinear damping entirely.
Experiment design alone stops short
Bayesian experiment design moves toward the right components but still converges to a fit-driven approximation.
Robustness to Observation Noise
Gaussian noise injected at increasing levels. Bars are exact accuracy; lines are RMSLE (NewtonBench, Chem) or edge F1 (GRN). Symbolic recovery degrades in steps while errors rise smoothly, and on GRN noise mainly breaks exact topology while edge-level structure largely survives.
BibTeX
@inproceedings{kabra2026llmautoscilab,
title={{LLM}-AutoSciLab: Closed-Loop Scientific Law Discovery via Active Experimentation with {LLM}s},
author={Kabra, Sanchit and Abhyankar, Nikhil and Desai, Saaketh and Iyer, Prasad and Reddy, Chandan K},
booktitle={The Fortieth Annual Conference on Neural Information Processing Systems (NeurIPS)},
year={2026},
}