Each pilot specifies a comparable task, independent evaluation and a falsifiable performance criterion. None is claimed to have run.
Ingest a defined set of machine-generated conjectures, proofs and linked natural-language descriptions. Verify formal proofs in a trusted checker, inspect assumptions and independently map informal claims to exact theorems. Report the proportion of proof-valid and claim-aligned results separately.
Acceptance: Success: independent proof replay, inspected axioms, and detection of intentionally altered formal statements.
Challenge a corpus of published empirical papers with independent tools that recover effect sizes, sample units, analysis flexibility, data leakage and cited evidence. Evaluate them on seeded errors and unseen expert-reviewed examples; do not score by the volume of objections.
Acceptance: Success: high error recall with bounded false accusation rates and reproducible audit reports.
Translate a bounded laboratory or computational protocol into typed steps, immutable event records, calibrated instruments and automatic deviation checks. Preserve safe human oversight and audit any intervention that changes the declared plan.
Acceptance: Success: independent replay and reliable detection of missing steps, data alteration and instrument drift.
Supply several agents with the same competing models, prior data and limited budget. Compare fixed designs, adaptive information-gain policies and human-directed plans on a safe controlled domain with withheld physical outcomes.
Acceptance: Success: improved out-of-sample discrimination per unit cost without sacrificing uncertainty calibration.
Register all accepted investigations at inception, publish every completed outcome and connect results to existing claims. Audit missing-result detection and the time between an incompatible observation and dependency invalidation.
Acceptance: Success: no unaccounted completed study in the benchmark and timely downstream claim revision.
An AI system that writes more articles or generates more apparently impressive proofs has not yet shown that it improves science. Evaluate calibration, hidden-error detection, independent replication, out-of-sample predictive validity and error-correction latency. Protect challenge sets from leakage and record false accusations as seriously as overlooked mistakes.
Humans retain responsibility for acceptable risk and authorization. These pilots are designed to avoid automatically treating model outputs as verified empirical results.