Findings · September 2026
Can AI read the literature, and do models learn from it?
Two questions behind this atlas. Can Claude pull enzyme-potency values out of paper tables accurately, including tables published as images? And do potency models trained on ChEMBL hold up on data they have never seen, from recent papers? Short answers: yes, at about 90% precision for a few cents per paper; and not reliably: a model that looked good on one test set did no better than a simple baseline on 56 independent papers.
1 · Extraction
Claude reads tables that rules miss
45 open-access papers that ChEMBL curators had already processed (mTOR, PI3Kα, GSK-3β, JAK2, PARP1) served as a benchmark: ChEMBL's 408 exact values are the reference. Every disagreement was traced back to the paper by hand.
Extraction methods against ChEMBL
| Method | Values found | Precision | Recall | Cost (45 papers) |
|---|---|---|---|---|
| Rules (deterministic parser) | 166 | 91% | 37% | $0 |
| Hybrid: rules + Claude on image tables | 263 | 89% | 57% | ~$0.50 |
| Claude Sonnet 4.5 on every table | 313 | 90% | 69% | $1.65 |
All 30 Claude values that ChEMBL does not list were checked against the source: none were invented. The one real error was a PARP2 column recorded as PARP1, which Claude's own reported column header exposed; a one-line check now rejects that pattern.
Cheaper models, same benchmark
| Model | Precision | Recall | Cost |
|---|---|---|---|
| Sonnet 4.5 | 87–97% | 54–82% | $1.65 |
| Sonnet 5 (used for scale-up) | 87–100% | 54–80% | $1.12 |
| Haiku 4.5 | 60–72% | 51–54% | $0.62 |
Ranges span the two benchmark halves (raw scores, before adjudication). Haiku is cheaper but loses 15–38 points of precision and returned units outside the output schema.
2 · Scale-up
397 new papers, every value checked by rules
A search of Europe PMC found 397 CC BY papers with potency tables for 25 enzymes that were not yet in the atlas. Sonnet 5 extracted 5,470 values from 1,511 tables through the Batch API for $5.33. Sonnet 5 usually skipped the optional "which column" field, so a separate checker finds each value in the source table, reads its column header and applies the atlas's strict target rules.
What happened to the 5,470 extracted values
Only "verified" values are used in the model test below.
The rejected values are real model errors: enzyme from eel or horse rather than human (130), values in µg/mL reported as µM (101), and a handful assigned to the wrong enzyme. 816 values come from review articles and are flagged as second-hand.
3 · Generalization
Models trained on ChEMBL, tested on new papers
Almost none of the atlas papers are in ChEMBL (1 of 191), so their values are data public models have never seen. Four models were trained on ChEMBL 37 for 25 enzymes (113k molecule–enzyme pairs, every test paper removed) and scored on 215 paper values for compounds ChEMBL has never measured on that enzyme. Errors are in log units: 1.0 means off by 10×.
Prediction error: ChEMBL's own test data vs new papers
RMSE, lower is better. The dashed line is the noise floor: how far two labs' values for the same compound differ (0.71).
Ranking analogs within one paper
Spearman ρ between predicted and measured potency inside each paper's compound series (unseen compounds, 17 series), with 95% bootstrap intervals. On ChEMBL's own data the same models reach ρ ≈ 0.84 within an enzyme.
- Pre-training pays off only out of distribution. The fine-tuned CheMeleon GNN ties the random forest on ChEMBL (0.63) but beats it on new papers by 0.10 (95% CI 0.02–0.18). The gap is largest for compounds unlike anything in training: 0.77 vs 1.03.
- ChEMBL benchmarks flatter models. Every model's error rises 25–40% on new papers.
- On one test set, error is near the noise floor, but ranking isn't solved, and the next section shows even the error does not transfer. Models get a chemical series' ballpark right but not the order of analogs within it, which is the question a medicinal chemist asks.
Replication on 56 newly extracted papers: it does not hold up
| First test (97 papers) | Replication (56 new papers) | |
|---|---|---|
| Unseen compounds | 215 | 273 |
| CheMeleon error (RMSE) | 0.79 [0.72–0.86] | 1.24 [1.15–1.34] |
| "Predict the average" baseline | 1.15 | 1.12 |
| Ranking within a paper's series (ρ) | 0.40 [0.22–0.54] | 0.20 [0.06–0.35] |
Same checkpoint, no retraining. On the larger independent set the model is no better than predicting each enzyme's average. Most of the error is paper-level: removing each paper's average offset cuts it from 1.24 to 0.57. Whole series sit 1.5–2.7 log units away from what ChEMBL-trained models expect; one paper reports every compound at 40–60 nM against four enzymes, which the model places near 25 µM. The extraction is faithful there, so the disagreement flags the source, not the reader. An earlier, smaller check (159 values) looked favourable and was too small to trust.
4 · Limits
What this does not show
- Values are AI-extracted and rule-checked, not validated by a scientist.
- Test sets are small (5–44 values per enzyme); one training run per model, no tuning.
- Assay conditions differ between papers and are not normalised; this is part of the noise floor.
- Most extracted compounds have no structure yet. Reading structures from drawings is the next bottleneck.