SOURCE NOT AVAILABLE. data/site-data.json did not parse, so no figure and no axis are drawn on this page. Every figure drawn on an axis, and every number carrying a source, is read at load from that file.
We assess drug targets, and we run the controls that decide whether the assessment is worth acting on.
Structure prediction, binding affinity, absorption and toxicity, and the literature behind each claim, run as one pipeline on hardware we own. Across ·· targets of the public FEP+ set, ··· compounds, it reaches Pearson ··· within target. Every affinity number it produces is then scored against a model given no protein at all, and the verdict is printed whichever way it falls. On the target below, it fell against us.
- FEP+ within target, Pearson
- ··· over ·· targets
- control arms per claim
- 4
- our claims the controls overturned
- 1, before release
- ADMET endpoints, no failures
- ··
Flat plotter inks. Protein drawn in outline on the same stock, ligand in clay.
··· ··· · ··· · ···
source: data/site-data.json, specimen
ABL1, and what the control said about it
ABL1 is the kinase that drives chronic myeloid leukaemia. We gave the pipeline its sequence and thirty compounds it had never seen, one per chemical series, with ·· held out and ··· measured assays behind their affinities. It folded every complex and ranked them.
| arm | what it sees | Spearman |
|---|
The ligand model won, and we are publishing that
A random forest on 2D fingerprints, given no protein and no structure, ranked these compounds better than the folded complexes did. The gap is ··· Spearman with a 95% interval that clears zero, which is the first time this battery has returned a resolved verdict rather than an inconclusive one. It resolved against us.
Our own interval, ·· to ··, contains zero. On chemistry with no related series to lean on we cannot show this arm ranks better than chance. Median error is ·· fold and ·· of ·· land within tenfold.
Why this is a measurement and not a malfunction
The same installation, on the public FEP+ set, reaches Pearson ··· within target across ·· proteins and ··· compounds. The code does what it is supposed to do.
That figure is not the one the model's authors published, and the two should not be set side by side. They reported ·· on four targets; we hold three of those four, and on those three we reach ··· across ·· compounds. A fifteen-target mean and a four-target mean are different quantities and the near match between them is a coincidence.
What changed is the question. FEP+ compounds are analogues around a shared core, and ranking those is the job Boltz-2's authors describe. Ranking unrelated chemotypes against one target is not, and their own blinded result is Pearson above 0.55 on three of eight internal assays. CASP16's blind affinity track, 28 groups on 122 compounds, had a best result near Spearman 0.56, and its assessors recorded that molecular weight tracks the measurements almost as well as the best submissions.
MIRAGE (arXiv 2609.14491) reports the same shape independently: co-folding accuracy rises with how much public data a protein family already has, while a family-disjoint random forest stays flat and leads on novel families.
The standard way of building a held-out set did not work
Held-out sets in this field are usually split by Bemis-Murcko scaffold, and that is what we did first. It reported Spearman 0.563 on thirty ABL1 compounds. That Murcko splitting fails to separate analogue series is established: Steshin's Lo-Hi benchmark showed it in 2023, and Guo, Hernández‑Hernández and Ballester put it in a title in 2024. We rediscovered it the expensive way.
Eight of those thirty were one chemical series. They share a benzamide, a piperazinylmethyl, a trifluoromethyl and an alkyne, and differ only in a fused heteroaryl cap. That cap is part of the ring system, so Murcko counts eight scaffolds where a chemist sees one series, and all eight were admitted. Removing them is not the same as removing any nine compounds: keeping one per Tanimoto cluster gives 0.250, while dropping nine at random and ignoring clusters leaves the score at 0.556, and a random subset does as badly as declustering 0.25% of the time.
Rebuilt with one compound per ECFP4 Tanimoto cluster at 0.6, the same thirty slots hold thirty genuine series and the number is ···. The training side leaked the same way: excluding by scaffold removed 111 compounds, excluding by similarity removes 441, so 330 near neighbours of the test set had been sitting in training. That inflated the baselines beating us as well, and both arms were re-measured.
What moved when the leak was closed
| arm | leaky | clean |
|---|---|---|
| ligand alone | 0.871 | 0.872 |
| nearest-neighbour | 0.663 | 0.726 |
| structure + affinity | 0.546 | 0.284 |
| molecular weight | 0.497 | 0.209 |
Molecular weight fell furthest, which says most of what it scored came from the series rather than from any relation between size and affinity. The ligand model did not move despite losing 330 training compounds, so its advantage was never the leak. Our own arm reads 0.284 against molecular weight's 0.209 on the clean set, and that gap does not resolve: the interval runs −0.295 to +0.431 and contains zero. By the rule stated above, this evaluation cannot tell the two apart.
Every number that moved is recorded in CORRECTIONS.md in the repository, with what moved it.
Does it get the direction of a change right?
Everything above asks how well the pipeline ranks a set of compounds. That is not the question it gets used for. A chemist changes one thing about a molecule and wants to know whether the change helped. So: every pair of compounds measured against the same protein is one such change, and ···· of them across ·· targets can be formed from predictions already made.
| real change | pairs | direction right | error |
|---|
Against a method that costs hours per pair
On the ··· pairs where the FEP+ reference also has a value, free energy perturbation calls the direction right ·· of the time against our ··, with a root mean square error of ·· kcal/mol against our ··.
It is ahead, and it should be. Free energy perturbation simulates the change explicitly and takes hours of compute for each pair. This runs in minutes on one consumer card. The useful comparison is not which wins but whether the cheap method is close enough to be worth pointing at a list before the expensive one is pointed at a shortlist.
These pairs are formed from predictions already made and reported above. It is the same evidence read in the frame the tool is used in, not a new measurement. Regenerate with scripts/paired_validation.py.
A number is not useful unless it comes with what it might be wrong by
Our affinity model ranks compounds well inside a chemical series and near chance across series, which is measured above and is not fixable: a search of the 2024–2026 literature found no published method that beats a model given no protein at all on a genuinely series‑separated single‑target task. So the question is not how to make the number better. It is what a reader is entitled to be told about it.
Every prediction now carries an interval, calibrated on compounds already measured for the same target. Across ··· compounds the intervals are asked to contain the truth ·· of the time, and they contain it ·· of the time, on every target tested, with none falling short.
| target | covered | interval width |
|---|
The width is the honest part
On one target the interval is tenfold, which a chemist can act on. On another it is wider than the entire range of affinities anyone would measure, which is the model reporting that it does not know. Both numbers are correct: the intervals cover on both targets. What differs is whether the prediction was worth making.
Calibrated instead on other proteins, with nothing measured for the one being asked about, coverage holds on average and collapses per target — from · on the worst to ·· on the best. An interval that is right on average and wrong for your protein is worse than no interval, so the tool asks for measured compounds before it offers one.
Split conformal prediction, calibrated on absolute residuals with the finite-sample correction. Regenerate with scripts/conformal_validation.py.
Programs
Vessari runs target assessments end to end. Give it a protein and a set of compounds and it predicts the bound structure, scores the binding, profiles absorption and toxicity, and pulls the literature behind every claim it makes. Each program below is a pipeline we run on our own hardware, and each one reports the measurement behind its numbers.
-
Running
Carbonic anhydrase II
Our reference target, and the one that taught us the most by not working. Densely measured in the public record, which is also why it sits inside the training data of the model we run, so a good score here proves less than it looks.
- held out
- ··· compounds
- rank correlation
- ···
-
Running
Benchmark reliability
Measuring whether the benchmarks the field ranks itself on can actually tell models apart. On the hardest tier of one public benchmark, the same model at thirteen random seeds spans nearly four times the published gap between the three methods the benchmark ranks.
- seed spread
- ·
- between-model spread
- ·
-
Running
ADMET properties
Absorption, metabolism and toxicity across 22 public endpoints, plus a live blind challenge on pharmaceutical data we have never seen. This is the part of the pipeline that holds up under its own controls.
- endpoints
- ··
- blind entry
- 750 compounds
A program moves from preparation to running when it has a scaffold separated held-out set large enough to carry an error bar. That is the bar for appearing above with a number next to it.
Held-out performance, and what it survives
- MAE
- ····· log₁₀ µM
- n
- ··· compounds
- split
- ···
- σ on ρ
- ··· = 1/√(n−1)
The control we ran against ourselves
A rank correlation on held-out compounds is evidence that the structure is doing work only if a model that never saw the protein does worse. We built that control, pointed it at our own result, and it came back inconclusive.
| arm | Spearman |
|---|---|
| our pipeline | · |
| ligand alone, no protein | · |
| nearest neighbour, pure recall | · |
| molecular weight, the floor | · |
The gap to the ligand-only arm is +0.083, 95% interval −0.014 to +0.195. That interval contains zero, so on this target we cannot show the structure contributes over the ligand alone. The model does clear pure recall by +··· and molecular weight by +···, so it is not a lookup table and the task is not trivial. Those are different claims and we keep them apart.
On a benchmark we did not choose, 52 compounds across CDK2, P38 and TYK2 from the public FEP set, the same control returns the same answer: our pipeline · against a ligand-only ·. Two inconclusive controls on two unrelated datasets is a pattern rather than an accident.
What the number is
The held-out set is ··· compounds separated from training by scaffold rather than by molecule. Scaffold splitting removed ··· near relatives that a molecule level split would have left in training, so the correlation above is measured against chemistry the model has not already seen a close cousin of.
Running the same pipeline twice moves a predicted value by about ··· log units. That spread is the floor under any comparison of two predictions. The standard error on a rank correlation at this sample size is 1/√(n−1), which is ···, and that is the width shaded on the axis above.
n··· split··· removed··· repeat spread···
What a scaffold split does to the score
A molecule level split can put two analogues of the same series on either side of the line, and the model then scores well by recognising the series. Splitting on Bemis-Murcko scaffold puts a whole series on one side. The reported correlation is lower under this split than under a molecule split, and that is the point of using it.
Residual strip
Every held-out compound with a prediction, ··· of them, sorted by measured affinity. Each lane carries two marks on one axis: the measured value in violet and the prediction in clay, with the miss drawn between them. Reading down the strip, the marks sit together at the tight end and pull apart as the compounds get weaker, which is the model losing resolution rather than losing accuracy evenly.
How it is checked
Citation verification is ···. A quoted sentence either appears in the source text or it does not, and no model is asked to judge whether a citation supports a claim. ADMET runs across ·· endpoints from the Therapeutics Data Commons benchmark group, scored below on the metric that group defines for each one. The endpoint names are theirs, so the table can be compared line by line against a public source.
| # | endpoint | metric | score | direction |
|---|
Where the endpoint list comes from
- benchmark archive
benchmarks/tdc/admet_group.zip- listing in code
src/vessari/admet/benchmark.py
The endpoint names above are the identifiers used by the Therapeutics Data Commons group, so the list can be compared line by line against a public source.
What the verifier does with cases where the answer is known
Run through the same entry point the pipeline uses, not a helper written for a demonstration. · of · behaved as expected.
Scope of this page
What the figures above cover, and what they do not. We would rather you arrive at this list from us than from your own reading of the numbers.
- One target in the viewer. The structure at the top of this page is one prediction for one protein with one ligand. It is an example of what the output looks like, not evidence of accuracy on any other target.
- No bench work of ours. The measured values in the residual strip carry ChEMBL identifiers and come from the source dataset. We have not measured any of them ourselves.
- The sample is small. A rank correlation on ··· compounds carries a standard error of ··. That width is drawn on the axis, not rounded away, and it is wide.
- One ADMET configuration, not twenty-two. The same model and the same features are fitted to every endpoint. A per endpoint search would score better and would stop the comparison being like for like, so the numbers below are one honest configuration rather than the best each endpoint could reach.
- Confidence is self reported. The Boltz-2 numbers beside the structure describe how well the model agrees with itself. They are not agreement with a solved structure.