Quantitative Structure–Activity Relationship (QSAR) modelling occupies a central position in computational drug discovery. By learning mappings from molecular representations to biological activity metrics — typically expressed as pIC50 or pKi values — QSAR models are employed to prioritise synthetic candidates, reduce experimental costs, and accelerate lead optimisation. Yet a persistent and underappreciated problem afflicts much of the published literature: reported performance metrics are systematically inflated by methodological artefacts that compromise their scientific validity.
The most consequential of these artefacts is structural leakage — a form of data contamination that arises from the chemical similarity structure of the training and test sets.
In standard random splits of molecular datasets, structurally similar compounds are distributed across both training and test partitions. Because molecular activity is strongly correlated with structural similarity (the principle underlying medicinal chemistry itself), a model evaluated on a test set containing near-neighbours of its training compounds will appear to generalise well even if it has learned nothing beyond local interpolation.
Formally, if the training set T and test set V are not structurally disjoint, then:
The inflated R² is not a measure of generalisation — it is a measure of the degree of structural overlap between the two partitions. This distinction matters enormously when the model's intended application is prospective prediction of compounds not yet synthesised.
Note: The following patterns are endemic in published QSAR benchmarks and should be treated as red flags during model evaluation.
At ActarusLab, we apply what we term the Honest Out-of-Fold (OOF) Protocol as a minimal standard for QSAR model evaluation. The protocol has three components:
Train and test partitions are constructed to ensure that Bemis–Murcko scaffolds present in the test set are absent from the training set. This enforces a more realistic prospective scenario.
All performance metrics reported on the training distribution are computed via k-fold cross-validation, with predictions made exclusively on held-out folds. This eliminates the possibility of reporting in-sample fit as a generalisation estimate.
Pairs of structurally similar compounds with large activity differentials are explicitly flagged. Model performance on activity cliff pairs is reported separately, as it is a more demanding and scientifically informative evaluation criterion than aggregate R².
In our pIC50 prediction study applied to a publicly available kinase inhibitor dataset, we observed the following divergence between evaluation protocols:
The delta of 0.17 represents the contribution of structural leakage to the reported performance — a contribution that would disappear entirely in prospective deployment. The OOF figure of 0.74 is a substantially more honest estimate of the model's scientific utility.
The practical consequences of this validation failure extend beyond academic debate. If compounds are prioritised for synthesis based on models whose performance is inflated by structural leakage, the false positive rate in experimental validation increases correspondingly. In a domain where each synthesis and assay cycle represents significant cost and time, this translates directly into inefficient resource allocation.
Rigorous validation is not a methodological luxury; it is a prerequisite for the scientific credibility of computational predictions.
Standard benchmarking practices in QSAR modelling routinely overestimate model performance through structural leakage and inadequate partitioning strategies. The Honest OOF Protocol provides a more demanding and scientifically valid evaluation framework, producing estimates that are lower but meaningfully predictive of prospective performance. We consider this distinction — between flattering and honest performance estimation — to be one of the most important and underappreciated issues in applied computational chemistry.