Delaney 2004: ESOL aqueous solubility, rule of five and descriptors with RDKit

The session

This is the session as the scientist sees it, in order. Decision cards show the answer that the script gave and where that answer comes from. Each step shows the program, its version, the input file hash and the outputs. Click a line to see more.

- Setup values:
Molecules:
- Salts and multi-fragment molecules (salt_handling): keep
- SMILES that RDKit cannot parse (invalid_smiles): stop
Descriptors:
- logP method (logp_method): crippen
- Hydrogen bond donor and acceptor definition (hbd_hba_definition): lipinski
Ask the scientist: Rule of five cutoffs (rule_of_five_cutoffs), Violations allowed for a pass (allowed_violations), Solubility equation (esol_equation), Fraction of molecules held out for testing (test_fraction), Random seed for the train and test split (split_seed).
The harness fills these values into your tool calls automatically. If you give a different value, the harness logs a deviation and shows it to the scientist.
- Mode is now watch: you run the steps.

Compound ID,ESOL predicted log solubility in mols per litre,Minimum Degree,Molecular Weight,Number of H-Bond Donors,Number of Rings,Number of Rotatable Bonds,Polar Surface Area,measured log solubility in mols per litre,smiles
Amigdalin,-0.9740000000000001,1,457.4320000000001,7,3,7,202.31999999999996,-0.77,OCC3OC(OCC2OC(OC(C#N)c1ccccc1)C(O)C(O)C2O)C(O)C(O)C3O 
Fenfuram,-2.885,1,201.22500000000002,1,2,2,42.24,-3.3,Cc1occc1C(=O)Nc2ccccc2
citral,-2.5789999999999997,1,152.237,0,0,4,17.07,-2.06,CC(C)=CCCC(C)=CC(=O)
Picene,-6.617999999999999,2,278.354,0,5,0,0.0,-7.87,c1ccc2c(c1)ccc3c2ccc4c5ccccc5ccc43
Thiophene,-2.2319999999999998,2,84.14299999999999,0,1,0,0.0,-1.33,c1ccsc1
benzothiazole,-2.733,2,135.191,0,2,0,12.89,-1.5,c2ccc1scnc1c2 
"2,2,4,6,6'-PCB",-6.545,1,326.437,0,2,1,0.0,-7.32,Clc1cc(Cl)c(c(Cl)c1)c2c(Cl)cccc2Cl
Estradiol,-4.138,1,272.388,2,4,0,40.46,-5.03,CC12CCC3C(CCc4cc(O)ccc34)C2CCC1O
Dieldrin,-4.533,1,380.913,0,5,0,12.53,-6.29,ClC4=C(Cl)C5(Cl)C3C1CC(C2OC12)C3C4(Cl)C5(Cl)Cl
Rotenone,-5.246,1,394.42300000000023,0,5,3,63.22,-4.42,COc5cc4OCC3Oc2c1CC(Oc1ccc2C(=O)C3c4cc5OC)C(C)=C 
2-pyrrolidone,0.243,1,85.10600000000001,1,1,0,29.1,1.07,O=C1CCCN1
2-Chloronapthalene,-4.063,1,162.61899999999997,0,2,0,0.0,-4.14,Clc1ccc2ccccc2c1
1-Pentene ,-2.01,1,70.135,0,0,2,0.0,-2.68,CCCC=C
Primidone,-1.8969999999999998,1,218.256,2,2,2,58.2,-2.64,CCC1(C(=O)NCNC1=O)c2ccccc2
Tetradecane,-5.45,1,198.39399999999995,0,0,11,0.0,-7.96,CCCCCCCCCCCCCC
2-Chloropropane,-1.585,1,78.542,0,0,0,0.0,-1.41,CC(C)

{
 "molecules": "mol1",
 "n_read": 1128,
 "n_parsed": 1128,
 "n_invalid": 0,
 "n_multi_fragment": 0,
 "invalid_ids": [],
 "salts": "keep",
 "rdkit_version": "2026.03.6",
 "invalid_handling": "stop"
}

{
 "descriptors": "desc2",
 "n_molecules": 1128,
 "rdkit_version": "2026.03.6",
 "mean_mw": 203.9370744680851,
 "mean_logp": 2.447519539007093,
 "mean_tpsa": 34.87288120567376,
 "mean_rotatable_bonds": 2.1773049645390072,
 "mean_aromatic_proportion": 0.36354385527552907,
 "table": {
  "columns": [
   "id",
   "smiles",
   "mw",
   "logp",
   "tpsa",
   "hbd",
   "hba",
   "rb",
   "ap",
   "measured"
  ],
  "rows": [
   [
    "Amigdalin",
    "N#CC(OC1OC(COC2OC(CO)C(O)C(O)C2O)C(O)C(O)C1O)c1ccccc1",
    457.432,
    -3.108,
    202.32,
    7,
    12,
    7,
    0.1875,
    -0.77
   ],
   [
    "Fenfuram",
    "Cc1occc1C(=O)Nc1ccccc1",
    201.225,
    2.8403,
    42.24,
    1,
    2,
    2,
    0.7333,
    -3.3
   ],
   [
    "citral",
    "CC(C)=CCCC(C)=CC=O",
    152.237,
    2.878,
    17.07,
    0,
    1,
    4,
    0,
    -2.06
   ],
   [
    "Picene",
    "c1ccc2c(c1)ccc1c2ccc2c3ccccc3ccc21",
    278.354,
    6.2994,
    0,
    0,
    0,
    0,
    1,
    -7.87
   ],
   [
    "Thiophene",
    "c1ccsc1",
    84.143,
    1.7481,
    0,
    0,
    1,
    0,
    1,
    -1.33
   ],
   [
    "benzothiazole",
    "c1ccc2scnc2c1",
    135.191,
    2.2963,
    12.89,
    0,
    2,
    0,
    1,
    -1.5
   ],
   [
    "2,2,4,6,6'-PCB",
    "Clc1cc(Cl)c(-c2c(Cl)cccc2Cl)c(Cl)c1",
    326.437,
    6.6206,
    0,
    0,
    0,
    1,
    0.7059,
    -7.32
   ],
   [
    "Estradiol",
    "CC12CCC3c4ccc(O)cc4CCC3C1CCC2O",
    272.388,
    3.6092,
    40.46,
    2,
    2,
    0,
    0.3,
    -5.03
   ]
  ],
  "n_rows": 1128,
  "path": "descriptors.csv"
 }
}

{
 "n_molecules": 1128,
 "n_zero_violations": 1018,
 "n_one_violation": 98,
 "n_two_or_more_violations": 12,
 "n_pass": 1116,
 "n_fail": 12,
 "fraction_pass": 0.9893617021276596,
 "n_mw_over_500": 9,
 "n_logp_over_5": 97,
 "n_hbd_over_5": 10,
 "n_hba_over_10": 8,
 "allowed_violations": 1,
 "hbd_hba": "lipinski"
}

{
 "equation": "published",
 "intercept": 0.16,
 "coef_logp": -0.63,
 "coef_mw": -0.0062,
 "coef_rb": 0.066,
 "coef_ap": -0.74,
 "all_r2": 0.7247613093988002,
 "all_r_squared_correlation": 0.7519384878143138,
 "all_rmse": 1.0993724206558089,
 "all_n": 1128,
 "outliers": [
  {
   "id": "Tetradecane",
   "smiles": "CCCCCCCCCCCCCC",
   "measured": -7.96,
   "predicted": -3.939704800000003,
   "residual": -4.020295199999997
  },
  {
   "id": "1-Octadecanol",
   "smiles": "CCCCCCCCCCCCCCCCCCO",
   "measured": -8.4,
   "predicted": -4.392432200000004,
   "residual": -4.007567799999997
  },
  {
   "id": "Hexadecane",
   "smiles": "CCCCCCCCCCCCCCCC",
   "measured": -8.4,
   "predicted": -4.473165600000003,
   "residual": -3.926834399999997
  },
  {
   "id": "uric acid",
   "smiles": "O=c1[nH]c(=O)c2[nH]c(=O)[nH]c2[nH]1",
   "measured": -3.93,
   "predicted": -0.32395839999999965,
   "residual": -3.6060416000000006
  },
  {
   "id": "vamidothion",
   "smiles": "CNC(=O)C(C)SCCSP(=O)(OC)OC",
   "measured": 1.1440000000000001,
   "predicted": -2.3460925999999995,
   "residual": 3.4900925999999997
  },
  {
   "id": "alloxantin",
   "smiles": "O=C1NC(=O)C(O)(C2(O)C(=O)NC(=O)NC2=O)C(=O)N1",
   "measured": -1.99,
   "predicted": 1.4880547999999985,
   "residual": -3.4780547999999984
  },
  {
   "id": "bromadiolone",
   "smiles": "O=c1oc2ccccc2c(O)c1C(CC(O)c1ccc(-c2ccc(Br)cc2)cc1)c1ccccc1",
   "measured": -4.445,
   "predicted": -7.831697800000005,
   "residual": 3.3866978000000048
  },
  {
   "id": "Benzo(b)fluorene",
   "smiles": "c1ccc2c(c1)Cc1cc3ccccc3cc1-2",
   "measured": -8.04,
   "predicted": -4.656355188235295,
   "residual": -3.3836448117647038
  },
  {
   "id": "Napthacene",
   "smiles": "c1ccc2cc3cc4ccccc4cc3cc2c1",
   "measured": -8.6,
   "predicted": -5.2375288000000015,
   "residual": -3.362471199999998
  },
  {
   "id": "Nonane",
   "smiles": "CCCCCCCCC",
   "measured": -5.88,
   "predicted": -2.6060528000000014,
   "residual": -3.2739471999999985
  }
 ]
}

{
 "equation": "refit",
 "seed": 42,
 "test_fraction": 0.2,
 "intercept": 0.22947250736081581,
 "coef_logp": -0.727414495840552,
 "coef_mw": -0.00663888797180681,
 "coef_rb": 0.005521973350054075,
 "coef_ap": -0.4682504280820903,
 "train_r2": 0.7644781988311372,
 "train_r_squared_correlation": 0.7644781988311373,
 "train_rmse": 1.013078408468007,
 "train_n": 902,
 "test_r2": 0.7797246477519745,
 "test_r_squared_correlation": 0.7830687560768689,
 "test_rmse": 0.9975863350362573,
 "test_n": 226,
 "evaluated_on": "held-out test set",
 "outliers": [
  {
   "id": "bromadiolone",
   "smiles": "O=c1oc2ccccc2c(O)c1C(CC(O)c1ccc(-c2ccc(Br)cc2)cc1)c1ccccc1",
   "measured": -4.445,
   "predicted": -8.838965969536828,
   "residual": 4.393965969536827
  },
  {
   "id": "vamidothion",
   "smiles": "CNC(=O)C(C)SCCSP(=O)(OC)OC",
   "measured": 1.1440000000000001,
   "predicted": -3.080235192951821,
   "residual": 4.224235192951821
  },
  {
   "id": "Dimecron",
   "smiles": "CCN(CC)C(=O)C(Cl)=CCOP(=O)(OC)OC",
   "measured": 0.523,
   "predicted": -3.458124398535628,
   "residual": 3.981124398535628
  },
  {
   "id": "uric acid",
   "smiles": "O=c1[nH]c(=O)c2[nH]c(=O)[nH]c2[nH]1",
   "measured": -3.93,
   "predicted": 0.04769484863228535,
   "residual": -3.9776948486322854
  },
  {
   "id": "alloxantin",
   "smiles": "O=C1NC(=O)C(O)(C2(O)C(=O)NC(=O)NC2=O)C(=O)N1",
   "measured": -1.99,
   "predicted": 1.8409382755044752,
   "residual": -3.830938275504475
  },
  {
   "id": "Benzo(b)fluorene",
   "smiles": "c1ccc2c(c1)Cc1cc3ccccc3cc1-2",
   "measured": -8.04,
   "predicted": -4.855737726251885,
   "residual": -3.1842622737481143
  },
  {
   "id": "Antipyrene",
   "smiles": "Cc1cc(=O)n(-c2ccccc2)n1C",
   "measured": 0.715,
   "predicted": -2.462343078773886,
   "residual": 3.1773430787738857
  },
  {
   "id": "Napthacene",
   "smiles": "c1ccc2cc3cc4ccccc4cc3cc2c1",
   "measured": -8.6,
   "predicted": -5.497816689851589,
   "residual": -3.102183310148411
  },
  {
   "id": "guanine",
   "smiles": "Nc1nc2[nH]cnc2c(=O)[nH]1",
   "measured": -3.583,
   "predicted": -0.5956969545524249,
   "residual": -2.9873030454475753
  },
  {
   "id": "Triethyl phosphate",
   "smiles": "CCOP(=O)(OCC)OCC",
   "measured": 0.43,
   "predicted": -2.5499304787638772,
   "residual": 2.9799304787638774
  }
 ]
}

Solubility equation  seed  test_fraction  intercept  coef_logp  coef_mw    coef_rb   coef_ap  train_r2  train_r_squared_correlation  train_rmse  train_n  test_r2  test_r_squared_correlation  test_rmse  test_n  Result
refit                42    0.2            0.2295     -0.7274    -0.006639  0.005522  -0.4683  0.7645    0.7645                       1.013       902      0.7797   0.7831                      0.9976     226     ok

Compared 1 values of Solubility equation as comparison runs of the last step, fit_solubility_model (descriptors=desc2, n_outliers=10, test_fraction=0.2, seed=42). Only that step was repeated. To compare another step, run it again first. The record still says "published".
Solubility equation  seed  test_fraction  intercept  coef_logp  coef_mw    coef_rb   coef_ap  train_r2  train_r_squared_correlation  train_rmse  train_n  test_r2  test_r_squared_correlation  test_rmse  test_n  Result
refit                42    0.2            0.2295     -0.7274    -0.006639  0.005522  -0.4683  0.7645    0.7645                       1.013       902      0.7797   0.7831                      0.9976     226     ok

import pandas as pd, numpy as np, matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
from ga_api import save_table, save_figure
d = pd.read_csv("compute_descriptors-1/descriptors.csv")
d["predicted_published"] = 0.16 - 0.63*d.logp - 0.0062*d.mw + 0.066*d.rb - 0.74*d.ap
d["residual"] = d.measured - d.predicted_published
save_table("esol_published_predictions", d[["id","smiles","logp","mw","rb","ap","measured","predicted_published","residual"]].to_dict("records"))
ss_res = ((d.residual)**2).sum(); ss_tot = ((d.measured-d.measured.mean())**2).sum()
print("check r2", 1-ss_res/ss_tot, "rmse", np.sqrt((d.residual**2).mean()), "n", len(d))
fig, ax = plt.subplots(figsize=(5,5))
ax.scatter(d.predicted_published, d.measured, s=8, alpha=0.5)
lo, hi = -12, 2.5
ax.plot([lo,hi],[lo,hi],"k--",lw=1,label="measured = predicted")
ax.set_xlim(lo,hi); ax.set_ylim(lo,hi)
ax.set_xlabel("Predicted log solubility (log10 mol/L), published ESOL equation")
ax.set_ylabel("Measured log solubility (log10 mol/L)")
ax.set_title("ESOL published equation, n = %d" % len(d))
ax.legend(loc="upper left")
plt.tight_layout()
save_figure("esol_measured_vs_predicted")

check r2 0.7247613093988003 rmse 1.0993724206558086 n 1128

- The answer uses 902, 226, 0.2, 42, 0.2295, 0.005522, 0.78, 0.765, 0.783, 0.765, 0.998, 1.013 from a comparison run of another option (esol_equation), not from the setting that was chosen. Use the result of the chosen setting, or say clearly that the number is from the comparison.
- The answer names refit for "Solubility equation", but the decision record says published. Report the value that was used.
- 9 numbers in the answer match no logged tool result: 1997, 500, 0.63, 0.0062, 0.74, 0.006639, 0.4683, 2004. Run the tool that measures each number, or remove the number, or say how you derived it from logged results.
Fix these problems. Then give the answer again.

The final answer

The model wrote this answer. The harness shows it to the scientist after the review.

Step results that the answer does not mention, from the record:
- n6 run_script: The script ran in {work} and wrote 3 new file(s) to {work}.

Settings used, from the decision record: Salts and multi-fragment molecules: keep · SMILES that RDKit cannot parse: stop · logP method: crippen · Hydrogen bond donor and acceptor definition: lipinski · Rule of five cutoffs: lipinski_1997 · Violations allowed for a pass: 1 · Solubility equation: published · Fraction of molecules held out for testing: 0.2 · Random seed for the train and test split: 42.

Checks

Review findings

The review recorded 10 findings. A rule finding comes from a fixed check in the harness. A referee finding comes from a second model that reads the record. The harness shows the findings to the scientist with the final answer. The record does not mark a finding as fixed. Thus a finding from an early review round can apply to a draft that the model corrected later.

Numbers in the answer

The last claim check read 77 numbers in the answer. 73 numbers match a logged result. 2 numbers have no source in the record.

Deviations

The model did not try to change a choice of the scientist.

Failed tool calls

1 tool call failed. The model then tried again or used another tool. The session above shows each failure.

Data integrity

Each data file has the same SHA-256 hash now as at the time of the step that read it. Where the download script (fetch.sh) gives a hash, the file also has that hash. The run did not change the data.

A SHA-256 hash is a fingerprint of the file contents. If one byte of the file changes, the hash changes. The table shows the first 12 characters.

How to repeat it

Get the data. The script downloads the files and checks their SHA-256 hashes where it lists them.

CUVETTE_DATA={data} bash bench/papers/delaney2004-esol/fetch.sh

Run the same case with Cuvette. The script gives the same answers from bench/papers/delaney2004-esol/bench.yaml.

cuvette bench papers --papers delaney2004-esol --models claude:claude-opus-5-5

Repeat each step by hand in the program. For each step, the harness records a manual route: the menu path or the code that gives the same result. This list does not include comparison runs.

1. load_molecules (step n1)

   In Python

   ga_rdkit.load_molecules(path, smiles_column, id_column, target_column, salts, invalid). Each row calls Chem.MolFromSmiles(smiles.strip()).

   • CSV file

     {data}/delaney2004-esol/delaney-processed.csv

   • SMILES column = smiles
   • Fragment handling = keep
   • Invalid SMILES = stop
   • Warning: If you keep the default , you get a different result.
   • Note: The route needs the wrapper module ga_rdkit.py in the adapter folder. No person ran the route by hand.

   The manual route that the harness recorded

   ga_rdkit.load_molecules(path="{data}/delaney2004-esol/delaney-processed.csv", smiles_column="smiles", id_column="Compound ID", target_column="measured log solubility in mols per litre", salts="keep", invalid="stop")

   The manual route uses the same method. The note in the route gives the known difference.

2. compute_descriptors (step n2)

   In Python

   Descriptors.MolWt(m), Crippen.MolLogP(m), rdMolDescriptors.CalcTPSA(m), Lipinski.NumHDonors(m), Lipinski.NumHAcceptors(m), Lipinski.NumRotatableBonds(m), aromatic heavy atoms / m.GetNumHeavyAtoms()

   • logP function = crippen
   • Donor and acceptor functions = lipinski

   The manual route that the harness recorded

   ga_rdkit.compute_descriptors(molecules="mol1", logp_method="crippen", hbd_hba="lipinski")

   The manual route gives the same numbers. An automatic test in Cuvette checks this.

3. rule_of_five (step n3)

   In Python

   count a violation when MolWt >
   500, MolLogP >
   5, NumHDonors >
   5 or NumHAcceptors >
   10.

   • Cutoffs = lipinski_1997
   • Violations allowed = 1
   • Note: The cutoffs come from Lipinski 1997 as cited in the benchmark notes. The paper was not read. No person ran the route by hand.

   The manual route that the harness recorded

   ga_rdkit.rule_of_five(descriptors="desc2", cutoffs="lipinski_1997", allowed_violations=1)

   The manual route uses the same method. The note in the route gives the known difference.

4. fit_solubility_model (step n4)

   In Python

   logS = 0.16 - 0.63*logP - 0.0062*MW + 0.066*RB - 0.74*AP, or numpy.linalg.lstsq on a random training split.

   • Equation = published
   • Test fraction = 0.2
   • Random seed = 42
   • Note: The published equation comes from a secondary source (a 2018 blog post), not from the ACS text. The 2004 paper used Daylight cLogP and nine descriptors. This tool uses Crippen logP and four descriptors.

   The manual route that the harness recorded

   ga_rdkit.fit_solubility_model(descriptors="desc2", equation="published", test_fraction=0.2, seed=42, n_outliers=10)

   The manual route uses the same method. The note in the route gives the known difference.

5. run_script (step n6)

   Run the Python code in {work}/script-1/script.py

   • Code only: this step has no route in the program menus. Run it with the script or flow export.

   The program has no menu route for this step. To repeat it, run the code.

Figure

Run facts

The session

This is the session as the scientist sees it, in order. Decision cards show the answer that the script gave and where that answer comes from. Each step shows the program, its version, the input file hash and the outputs. Click a line to see more.

- Setup values:
Molecules:
- Salts and multi-fragment molecules (salt_handling): keep
- SMILES that RDKit cannot parse (invalid_smiles): stop
Descriptors:
- logP method (logp_method): crippen
- Hydrogen bond donor and acceptor definition (hbd_hba_definition): lipinski
Ask the scientist: Rule of five cutoffs (rule_of_five_cutoffs), Violations allowed for a pass (allowed_violations), Solubility equation (esol_equation), Fraction of molecules held out for testing (test_fraction), Random seed for the train and test split (split_seed).
The harness fills these values into your tool calls automatically. If you give a different value, the harness logs a deviation and shows it to the scientist.
- Mode is now watch: you run the steps.

{
 "molecules": "mol1",
 "n_read": 1128,
 "n_parsed": 1128,
 "n_invalid": 0,
 "n_multi_fragment": 0,
 "invalid_ids": [],
 "salts": "keep",
 "rdkit_version": "2026.03.6",
 "invalid_handling": "stop"
}

{
 "descriptors": "desc2",
 "n_molecules": 1128,
 "rdkit_version": "2026.03.6",
 "mean_mw": 203.9370744680851,
 "mean_logp": 2.447519539007093,
 "mean_tpsa": 34.87288120567376,
 "mean_rotatable_bonds": 2.1773049645390072,
 "mean_aromatic_proportion": 0.36354385527552907,
 "table": {
  "columns": [
   "id",
   "smiles",
   "mw",
   "logp",
   "tpsa",
   "hbd",
   "hba",
   "rb",
   "ap",
   "measured"
  ],
  "rows": [
   [
    "Amigdalin",
    "N#CC(OC1OC(COC2OC(CO)C(O)C(O)C2O)C(O)C(O)C1O)c1ccccc1",
    457.432,
    -3.108,
    202.32,
    7,
    12,
    7,
    0.1875,
    -0.77
   ],
   [
    "Fenfuram",
    "Cc1occc1C(=O)Nc1ccccc1",
    201.225,
    2.8403,
    42.24,
    1,
    2,
    2,
    0.7333,
    -3.3
   ],
   [
    "citral",
    "CC(C)=CCCC(C)=CC=O",
    152.237,
    2.878,
    17.07,
    0,
    1,
    4,
    0,
    -2.06
   ],
   [
    "Picene",
    "c1ccc2c(c1)ccc1c2ccc2c3ccccc3ccc21",
    278.354,
    6.2994,
    0,
    0,
    0,
    0,
    1,
    -7.87
   ],
   [
    "Thiophene",
    "c1ccsc1",
    84.143,
    1.7481,
    0,
    0,
    1,
    0,
    1,
    -1.33
   ],
   [
    "benzothiazole",
    "c1ccc2scnc2c1",
    135.191,
    2.2963,
    12.89,
    0,
    2,
    0,
    1,
    -1.5
   ],
   [
    "2,2,4,6,6'-PCB",
    "Clc1cc(Cl)c(-c2c(Cl)cccc2Cl)c(Cl)c1",
    326.437,
    6.6206,
    0,
    0,
    0,
    1,
    0.7059,
    -7.32
   ],
   [
    "Estradiol",
    "CC12CCC3c4ccc(O)cc4CCC3C1CCC2O",
    272.388,
    3.6092,
    40.46,
    2,
    2,
    0,
    0.3,
    -5.03
   ]
  ],
  "n_rows": 1128,
  "path": "descriptors.csv"
 }
}

{
 "n_molecules": 1128,
 "n_zero_violations": 1018,
 "n_one_violation": 98,
 "n_two_or_more_violations": 12,
 "n_pass": 1116,
 "n_fail": 12,
 "fraction_pass": 0.9893617021276596,
 "n_mw_over_500": 9,
 "n_logp_over_5": 97,
 "n_hbd_over_5": 10,
 "n_hba_over_10": 8,
 "allowed_violations": 1,
 "hbd_hba": "lipinski"
}

{
 "equation": "published",
 "intercept": 0.16,
 "coef_logp": -0.63,
 "coef_mw": -0.0062,
 "coef_rb": 0.066,
 "coef_ap": -0.74,
 "all_r2": 0.7247613093988002,
 "all_r_squared_correlation": 0.7519384878143138,
 "all_rmse": 1.0993724206558089,
 "all_n": 1128,
 "outliers": [
  {
   "id": "Tetradecane",
   "smiles": "CCCCCCCCCCCCCC",
   "measured": -7.96,
   "predicted": -3.939704800000003,
   "residual": -4.020295199999997
  },
  {
   "id": "1-Octadecanol",
   "smiles": "CCCCCCCCCCCCCCCCCCO",
   "measured": -8.4,
   "predicted": -4.392432200000004,
   "residual": -4.007567799999997
  },
  {
   "id": "Hexadecane",
   "smiles": "CCCCCCCCCCCCCCCC",
   "measured": -8.4,
   "predicted": -4.473165600000003,
   "residual": -3.926834399999997
  },
  {
   "id": "uric acid",
   "smiles": "O=c1[nH]c(=O)c2[nH]c(=O)[nH]c2[nH]1",
   "measured": -3.93,
   "predicted": -0.32395839999999965,
   "residual": -3.6060416000000006
  },
  {
   "id": "vamidothion",
   "smiles": "CNC(=O)C(C)SCCSP(=O)(OC)OC",
   "measured": 1.1440000000000001,
   "predicted": -2.3460925999999995,
   "residual": 3.4900925999999997
  },
  {
   "id": "alloxantin",
   "smiles": "O=C1NC(=O)C(O)(C2(O)C(=O)NC(=O)NC2=O)C(=O)N1",
   "measured": -1.99,
   "predicted": 1.4880547999999985,
   "residual": -3.4780547999999984
  },
  {
   "id": "bromadiolone",
   "smiles": "O=c1oc2ccccc2c(O)c1C(CC(O)c1ccc(-c2ccc(Br)cc2)cc1)c1ccccc1",
   "measured": -4.445,
   "predicted": -7.831697800000005,
   "residual": 3.3866978000000048
  },
  {
   "id": "Benzo(b)fluorene",
   "smiles": "c1ccc2c(c1)Cc1cc3ccccc3cc1-2",
   "measured": -8.04,
   "predicted": -4.656355188235295,
   "residual": -3.3836448117647038
  },
  {
   "id": "Napthacene",
   "smiles": "c1ccc2cc3cc4ccccc4cc3cc2c1",
   "measured": -8.6,
   "predicted": -5.2375288000000015,
   "residual": -3.362471199999998
  },
  {
   "id": "Nonane",
   "smiles": "CCCCCCCCC",
   "measured": -5.88,
   "predicted": -2.6060528000000014,
   "residual": -3.2739471999999985
  }
 ]
}

- 6 numbers in the answer match no logged tool result: 1997, 500, 0.63, 0.0062, 0.74, 2004. Run the tool that measures each number, or remove the number, or say how you derived it from logged results.
- The answer says RDKit version 2026.03.6. No logged step reports an RDKit version. The version has no source.
Fix these problems. Then give the answer again.

The final answer

The model wrote this answer. The harness shows it to the scientist after the review.

Settings used, from the decision record: Salts and multi-fragment molecules: keep · SMILES that RDKit cannot parse: stop · logP method: crippen · Hydrogen bond donor and acceptor definition: lipinski · Rule of five cutoffs: lipinski_1997 · Violations allowed for a pass: 1 · Solubility equation: published · Fraction of molecules held out for testing: 0.2 · Random seed for the train and test split: 42.

Checks

Review findings

The review recorded 10 findings. A rule finding comes from a fixed check in the harness. A referee finding comes from a second model that reads the record. The harness shows the findings to the scientist with the final answer. The record does not mark a finding as fixed. Thus a finding from an early review round can apply to a draft that the model corrected later.

Numbers in the answer

The last claim check read 65 numbers in the answer. 62 numbers match a logged result. 3 numbers have no source in the record.

Deviations

The model did not try to change a choice of the scientist.

Failed tool calls

1 tool call failed. The model then tried again or used another tool. The session above shows each failure.

Data integrity

Each data file has the same SHA-256 hash now as at the time of the step that read it. Where the download script (fetch.sh) gives a hash, the file also has that hash. The run did not change the data.

A SHA-256 hash is a fingerprint of the file contents. If one byte of the file changes, the hash changes. The table shows the first 12 characters.

How to repeat it

Get the data. The script downloads the files and checks their SHA-256 hashes where it lists them.

CUVETTE_DATA={data} bash bench/papers/delaney2004-esol/fetch.sh

Run the same case with Cuvette. The script gives the same answers from bench/papers/delaney2004-esol/bench.yaml.

cuvette bench papers --papers delaney2004-esol --models claude:claude-sonnet-5-5

Repeat each step by hand in the program. For each step, the harness records a manual route: the menu path or the code that gives the same result. This list does not include comparison runs.

1. load_molecules (step n1)

   In Python

   ga_rdkit.load_molecules(path, smiles_column, id_column, target_column, salts, invalid). Each row calls Chem.MolFromSmiles(smiles.strip()).

   • CSV file

     {data}/delaney2004-esol/delaney-processed.csv

   • SMILES column = smiles
   • Fragment handling = keep
   • Invalid SMILES = stop
   • Warning: If you keep the default , you get a different result.
   • Note: The route needs the wrapper module ga_rdkit.py in the adapter folder. No person ran the route by hand.

   The manual route that the harness recorded

   ga_rdkit.load_molecules(path="{data}/delaney2004-esol/delaney-processed.csv", smiles_column="smiles", id_column="Compound ID", target_column="measured log solubility in mols per litre", salts="keep", invalid="stop")

   The manual route uses the same method. The note in the route gives the known difference.

2. compute_descriptors (step n2)

   In Python

   Descriptors.MolWt(m), Crippen.MolLogP(m), rdMolDescriptors.CalcTPSA(m), Lipinski.NumHDonors(m), Lipinski.NumHAcceptors(m), Lipinski.NumRotatableBonds(m), aromatic heavy atoms / m.GetNumHeavyAtoms()

   • logP function = crippen
   • Donor and acceptor functions = lipinski

   The manual route that the harness recorded

   ga_rdkit.compute_descriptors(molecules="mol1", logp_method="crippen", hbd_hba="lipinski")

   The manual route gives the same numbers. An automatic test in Cuvette checks this.

3. rule_of_five (step n3)

   In Python

   count a violation when MolWt >
   500, MolLogP >
   5, NumHDonors >
   5 or NumHAcceptors >
   10.

   • Cutoffs = lipinski_1997
   • Violations allowed = 1
   • Note: The cutoffs come from Lipinski 1997 as cited in the benchmark notes. The paper was not read. No person ran the route by hand.

   The manual route that the harness recorded

   ga_rdkit.rule_of_five(descriptors="desc2", cutoffs="lipinski_1997", allowed_violations=1)

   The manual route uses the same method. The note in the route gives the known difference.

4. fit_solubility_model (step n4)

   In Python

   logS = 0.16 - 0.63*logP - 0.0062*MW + 0.066*RB - 0.74*AP, or numpy.linalg.lstsq on a random training split.

   • Equation = published
   • Test fraction = 0.2
   • Random seed = 42
   • Note: The published equation comes from a secondary source (a 2018 blog post), not from the ACS text. The 2004 paper used Daylight cLogP and nine descriptors. This tool uses Crippen logP and four descriptors.

   The manual route that the harness recorded

   ga_rdkit.fit_solubility_model(descriptors="desc2", equation="published", test_fraction=0.2, seed=42, n_outliers=10)

   The manual route uses the same method. The note in the route gives the known difference.

Figure

Run facts

The session

This is the session as the scientist sees it, in order. Decision cards show the answer that the script gave and where that answer comes from. Each step shows the program, its version, the input file hash and the outputs. Click a line to see more.

- Setup values:
Molecules:
- Salts and multi-fragment molecules (salt_handling): keep
- SMILES that RDKit cannot parse (invalid_smiles): stop
Descriptors:
- logP method (logp_method): crippen
- Hydrogen bond donor and acceptor definition (hbd_hba_definition): lipinski
Ask the scientist: Rule of five cutoffs (rule_of_five_cutoffs), Violations allowed for a pass (allowed_violations), Solubility equation (esol_equation), Fraction of molecules held out for testing (test_fraction), Random seed for the train and test split (split_seed).
The harness fills these values into your tool calls automatically. If you give a different value, the harness logs a deviation and shows it to the scientist.
- Mode is now watch: you run the steps.

{
 "molecules": "mol1",
 "n_read": 1128,
 "n_parsed": 1128,
 "n_invalid": 0,
 "n_multi_fragment": 0,
 "invalid_ids": [],
 "salts": "keep",
 "rdkit_version": "2026.03.6",
 "invalid_handling": "stop"
}

{
 "descriptors": "desc2",
 "n_molecules": 1128,
 "rdkit_version": "2026.03.6",
 "mean_mw": 203.9370744680851,
 "mean_logp": 2.447519539007093,
 "mean_tpsa": 34.87288120567376,
 "mean_rotatable_bonds": 2.1773049645390072,
 "mean_aromatic_proportion": 0.36354385527552907,
 "table": {
  "columns": [
   "id",
   "smiles",
   "mw",
   "logp",
   "tpsa",
   "hbd",
   "hba",
   "rb",
   "ap",
   "measured"
  ],
  "rows": [
   [
    "1",
    "N#CC(OC1OC(COC2OC(CO)C(O)C(O)C2O)C(O)C(O)C1O)c1ccccc1",
    457.432,
    -3.108,
    202.32,
    7,
    12,
    7,
    0.1875,
    -0.77
   ],
   [
    "2",
    "Cc1occc1C(=O)Nc1ccccc1",
    201.225,
    2.8403,
    42.24,
    1,
    2,
    2,
    0.7333,
    -3.3
   ],
   [
    "3",
    "CC(C)=CCCC(C)=CC=O",
    152.237,
    2.878,
    17.07,
    0,
    1,
    4,
    0,
    -2.06
   ],
   [
    "4",
    "c1ccc2c(c1)ccc1c2ccc2c3ccccc3ccc21",
    278.354,
    6.2994,
    0,
    0,
    0,
    0,
    1,
    -7.87
   ],
   [
    "5",
    "c1ccsc1",
    84.143,
    1.7481,
    0,
    0,
    1,
    0,
    1,
    -1.33
   ],
   [
    "6",
    "c1ccc2scnc2c1",
    135.191,
    2.2963,
    12.89,
    0,
    2,
    0,
    1,
    -1.5
   ],
   [
    "7",
    "Clc1cc(Cl)c(-c2c(Cl)cccc2Cl)c(Cl)c1",
    326.437,
    6.6206,
    0,
    0,
    0,
    1,
    0.7059,
    -7.32
   ],
   [
    "8",
    "CC12CCC3c4ccc(O)cc4CCC3C1CCC2O",
    272.388,
    3.6092,
    40.46,
    2,
    2,
    0,
    0.3,
    -5.03
   ]
  ],
  "n_rows": 1128,
  "path": "descriptors.csv"
 }
}

{
 "n_molecules": 1128,
 "n_zero_violations": 1018,
 "n_one_violation": 98,
 "n_two_or_more_violations": 12,
 "n_pass": 1116,
 "n_fail": 12,
 "fraction_pass": 0.9893617021276596,
 "n_mw_over_500": 9,
 "n_logp_over_5": 97,
 "n_hbd_over_5": 10,
 "n_hba_over_10": 8,
 "allowed_violations": 1,
 "hbd_hba": "lipinski"
}

{
 "equation": "published",
 "intercept": 0.16,
 "coef_logp": -0.63,
 "coef_mw": -0.0062,
 "coef_rb": 0.066,
 "coef_ap": -0.74,
 "all_r2": 0.7247613093988002,
 "all_r_squared_correlation": 0.7519384878143138,
 "all_rmse": 1.0993724206558089,
 "all_n": 1128,
 "outliers": [
  {
   "id": "15",
   "smiles": "CCCCCCCCCCCCCC",
   "measured": -7.96,
   "predicted": -3.939704800000003,
   "residual": -4.020295199999997
  },
  {
   "id": "789",
   "smiles": "CCCCCCCCCCCCCCCCCCO",
   "measured": -8.4,
   "predicted": -4.392432200000004,
   "residual": -4.007567799999997
  },
  {
   "id": "976",
   "smiles": "CCCCCCCCCCCCCCCC",
   "measured": -8.4,
   "predicted": -4.473165600000003,
   "residual": -3.926834399999997
  },
  {
   "id": "853",
   "smiles": "O=c1[nH]c(=O)c2[nH]c(=O)[nH]c2[nH]1",
   "measured": -3.93,
   "predicted": -0.32395839999999965,
   "residual": -3.6060416000000006
  },
  {
   "id": "1065",
   "smiles": "CNC(=O)C(C)SCCSP(=O)(OC)OC",
   "measured": 1.1440000000000001,
   "predicted": -2.3460925999999995,
   "residual": 3.4900925999999997
  },
  {
   "id": "734",
   "smiles": "O=C1NC(=O)C(O)(C2(O)C(=O)NC(=O)NC2=O)C(=O)N1",
   "measured": -1.99,
   "predicted": 1.4880547999999985,
   "residual": -3.4780547999999984
  },
  {
   "id": "940",
   "smiles": "O=c1oc2ccccc2c(O)c1C(CC(O)c1ccc(-c2ccc(Br)cc2)cc1)c1ccccc1",
   "measured": -4.445,
   "predicted": -7.831697800000005,
   "residual": 3.3866978000000048
  },
  {
   "id": "846",
   "smiles": "c1ccc2c(c1)Cc1cc3ccccc3cc1-2",
   "measured": -8.04,
   "predicted": -4.656355188235295,
   "residual": -3.3836448117647038
  },
  {
   "id": "820",
   "smiles": "c1ccc2cc3cc4ccccc4cc3cc2c1",
   "measured": -8.6,
   "predicted": -5.2375288000000015,
   "residual": -3.362471199999998
  },
  {
   "id": "735",
   "smiles": "CCCCCCCCC",
   "measured": -5.88,
   "predicted": -2.6060528000000014,
   "residual": -3.2739471999999985
  }
 ]
}

{
 "equation": "published",
 "intercept": 0.16,
 "coef_logp": -0.63,
 "coef_mw": -0.0062,
 "coef_rb": 0.066,
 "coef_ap": -0.74,
 "all_r2": 0.7247613093988002,
 "all_r_squared_correlation": 0.7519384878143138,
 "all_rmse": 1.0993724206558089,
 "all_n": 1128,
 "outliers": [
  {
   "id": "15",
   "smiles": "CCCCCCCCCCCCCC",
   "measured": -7.96,
   "predicted": -3.939704800000003,
   "residual": -4.020295199999997
  },
  {
   "id": "789",
   "smiles": "CCCCCCCCCCCCCCCCCCO",
   "measured": -8.4,
   "predicted": -4.392432200000004,
   "residual": -4.007567799999997
  },
  {
   "id": "976",
   "smiles": "CCCCCCCCCCCCCCCC",
   "measured": -8.4,
   "predicted": -4.473165600000003,
   "residual": -3.926834399999997
  },
  {
   "id": "853",
   "smiles": "O=c1[nH]c(=O)c2[nH]c(=O)[nH]c2[nH]1",
   "measured": -3.93,
   "predicted": -0.32395839999999965,
   "residual": -3.6060416000000006
  },
  {
   "id": "1065",
   "smiles": "CNC(=O)C(C)SCCSP(=O)(OC)OC",
   "measured": 1.1440000000000001,
   "predicted": -2.3460925999999995,
   "residual": 3.4900925999999997
  },
  {
   "id": "734",
   "smiles": "O=C1NC(=O)C(O)(C2(O)C(=O)NC(=O)NC2=O)C(=O)N1",
   "measured": -1.99,
   "predicted": 1.4880547999999985,
   "residual": -3.4780547999999984
  },
  {
   "id": "940",
   "smiles": "O=c1oc2ccccc2c(O)c1C(CC(O)c1ccc(-c2ccc(Br)cc2)cc1)c1ccccc1",
   "measured": -4.445,
   "predicted": -7.831697800000005,
   "residual": 3.3866978000000048
  },
  {
   "id": "846",
   "smiles": "c1ccc2c(c1)Cc1cc3ccccc3cc1-2",
   "measured": -8.04,
   "predicted": -4.656355188235295,
   "residual": -3.3836448117647038
  },
  {
   "id": "820",
   "smiles": "c1ccc2cc3cc4ccccc4cc3cc2c1",
   "measured": -8.6,
   "predicted": -5.2375288000000015,
   "residual": -3.362471199999998
  },
  {
   "id": "735",
   "smiles": "CCCCCCCCC",
   "measured": -5.88,
   "predicted": -2.6060528000000014,
   "residual": -3.2739471999999985
  }
 ]
}

- Your plan still has steps that are not done: Load the SMILES and measured log solubility from the CSV file.; Compute descriptors (Crippen logP, Lipinski donor and acceptor counts).; Count rule of five violations.; Fit the ESOL solubility model and report test results.; Report counts, equation, R squared definition and RMSE.. Do the next one now with a tool. If a step cannot be done, call plan and mark it skipped with the reason. Then give the answer, with every number the request asks for.

- 6 numbers in the answer match no logged tool result: 1997, 500, 0.63, 0.0062, 0.74, 2004. Run the tool that measures each number, or remove the number, or say how you derived it from logged results.
- The answer describes the largest residuals and which molecules have them. The log has no residual values or per-molecule predictions. This claim has no source.
Fix these problems. Then give the answer again.

import pandas as pd, numpy as np
df = pd.read_csv("compute_descriptors-1/descriptors.csv")
df["pred"] = 0.16 - 0.63*df["logp"] - 0.0062*df["mw"] + 0.066*df["rb"] - 0.74*df["ap"]
df["resid"] = df["measured"] - df["pred"]
rmse = np.sqrt(np.mean(df["resid"]**2))
print("n =", len(df))
print("RMSE (log units) =", round(rmse, 4))
top = df.reindex(df["resid"].abs().sort_values(ascending=False).index).head(10)
print(top[["id","smiles","measured","pred","resid"]].to_string(index=False))
out = df.reindex(df["resid"].abs().sort_values(ascending=False).index)
out[["id","smiles","measured","pred","resid"]].to_csv("published_residuals.csv", index=False)

n = 1128
RMSE (log units) = 1.0994
  id                                                     smiles  measured      pred     resid
  15                                             CCCCCCCCCCCCCC    -7.960 -3.939705 -4.020295
 789                                        CCCCCCCCCCCCCCCCCCO    -8.400 -4.392432 -4.007568
 976                                           CCCCCCCCCCCCCCCC    -8.400 -4.473166 -3.926834
 853                        O=c1[nH]c(=O)c2[nH]c(=O)[nH]c2[nH]1    -3.930 -0.323958 -3.606042
1065                                 CNC(=O)C(C)SCCSP(=O)(OC)OC     1.144 -2.346093  3.490093
 734               O=C1NC(=O)C(O)(C2(O)C(=O)NC(=O)NC2=O)C(=O)N1    -1.990  1.488055 -3.478055
 940 O=c1oc2ccccc2c(O)c1C(CC(O)c1ccc(-c2ccc(Br)cc2)cc1)c1ccccc1    -4.445 -7.831698  3.386698
 846                               c1ccc2c(c1)Cc1cc3ccccc3cc1-2    -8.040 -4.656355 -3.383645
 820                                 c1ccc2cc3cc4ccccc4cc3cc2c1    -8.600 -5.237529 -3.362471
 735                                                  CCCCCCCCC    -5.880 -2.606053 -3.273947

The final answer

The model wrote this answer. The harness shows it to the scientist after the review.

Step results that the answer does not mention, from the record:
- n6 run_script: The script ran in {work} and wrote 1 new file(s) to {work}.

Settings used, from the decision record: Salts and multi-fragment molecules: keep · SMILES that RDKit cannot parse: stop · logP method: crippen · Hydrogen bond donor and acceptor definition: lipinski · Rule of five cutoffs: lipinski_1997 · Violations allowed for a pass: 1 · Solubility equation: published · Fraction of molecules held out for testing: 0.2 · Random seed for the train and test split: 42.

Checks

Review findings

The review recorded 7 findings. A rule finding comes from a fixed check in the harness. A referee finding comes from a second model that reads the record. The harness shows the findings to the scientist with the final answer. The record does not mark a finding as fixed. Thus a finding from an early review round can apply to a draft that the model corrected later.

Numbers in the answer

The last claim check read 46 numbers in the answer. 45 numbers match a logged result. 0 numbers have no source in the record.

Deviations

The model did not try to change a choice of the scientist.

Failed tool calls

No tool call failed.

Data integrity

Each data file has the same SHA-256 hash now as at the time of the step that read it. Where the download script (fetch.sh) gives a hash, the file also has that hash. The run did not change the data.

A SHA-256 hash is a fingerprint of the file contents. If one byte of the file changes, the hash changes. The table shows the first 12 characters.

How to repeat it

Get the data. The script downloads the files and checks their SHA-256 hashes where it lists them.

CUVETTE_DATA={data} bash bench/papers/delaney2004-esol/fetch.sh

Run the same case with Cuvette. The script gives the same answers from bench/papers/delaney2004-esol/bench.yaml.

cuvette bench papers --papers delaney2004-esol --models claude:claude-haiku-5-5

Repeat each step by hand in the program. For each step, the harness records a manual route: the menu path or the code that gives the same result. This list does not include comparison runs.

1. load_molecules (step n1)

   In Python

   ga_rdkit.load_molecules(path, smiles_column, id_column, target_column, salts, invalid). Each row calls Chem.MolFromSmiles(smiles.strip()).

   • CSV file

     {data}/delaney2004-esol/delaney-processed.csv

   • SMILES column = smiles
   • Fragment handling = keep
   • Invalid SMILES = stop
   • Warning: If you keep the default , you get a different result.
   • Note: The route needs the wrapper module ga_rdkit.py in the adapter folder. No person ran the route by hand.

   The manual route that the harness recorded

   ga_rdkit.load_molecules(path="{data}/delaney2004-esol/delaney-processed.csv", smiles_column="smiles", target_column="measured log solubility in mols per litre", salts="keep", invalid="stop")

   The manual route uses the same method. The note in the route gives the known difference.

2. compute_descriptors (step n2)

   In Python

   Descriptors.MolWt(m), Crippen.MolLogP(m), rdMolDescriptors.CalcTPSA(m), Lipinski.NumHDonors(m), Lipinski.NumHAcceptors(m), Lipinski.NumRotatableBonds(m), aromatic heavy atoms / m.GetNumHeavyAtoms()

   • logP function = crippen
   • Donor and acceptor functions = lipinski

   The manual route that the harness recorded

   ga_rdkit.compute_descriptors(molecules="mol1", logp_method="crippen", hbd_hba="lipinski")

   The manual route gives the same numbers. An automatic test in Cuvette checks this.

3. rule_of_five (step n3)

   In Python

   count a violation when MolWt >
   500, MolLogP >
   5, NumHDonors >
   5 or NumHAcceptors >
   10.

   • Cutoffs = lipinski_1997
   • Violations allowed = 1
   • Note: The cutoffs come from Lipinski 1997 as cited in the benchmark notes. The paper was not read. No person ran the route by hand.

   The manual route that the harness recorded

   ga_rdkit.rule_of_five(descriptors="desc2", cutoffs="lipinski_1997", allowed_violations=1)

   The manual route uses the same method. The note in the route gives the known difference.

4. fit_solubility_model (step n4)

   In Python

   logS = 0.16 - 0.63*logP - 0.0062*MW + 0.066*RB - 0.74*AP, or numpy.linalg.lstsq on a random training split.

   • Equation = published
   • Test fraction = 0.2
   • Random seed = 42
   • Note: The published equation comes from a secondary source (a 2018 blog post), not from the ACS text. The 2004 paper used Daylight cLogP and nine descriptors. This tool uses Crippen logP and four descriptors.

   The manual route that the harness recorded

   ga_rdkit.fit_solubility_model(descriptors="desc2", equation="published", test_fraction=0.2, seed=42, n_outliers=10)

   The manual route uses the same method. The note in the route gives the known difference.

5. fit_solubility_model (step n5)

   In Python

   logS = 0.16 - 0.63*logP - 0.0062*MW + 0.066*RB - 0.74*AP, or numpy.linalg.lstsq on a random training split.

   • Equation = published
   • Test fraction = 0.2
   • Random seed = 42
   • Note: The published equation comes from a secondary source (a 2018 blog post), not from the ACS text. The 2004 paper used Daylight cLogP and nine descriptors. This tool uses Crippen logP and four descriptors.

   The manual route that the harness recorded

   ga_rdkit.fit_solubility_model(descriptors="desc2", equation="published", test_fraction=0.2, seed=42, n_outliers=10)

   The manual route uses the same method. The note in the route gives the known difference.

6. run_script (step n6)

   Run the Python code in {work}/script-1/script.py

   • Code only: this step has no route in the program menus. Run it with the script or flow export.

   The program has no menu route for this step. To repeat it, run the code.

Figure

Run facts

The session

This is the session as the scientist sees it, in order. Decision cards show the answer that the script gave and where that answer comes from. Each step shows the program, its version, the input file hash and the outputs. Click a line to see more.

- Setup values:
Molecules:
- Salts and multi-fragment molecules (salt_handling): keep
- SMILES that RDKit cannot parse (invalid_smiles): stop
Descriptors:
- logP method (logp_method): crippen
- Hydrogen bond donor and acceptor definition (hbd_hba_definition): lipinski
Ask the scientist: Rule of five cutoffs (rule_of_five_cutoffs), Violations allowed for a pass (allowed_violations), Solubility equation (esol_equation), Fraction of molecules held out for testing (test_fraction), Random seed for the train and test split (split_seed).
The harness fills these values into your tool calls automatically. If you give a different value, the harness logs a deviation and shows it to the scientist.
- Mode is now watch: you run the steps.

- You said what you will do next, but you did not call a tool. Do it now with a tool call, or give your final answer.

Traceback (most recent call last):
  File "{other volume}/tools/overnight/checkout/src/adapters/kinds/python_worker.py", line 137, in <module>
    reply = handle_request(req)
  File "{other volume}/tools/overnight/checkout/src/adapters/kinds/python_worker.py", line 119, in handle_request
    result = fn(**args)
  File "{other volume}/tools/overnight/checkout/catalog/rdkit/ga_rdkit.py", line 73, in load_molecules
    i_key = _find(rows[0], id_column) if id_column else None
            ~~~~~^^^^^^^^^^^^^^^^^^^^
  File "{other volume}/tools/overnight/checkout/catalog/rdkit/ga_rdkit.py", line 64, in _find
    raise KeyError("no column named %r. Columns: %s" % (name, ", ".join(row.keys())))
KeyError: "no column named 'id'. Columns: Compound ID, ESOL predicted log solubility in mols per litre, Minimum Degree, Molecular Weight, Number of H-Bond Donors, Number of Rings, Number of Rotatable Bonds, Polar Surface Area, measured log solubility in mols per litre, smiles"

- You said what you will do next, but you did not call a tool. Do it now with a tool call, or give your final answer.

{
 "molecules": "mol1",
 "n_read": 1128,
 "n_parsed": 1128,
 "n_invalid": 0,
 "n_multi_fragment": 0,
 "invalid_ids": [],
 "salts": "keep",
 "rdkit_version": "2026.03.6",
 "invalid_handling": "stop"
}

{
 "descriptors": "desc2",
 "n_molecules": 1128,
 "rdkit_version": "2026.03.6",
 "mean_mw": 203.9370744680851,
 "mean_logp": 2.447519539007093,
 "mean_tpsa": 34.87288120567376,
 "mean_rotatable_bonds": 2.1773049645390072,
 "mean_aromatic_proportion": 0.36354385527552907,
 "table": {
  "columns": [
   "id",
   "smiles",
   "mw",
   "logp",
   "tpsa",
   "hbd",
   "hba",
   "rb",
   "ap",
   "measured"
  ],
  "rows": [
   [
    "1",
    "N#CC(OC1OC(COC2OC(CO)C(O)C(O)C2O)C(O)C(O)C1O)c1ccccc1",
    457.432,
    -3.108,
    202.32,
    7,
    12,
    7,
    0.1875,
    -0.77
   ],
   [
    "2",
    "Cc1occc1C(=O)Nc1ccccc1",
    201.225,
    2.8403,
    42.24,
    1,
    2,
    2,
    0.7333,
    -3.3
   ],
   [
    "3",
    "CC(C)=CCCC(C)=CC=O",
    152.237,
    2.878,
    17.07,
    0,
    1,
    4,
    0,
    -2.06
   ],
   [
    "4",
    "c1ccc2c(c1)ccc1c2ccc2c3ccccc3ccc21",
    278.354,
    6.2994,
    0,
    0,
    0,
    0,
    1,
    -7.87
   ],
   [
    "5",
    "c1ccsc1",
    84.143,
    1.7481,
    0,
    0,
    1,
    0,
    1,
    -1.33
   ],
   [
    "6",
    "c1ccc2scnc2c1",
    135.191,
    2.2963,
    12.89,
    0,
    2,
    0,
    1,
    -1.5
   ],
   [
    "7",
    "Clc1cc(Cl)c(-c2c(Cl)cccc2Cl)c(Cl)c1",
    326.437,
    6.6206,
    0,
    0,
    0,
    1,
    0.7059,
    -7.32
   ],
   [
    "8",
    "CC12CCC3c4ccc(O)cc4CCC3C1CCC2O",
    272.388,
    3.6092,
    40.46,
    2,
    2,
    0,
    0.3,
    -5.03
   ]
  ],
  "n_rows": 1128,
  "path": "descriptors.csv"
 }
}

{
 "n_molecules": 1128,
 "n_zero_violations": 1018,
 "n_one_violation": 98,
 "n_two_or_more_violations": 12,
 "n_pass": 1116,
 "n_fail": 12,
 "fraction_pass": 0.9893617021276596,
 "n_mw_over_500": 9,
 "n_logp_over_5": 97,
 "n_hbd_over_5": 10,
 "n_hba_over_10": 8,
 "allowed_violations": 1,
 "hbd_hba": "lipinski"
}

- 1 number in the answer match no logged tool result: 1997. Run the tool that measures each number, or remove the number, or say how you derived it from logged results.
Fix these problems. Then give the answer again.

The final answer

The model wrote this answer. The harness shows it to the scientist after the review.

Step results that the answer does not mention, from the record:
- n1 load_molecules: 1128 of 1128 SMILES parsed, 0 invalid.
- n2 compute_descriptors: Descriptors for 1128 molecules. Mean MW 203.937, mean logP 2.44752.

Settings used, from the decision record: Salts and multi-fragment molecules: keep · SMILES that RDKit cannot parse: stop · logP method: crippen · Hydrogen bond donor and acceptor definition: lipinski · Rule of five cutoffs: lipinski_1997 · Violations allowed for a pass: 1.

Checks

Review findings

The review recorded 7 findings. A rule finding comes from a fixed check in the harness. A referee finding comes from a second model that reads the record. The harness shows the findings to the scientist with the final answer. The record does not mark a finding as fixed. Thus a finding from an early review round can apply to a draft that the model corrected later.

Numbers in the answer

The last claim check read 10 numbers in the answer. 5 numbers match a logged result. 3 numbers have no source in the record.

Deviations

The model did not try to change a choice of the scientist.

Failed tool calls

1 tool call failed. The model then tried again or used another tool. The session above shows each failure.

Data integrity

Each data file has the same SHA-256 hash now as at the time of the step that read it. Where the download script (fetch.sh) gives a hash, the file also has that hash. The run did not change the data.

A SHA-256 hash is a fingerprint of the file contents. If one byte of the file changes, the hash changes. The table shows the first 12 characters.

How to repeat it

Get the data. The script downloads the files and checks their SHA-256 hashes where it lists them.

CUVETTE_DATA={data} bash bench/papers/delaney2004-esol/fetch.sh

Run the same case with Cuvette. The script gives the same answers from bench/papers/delaney2004-esol/bench.yaml.

cuvette bench papers --papers delaney2004-esol --models ollama:qwen3:8b

Repeat each step by hand in the program. For each step, the harness records a manual route: the menu path or the code that gives the same result. This list does not include comparison runs.

1. load_molecules (step n1)

   In Python

   ga_rdkit.load_molecules(path, smiles_column, id_column, target_column, salts, invalid). Each row calls Chem.MolFromSmiles(smiles.strip()).

   • CSV file

     {data}/delaney2004-esol/delaney-processed.csv

   • SMILES column = smiles
   • Fragment handling = keep
   • Invalid SMILES = stop
   • Warning: If you keep the default , you get a different result.
   • Note: The route needs the wrapper module ga_rdkit.py in the adapter folder. No person ran the route by hand.

   The manual route that the harness recorded

   ga_rdkit.load_molecules(path="{data}/delaney2004-esol/delaney-processed.csv", smiles_column="smiles", target_column="measured log solubility in mols per litre", salts="keep", invalid="stop")

   The manual route uses the same method. The note in the route gives the known difference.

2. compute_descriptors (step n2)

   In Python

   Descriptors.MolWt(m), Crippen.MolLogP(m), rdMolDescriptors.CalcTPSA(m), Lipinski.NumHDonors(m), Lipinski.NumHAcceptors(m), Lipinski.NumRotatableBonds(m), aromatic heavy atoms / m.GetNumHeavyAtoms()

   • logP function = crippen
   • Donor and acceptor functions = lipinski

   The manual route that the harness recorded

   ga_rdkit.compute_descriptors(molecules="mol1", logp_method="crippen", hbd_hba="lipinski")

   The manual route gives the same numbers. An automatic test in Cuvette checks this.

3. rule_of_five (step n3)

   In Python

   count a violation when MolWt >
   500, MolLogP >
   5, NumHDonors >
   5 or NumHAcceptors >
   10.

   • Cutoffs = lipinski_1997
   • Violations allowed = 1
   • Note: The cutoffs come from Lipinski 1997 as cited in the benchmark notes. The paper was not read. No person ran the route by hand.

   The manual route that the harness recorded

   ga_rdkit.rule_of_five(descriptors="desc2", cutoffs="lipinski_1997", allowed_violations=1)

   The manual route uses the same method. The note in the route gives the known difference.

Figure

Run facts