Lackey 2015: risk factors for tuberculosis treatment default in Lima, Peru

The session

This is the session as the scientist sees it, in order. Decision cards show the answer that the script gave and where that answer comes from. Each step shows the program, its version, the input file hash and the outputs. Click a line to see more.

- Setup values:
Design:
- What is the unit of replication? (replicate): subjects, patients or animals (one value per subject)
Inference:
- Significance level (alpha) (alpha): 0.05
Ask the scientist: Paired test (paired), Test family for two groups (two_group_test), Assume equal variances (unpaired t-test) (equal_variance), Sidedness (alternative), ANOVA variance assumption (anova_variance), Pairwise comparisons after ANOVA (posthoc), Correction for a list of p-values (p_adjust_method), Model formula (model_formula), Standard errors for the linear model (standard_errors), Random effects of the mixed model (random_effects), Fit by REML (reml), Covariates of the Cox model (cox_covariates), Ties method of the Cox model (ties), Strata of the Cox model (cox_strata), Type of sums of squares (ss_type), Comparisons of the cell means after a factorial ANOVA (factorial_posthoc), Sphericity correction for repeated measures (sphericity_correction), Correction after a rank test over many groups (rank_p_adjust), Exact or approximate p value for a rank test (exact_p), Covariates of the logistic model (logistic_covariates), Outcome value that counts as the event (logistic_event), Reference level of each categorical covariate (reference_levels), Yates continuity correction for a 2x2 chi-square test (chi_square_correction), Sidedness of the Fisher exact test (fisher_alternative), Method of the McNemar test (mcnemar_method).
The harness fills these values into your tool calls automatically. If you give a different value, the harness logs a deviation and shows it to the scientist.
- Mode is now watch: you run the steps.

{
 "ok": true,
 "summary": "1233 rows x 19 columns.",
 "metrics": {
  "n_rows": 1233,
  "n_columns": 19
 },
 "table": {
  "columns": [
   "column",
   "dtype",
   "n_missing",
   "n_unique",
   "min",
   "max",
   "first_values"
  ],
  "rows": [
   [
    "id",
    "int64",
    0,
    1233,
    883,
    2347,
    "1008, 1009, 1010, 1011, 1014"
   ],
   [
    "age_group",
    "object",
    0,
    4,
    null,
    null,
    "38 and older, 27 to 37, 22 to 26, 21 and younger"
   ],
   [
    "sex",
    "object",
    0,
    2,
    null,
    null,
    "Male, Female"
   ],
   [
    "marital_status",
    "object",
    0,
    4,
    null,
    null,
    "Single, Married/cohabitating, Divorced/separated, Widowed"
   ],
   [
    "poverty",
    "object",
    0,
    2,
    null,
    null,
    "Not in poverty, Poverty/extreme poverty"
   ],
   [
    "prison",
    "object",
    0,
    2,
    null,
    null,
    "No, Yes"
   ],
   [
    "secondary_education",
    "object",
    0,
    2,
    null,
    null,
    "Yes, No"
   ],
   [
    "smoking",
    "object",
    0,
    3,
    null,
    null,
    "Used to smoke, Never smoked, Currently smokes"
   ],
   [
    "alcohol_weekly",
    "object",
    0,
    2,
    null,
    null,
    "Yes, No"
   ],
   [
    "drug_use",
    "object",
    0,
    2,
    null,
    null,
    "No, Yes"
   ],
   [
    "rehab",
    "object",
    0,
    2,
    null,
    null,
    "No, Yes"
   ],
   [
    "mdr_tb",
    "object",
    0,
    2,
    null,
    null,
    "No, Yes"
   ],
   [
    "bmi",
    "object",
    0,
    3,
    null,
    null,
    "Underweight, Normal, Overweight/Obese"
   ],
   [
    "chronic_disease",
    "object",
    0,
    2,
    null,
    null,
    "No, Yes"
   ],
   [
    "hiv_status",
    "object",
    0,
    3,
    null,
    null,
    "Negative, Test not done, Positive"
   ]
  ],
  "n_rows": 19,
  "path": "{work}/inspect_table-1/columns.csv"
 }
}

column,dtype,n_missing,n_unique,min,max,first_values
id,int64,0,1233,883.0,2347.0,"1008, 1009, 1010, 1011, 1014"
age_group,object,0,4,,,"38 and older, 27 to 37, 22 to 26, 21 and younger"
sex,object,0,2,,,"Male, Female"
marital_status,object,0,4,,,"Single, Married/cohabitating, Divorced/separated, Widowed"
poverty,object,0,2,,,"Not in poverty, Poverty/extreme poverty"
prison,object,0,2,,,"No, Yes"
secondary_education,object,0,2,,,"Yes, No"
smoking,object,0,3,,,"Used to smoke, Never smoked, Currently smokes"
alcohol_weekly,object,0,2,,,"Yes, No"
drug_use,object,0,2,,,"No, Yes"
rehab,object,0,2,,,"No, Yes"
mdr_tb,object,0,2,,,"No, Yes"
bmi,object,0,3,,,"Underweight, Normal, Overweight/Obese"
chronic_disease,object,0,2,,,"No, Yes"
hiv_status,object,0,3,,,"Negative, Test not done, Positive"
diabetes,object,0,2,,,"No, Yes"
outcome,object,0,5,,,"Cured, Transferred out, Died, Default, Still in treatment"
default,int64,0,2,0.0,1.0,"0, 1"
days_to_default,float64,1114,104,2.0,973.0,"640.0, 353.0, 8.0, 107.0, 98.0"

{
 "ok": true,
 "summary": "Pearson chi-square test, 4 x 2 table, n = 1233: chi-square 12.43 on 3 df, p = 0.006039 (alpha 0.05). Cramer's V 0.1.",
 "metrics": {
  "chi2": 12.432624516640747,
  "df": 3,
  "p_value": 0.00603898829164167,
  "n": 1233,
  "cramers_v": 0.10041529546121677,
  "min_expected": 29.97323600973236,
  "n_cells_expected_below_5": 0,
  "fraction_cells_expected_below_5": 0
 },
 "data": {
  "test": "Pearson chi-square test",
  "correction": false,
  "alpha": 0.05,
  "significant": true,
  "warning": "",
  "expected": [
   [
    287.9367396593674,
    33.0632603406326
   ],
   [
    292.42173560421736,
    33.57826439578265
   ],
   [
    261.0267639902676,
    29.97323600973236
   ],
   [
    264.6147607461476,
    30.385239253852394
   ]
  ],
  "rows": [
   "21 and younger",
   "22 to 26",
   "27 to 37",
   "38 and older"
  ],
  "columns": [
   "0",
   "1"
  ]
 },
 "table": {
  "columns": [
   "age_group",
   "0",
   "1"
  ],
  "rows": [
   [
    "21 and younger",
    284,
    37
   ],
   [
    "22 to 26",
    290,
    36
   ],
   [
    "27 to 37",
    252,
    39
   ],
   [
    "38 and older",
    280,
    15
   ]
  ],
  "n_rows": 4,
  "path": "{work}/chi_square_test-1/observed_counts.csv"
 },
 "outputs": [
  {
   "path": "{work}/chi_square_test-1/expected_counts.csv",
   "kind": "table",
   "name": "expected_counts.csv"
  }
 ]
}

{
 "ok": true,
 "summary": "Pearson chi-square test, 2 x 2 table, n = 1233: chi-square 16.9 on 1 df, p = 3.945e-05 (alpha 0.05). Cramer's V 0.117.",
 "metrics": {
  "chi2": 16.897624469314554,
  "df": 1,
  "p_value": 0.00003945092420796785,
  "n": 1233,
  "cramers_v": 0.11706613733438456,
  "min_expected": 50.470397404703974,
  "n_cells_expected_below_5": 0,
  "fraction_cells_expected_below_5": 0
 },
 "data": {
  "test": "Pearson chi-square test",
  "correction": false,
  "alpha": 0.05,
  "significant": true,
  "warning": "",
  "expected": [
   [
    439.529602595296,
    50.470397404703974
   ],
   [
    666.4703974047039,
    76.52960259529603
   ]
  ],
  "rows": [
   "Female",
   "Male"
  ],
  "columns": [
   "0",
   "1"
  ]
 },
 "table": {
  "columns": [
   "sex",
   "0",
   "1"
  ],
  "rows": [
   [
    "Female",
    461,
    29
   ],
   [
    "Male",
    645,
    98
   ]
  ],
  "n_rows": 2,
  "path": "{work}/chi_square_test-2/observed_counts.csv"
 },
 "outputs": [
  {
   "path": "{work}/chi_square_test-2/expected_counts.csv",
   "kind": "table",
   "name": "expected_counts.csv"
  }
 ]
}

{
 "ok": true,
 "summary": "Pearson chi-square test, 4 x 2 table, n = 1233: chi-square 1.504 on 3 df, p = 0.6815 (alpha 0.05). Cramer's V 0.0349. Warning: 1 of 8 cells (12%) have an expected count below 5 (smallest 3.71). The chi-square p-value can be wrong. Use fisher_exact.",
 "metrics": {
  "chi2": 1.5035356413644951,
  "df": 3,
  "p_value": 0.6814545468936668,
  "n": 1233,
  "cramers_v": 0.03492008767758139,
  "min_expected": 3.708029197080292,
  "n_cells_expected_below_5": 1,
  "fraction_cells_expected_below_5": 0.125
 },
 "data": {
  "test": "Pearson chi-square test",
  "correction": false,
  "alpha": 0.05,
  "significant": false,
  "warning": "Warning: 1 of 8 cells (12%) have an expected count below 5 (smallest 3.71). The chi-square p-value can be wrong. Use fisher_exact.",
  "expected": [
   [
    81.62692619626927,
    9.373073803730739
   ],
   [
    420.6926196269262,
    48.307380373073805
   ],
   [
    571.3884833738848,
    65.61151662611516
   ],
   [
    32.29197080291971,
    3.708029197080292
   ]
  ],
  "rows": [
   "Divorced/separated",
   "Married/cohabitating",
   "Single",
   "Widowed"
  ],
  "columns": [
   "0",
   "1"
  ]
 },
 "table": {
  "columns": [
   "marital_status",
   "0",
   "1"
  ],
  "rows": [
   [
    "Divorced/separated",
    81,
    10
   ],
   [
    "Married/cohabitating",
    424,
    45
   ],
   [
    "Single",
    567,
    70
   ],
   [
    "Widowed",
    34,
    2
   ]
  ],
  "n_rows": 4,
  "path": "{work}/chi_square_test-3/observed_counts.csv"
 },
 "outputs": [
  {
   "path": "{work}/chi_square_test-3/expected_counts.csv",
   "kind": "table",
   "name": "expected_counts.csv"
  }
 ]
}

{
 "ok": true,
 "summary": "Pearson chi-square test, 2 x 2 table, n = 1233: chi-square 4.477 on 1 df, p = 0.03436 (alpha 0.05). Cramer's V 0.0603.",
 "metrics": {
  "chi2": 4.47652666051148,
  "df": 1,
  "p_value": 0.03436348964265445,
  "n": 1233,
  "cramers_v": 0.060254439286915616,
  "min_expected": 21.527169505271694,
  "n_cells_expected_below_5": 0,
  "fraction_cells_expected_below_5": 0
 },
 "data": {
  "test": "Pearson chi-square test",
  "correction": false,
  "alpha": 0.05,
  "significant": true,
  "warning": "",
  "expected": [
   [
    918.5271695052717,
    105.4728304947283
   ],
   [
    187.4728304947283,
    21.527169505271694
   ]
  ],
  "rows": [
   "Not in poverty",
   "Poverty/extreme poverty"
  ],
  "columns": [
   "0",
   "1"
  ]
 },
 "table": {
  "columns": [
   "poverty",
   "0",
   "1"
  ],
  "rows": [
   [
    "Not in poverty",
    927,
    97
   ],
   [
    "Poverty/extreme poverty",
    179,
    30
   ]
  ],
  "n_rows": 2,
  "path": "{work}/chi_square_test-4/observed_counts.csv"
 },
 "outputs": [
  {
   "path": "{work}/chi_square_test-4/expected_counts.csv",
   "kind": "table",
   "name": "expected_counts.csv"
  }
 ]
}

{
 "ok": true,
 "summary": "Fisher exact test, 4 x 2 table, n = 1233, two-sided: p = 0.742 (alpha 0.05). Method: exact (all tables with the same margins).",
 "metrics": {
  "n": 1233,
  "p_value": 0.741954958020354
 },
 "data": {
  "alternative": "two-sided",
  "alpha": 0.05,
  "rows": [
   "Divorced/separated",
   "Married/cohabitating",
   "Single",
   "Widowed"
  ],
  "columns": [
   "0",
   "1"
  ],
  "method": "exact (all tables with the same margins)",
  "significant": false
 },
 "table": {
  "columns": [
   "marital_status",
   "0",
   "1"
  ],
  "rows": [
   [
    "Divorced/separated",
    81,
    10
   ],
   [
    "Married/cohabitating",
    424,
    45
   ],
   [
    "Single",
    567,
    70
   ],
   [
    "Widowed",
    34,
    2
   ]
  ],
  "n_rows": 4,
  "path": "{work}/fisher_exact-1/fisher_counts.csv"
 }
}

import pandas as pd
base='{work}/'
for i in range(1,5):
    d=pd.read_csv(base+f'chi_square_test-{i}/observed_counts.csv')
    d['total']=d['0']+d['1']; d['pct_default']=(100*d['1']/d['total']).round(1)
    print(d.to_string(index=False)); print()

     age_group   0  1  total  pct_default
21 and younger 284 37    321         11.5
      22 to 26 290 36    326         11.0
      27 to 37 252 39    291         13.4
  38 and older 280 15    295          5.1

   sex   0  1  total  pct_default
Female 461 29    490          5.9
  Male 645 98    743         13.2

      marital_status   0  1  total  pct_default
  Divorced/separated  81 10     91         11.0
Married/cohabitating 424 45    469          9.6
              Single 567 70    637         11.0
             Widowed  34  2     36          5.6

                poverty   0  1  total  pct_default
         Not in poverty 927 97   1024          9.5
Poverty/extreme poverty 179 30    209         14.4

{"ok":true,"summary":"Logistic regression of default = 1 against 0, n = 1233, 127 events. drug_use[Yes] OR 4.782 (95% CI 3.052 to 7.491, p = 8.38e-12); bmi[Overweight/Obese] OR 0.8777 (95% CI 0.4444 to 1.733, p = 0.707); bmi[Underweight] OR 2.08 (95% CI 1.214 to 3.563, p = 0.0077); mdr_tb[Yes] OR 3.038 (95% CI 1.578 to 5.851, p = 0.000886); alcohol_weekly[Yes] OR 2.222 (95% CI 1.401 to 3.525, p = 0.000695); hiv_status[Positive] OR 1.392 (95% CI 0.4167 to 4.654, p = 0.591); hiv_status[Test not done] OR 2.303 (95% CI 1.5 to 3.536, p = 0.000137); secondary_education[No] OR 1.55 (95% CI 1.032 to 2.329, p = 0.0349). Likelihood-ratio test of the model: chi-square 139.5 on 8 df, p = 3.076e-26. Wald confidence intervals. Likelihood-ratio test of each categorical covariate: bmi chi-square 7.273 on 2 df, p = 0.02635; hiv_status chi-square 14.06 on 2 df, p = 0.0008845. Reference levels: drug_use = No, bmi = Normal, mdr_tb = No, alcohol_weekly = No, hiv_status = Negative, secondary_education = Yes.","metrics":{"n":1233,"events":127,"non_events":1106,"n_dropped":0,"log_likelihood":-339.16700926664714,"null_log_likelihood":-408.8958940401771,"lr_chi2":139.4577695470599,"lr_df":8,"lr_p":3.076324272310018e-26,"aic":696.3340185332943,"pseudo_r2_mcfadden":0.1705296770886101,"events_per_parameter":15.875,"converged":1,"coef_const":-3.497632150067933,"se_const":0.2124385984966824,"p_const":6.633082638853354e-61,"or_const":0.030268971015494868,"or_lo_const":0.01996041629112351,"or_hi_const":0.045901377655350385,"coef_drug_use_Yes":1.5647650527214947,"se_drug_use_Yes":0.2290396468725688,"p_drug_use_Yes":8.382509658407017e-12,"or_drug_use_Yes":4.78155139142713,"or_lo_drug_use_Yes":3.0521784873914464,"or_hi_drug_use_Yes":7.490791840420467,"coef_bmi_Overweight_Obese":-0.13043635986406035,"se_bmi_Overweight_Obese":0.3472058878406436,"p_bmi_Overweight_Obese":0.7071589799396698,"or_bmi_Overweight_Obese":0.8777123489045796,"or_lo_bmi_Overweight_Obese":0.4444368093830721,"or_hi_bmi_Overweight_Obese":1.7333824542772827,"coef_bmi_Underweight":0.7321286903849328,"se_bmi_Underweight":0.2747274170249356,"p_bmi_Underweight":0.007700488734103748,"or_bmi_Underweight":2.079502516579366,"or_lo_bmi_Underweight":1.2136991532262786,"or_hi_bmi_Underweight":3.5629346077773034,"coef_mdr_tb_Yes":1.1113587461354562,"se_mdr_tb_Yes":0.33430051843768804,"p_mdr_tb_Yes":0.0008859940232706892,"or_mdr_tb_Yes":3.0384841194494236,"or_lo_mdr_tb_Yes":1.577974038264568,"or_hi_mdr_tb_Yes":5.8507843096708845,"coef_alcohol_weekly_Yes":0.7984029127919758,"se_alcohol_weekly_Yes":0.23539707354057635,"p_alcohol_weekly_Yes":0.0006945327829807024,"or_alcohol_weekly_Yes":2.2219893823639905,"or_lo_alcohol_weekly_Yes":1.400785466210615,"or_hi_alcohol_weekly_Yes":3.5246202465924,"coef_hiv_status_Positive":0.3310904815764532,"se_hiv_status_Positive":0.6155952108100741,"p_hiv_status_Positive":0.5906889122875556,"or_hiv_status_Positive":1.3924857809169304,"or_lo_hiv_status_Positive":0.41667282366518404,"or_hi_hiv_status_
... (1000 more characters in the session record)

{"items":[{"name":"pct_default_overall","expression":"100 * 127 / 1233"}],"purpose":"Find the percent of patients who defaulted."}

pct_default_overall = 100 * 127 / 1233 = 10.3000811

- 1 number in the answer match no logged tool result: 95. Run the tool that measures each number, or remove the number, or say how you derived it from logged results.
Fix these problems. Then give the answer again.

{"items":[{"name":"confidence_level_pct","expression":"100 * (1 - 0.05)"}],"purpose":"Show that the confidence level comes from alpha 0.05."}

confidence_level_pct = 100 * (1 - 0.05) = 95

The final answer

The model wrote this answer. The harness shows it to the scientist after the review.

Settings used, from the decision record: Significance level (alpha): 0.05 · Covariates of the logistic model: drug_use, bmi, mdr_tb, alcohol_weekly, hiv_status, secondary_education · Outcome value that counts as the event: 1 · Reference level of each categorical covariate: drug_use=No, bmi=Normal, mdr_tb=No, alcohol_weekly=No, hiv_status=Negative, secondary_education=Yes · Yates continuity correction for a 2x2 chi-square test: false · Sidedness of the Fisher exact test: two-sided.

Values that are not scored

• A trap value is the result of a wrong method, for example an unpaired test on paired data. A trap value in a main step means that the run used the wrong method. A trap value in a comparison run is correct, because the record keeps the scientist's choice.
• An optional value is a second result of the same method. The run can compute it or not.
• A reference value comes from a different program version or a check run by us. We show it for comparison.

Checks

Review findings

The review recorded 7 findings. A rule finding comes from a fixed check in the harness. A referee finding comes from a second model that reads the record. The harness shows the findings to the scientist with the final answer. The record does not mark a finding as fixed. Thus a finding from an early review round can apply to a draft that the model corrected later.

Numbers in the answer

The last claim check read 133 numbers in the answer. 131 numbers match a logged result. 0 numbers have no source in the record.

Deviations

The model did not try to change a choice of the scientist.

Failed tool calls

No tool call failed.

Data integrity

Each data file has the same SHA-256 hash now as at the time of the step that read it. Where the download script (fetch.sh) gives a hash, the file also has that hash. The run did not change the data.

A SHA-256 hash is a fingerprint of the file contents. If one byte of the file changes, the hash changes. The table shows the first 12 characters.

How to repeat it

Get the data. The script downloads the files and checks their SHA-256 hashes where it lists them.

CUVETTE_DATA={data} bash bench/papers/lackey2015-tb-default/fetch.sh

Run the same case with Cuvette. The script gives the same answers from bench/papers/lackey2015-tb-default/bench.yaml.

cuvette bench papers --papers lackey2015-tb-default --models claude:claude-opus-5-5

Repeat each step by hand in the program. For each step, the harness records a manual route: the menu path or the code that gives the same result. This list does not include comparison runs.

1. inspect_table (step n1)

   Code

   print(pd.read_csv(path).describe(include='all'))

   • path

     {data}/lackey2015-tb-default/tb_default.csv

   • Note: The tool also counts distinct values and missing values for each column.

   The manual route that the harness recorded

   ga_biostats.inspect_table(path="{data}/lackey2015-tb-default/tb_default.csv")

   The manual route uses the same method. The note in the route gives the known difference.

2. chi_square_test (step n2)

   Code

   table = pandas.crosstab(df[row], df[column])
   chi2, p, dof, expected = scipy.stats.chi2_contingency(table, correction=False)   # correction=True is Yates, 2x2 only
   R: chisq.test(table(df$row, df$column), correct = FALSE); chisq.test(...)$expected

   • Make the cross-table of the two columns, or type the counts.
   • Run the Pearson chi-square test. For a 2x2 table, choose the Yates correction or no correction.
   • Read the expected counts. If a cell has an expected count below 5, use the Fisher exact test.
   • In SPSS Analyze>Descriptive Statistics>Crosstabs. Put one variable in Row(s) and one in Column(s). Click Statistics and select Chi-square. Click Cells and select Expected. SPSS shows the Pearson test, the continuity correction for a 2x2 table and a footnote with the number of cells below 5.
   • In Stata tabulate row column, chi2 expected (menu Statistics>Summaries, tables, and tests > Frequency tables > Two-way table with measures of association).
   • In GraphPad Prism enter a Contingency table, then Analyze>Contingency table analyses > Chi-square (and Fisher's exact) test. In JMP: Analyze>Fit Y by X with two nominal columns.
   • Continuity Correction (SPSS row) or Yates correction = false
   • Warning: If you keep the default both shown (SPSS); Yates on (R, 2x2), you get a different result.

   The manual route that the harness recorded

   ga_biostats.chi_square_test(path="{data}/lackey2015-tb-default/tb_default.csv", row="age_group", column="default", correction=False, alpha=0.05)

   The manual route gives the same numbers. An automatic test in Cuvette checks this.

3. chi_square_test (step n3)

   Code

   table = pandas.crosstab(df[row], df[column])
   chi2, p, dof, expected = scipy.stats.chi2_contingency(table, correction=False)   # correction=True is Yates, 2x2 only
   R: chisq.test(table(df$row, df$column), correct = FALSE); chisq.test(...)$expected

   • Make the cross-table of the two columns, or type the counts.
   • Run the Pearson chi-square test. For a 2x2 table, choose the Yates correction or no correction.
   • Read the expected counts. If a cell has an expected count below 5, use the Fisher exact test.
   • In SPSS Analyze>Descriptive Statistics>Crosstabs. Put one variable in Row(s) and one in Column(s). Click Statistics and select Chi-square. Click Cells and select Expected. SPSS shows the Pearson test, the continuity correction for a 2x2 table and a footnote with the number of cells below 5.
   • In Stata tabulate row column, chi2 expected (menu Statistics>Summaries, tables, and tests > Frequency tables > Two-way table with measures of association).
   • In GraphPad Prism enter a Contingency table, then Analyze>Contingency table analyses > Chi-square (and Fisher's exact) test. In JMP: Analyze>Fit Y by X with two nominal columns.
   • Continuity Correction (SPSS row) or Yates correction = false
   • Warning: If you keep the default both shown (SPSS); Yates on (R, 2x2), you get a different result.

   The manual route that the harness recorded

   ga_biostats.chi_square_test(path="{data}/lackey2015-tb-default/tb_default.csv", row="sex", column="default", correction=False, alpha=0.05)

   The manual route gives the same numbers. An automatic test in Cuvette checks this.

4. chi_square_test (step n4)

   Code

   table = pandas.crosstab(df[row], df[column])
   chi2, p, dof, expected = scipy.stats.chi2_contingency(table, correction=False)   # correction=True is Yates, 2x2 only
   R: chisq.test(table(df$row, df$column), correct = FALSE); chisq.test(...)$expected

   • Make the cross-table of the two columns, or type the counts.
   • Run the Pearson chi-square test. For a 2x2 table, choose the Yates correction or no correction.
   • Read the expected counts. If a cell has an expected count below 5, use the Fisher exact test.
   • In SPSS Analyze>Descriptive Statistics>Crosstabs. Put one variable in Row(s) and one in Column(s). Click Statistics and select Chi-square. Click Cells and select Expected. SPSS shows the Pearson test, the continuity correction for a 2x2 table and a footnote with the number of cells below 5.
   • In Stata tabulate row column, chi2 expected (menu Statistics>Summaries, tables, and tests > Frequency tables > Two-way table with measures of association).
   • In GraphPad Prism enter a Contingency table, then Analyze>Contingency table analyses > Chi-square (and Fisher's exact) test. In JMP: Analyze>Fit Y by X with two nominal columns.
   • Continuity Correction (SPSS row) or Yates correction = false
   • Warning: If you keep the default both shown (SPSS); Yates on (R, 2x2), you get a different result.

   The manual route that the harness recorded

   ga_biostats.chi_square_test(path="{data}/lackey2015-tb-default/tb_default.csv", row="marital_status", column="default", correction=False, alpha=0.05)

   The manual route gives the same numbers. An automatic test in Cuvette checks this.

5. chi_square_test (step n5)

   Code

   table = pandas.crosstab(df[row], df[column])
   chi2, p, dof, expected = scipy.stats.chi2_contingency(table, correction=False)   # correction=True is Yates, 2x2 only
   R: chisq.test(table(df$row, df$column), correct = FALSE); chisq.test(...)$expected

   • Make the cross-table of the two columns, or type the counts.
   • Run the Pearson chi-square test. For a 2x2 table, choose the Yates correction or no correction.
   • Read the expected counts. If a cell has an expected count below 5, use the Fisher exact test.
   • In SPSS Analyze>Descriptive Statistics>Crosstabs. Put one variable in Row(s) and one in Column(s). Click Statistics and select Chi-square. Click Cells and select Expected. SPSS shows the Pearson test, the continuity correction for a 2x2 table and a footnote with the number of cells below 5.
   • In Stata tabulate row column, chi2 expected (menu Statistics>Summaries, tables, and tests > Frequency tables > Two-way table with measures of association).
   • In GraphPad Prism enter a Contingency table, then Analyze>Contingency table analyses > Chi-square (and Fisher's exact) test. In JMP: Analyze>Fit Y by X with two nominal columns.
   • Continuity Correction (SPSS row) or Yates correction = false
   • Warning: If you keep the default both shown (SPSS); Yates on (R, 2x2), you get a different result.

   The manual route that the harness recorded

   ga_biostats.chi_square_test(path="{data}/lackey2015-tb-default/tb_default.csv", row="poverty", column="default", correction=False, alpha=0.05)

   The manual route gives the same numbers. An automatic test in Cuvette checks this.

6. fisher_exact (step n6)

   Code

   scipy.stats.fisher_exact(table, alternative="two-sided")                      # 2x2 p
   scipy.stats.contingency.odds_ratio(table, kind="conditional").confidence_interval()
   R: fisher.test(table)   # same p and conditional odds ratio; r x c tables use the network algorithm

   • Make the cross-table of the two columns, or type the counts.
   • Run the Fisher exact test. For a 2x2 table, read the conditional odds ratio and its exact interval.
   • For a larger table, the tool counts all tables with the same margins. If there are too many, it gives a Monte Carlo p with a fixed seed.
   • In SPSS Analyze>Descriptive Statistics>Crosstabs>Statistics>Chi-square. SPSS shows Fisher's Exact Test for a 2x2 table. For a larger table, click Exact and select Exact (needs the Exact Tests module).
   • In Stata: tabulate row column, exact. In SAS: PROC FREQ; TABLES row*column / FISHER.
   • In GraphPad Prism Contingency table > Analyze>Chi-square (and Fisher's exact) test, then Fisher's exact test (2x2 tables only).
   • alternative = two-sided
   • Note: The p-value of a 2x2 table and of an enumerated larger table equals R fisher.test. A large table gets a Monte Carlo p (R uses an exact network algorithm). The upper bound of the odds ratio interval can differ from R in the fourth digit. SPSS and Stata print the sample odds ratio, not the conditional odds ratio (sample_odds_ratio in the result).

   The manual route that the harness recorded

   ga_biostats.fisher_exact(path="{data}/lackey2015-tb-default/tb_default.csv", row="marital_status", column="default", alternative="two-sided", alpha=0.05, seed=1)

   The manual route uses the same method. The note in the route gives the known difference.

7. run_script (step n7)

   Run the Python code in {work}/script-1/script.py

   • Code only: this step has no route in the program menus. Run it with the script or flow export.

   The program has no menu route for this step. To repeat it, run the code.

8. fit_logistic (step n8)

   Code

   X = pandas.get_dummies(df[covariates], columns=categorical, drop_first=True)   # first sorted level is the reference
   res = statsmodels.api.Logit(df[outcome], statsmodels.api.add_constant(X)).fit()
   numpy.exp(res.params); numpy.exp(res.conf_int()); res.llr, res.llr_pvalue
   R: glm(outcome ~ age + factor(race), family = binomial); exp(confint.default(fit)); drop1(fit, test = "LRT")

   • Drop rows with a missing value in the outcome or a covariate. Apply the subset first if there is one.
   • Code each categorical covariate as indicator columns. Leave out the reference level.
   • Fit the logistic model. Take exp() of each coefficient and of each Wald interval bound for the odds ratios.
   • For the likelihood-ratio test of a covariate, fit the model without it and compare twice the difference of the log-likelihoods with a chi-square distribution.
   • In SPSS Analyze>Regression>Binary Logistic. Put the outcome in Dependent and the covariates in Covariates. Click Categorical, move each categorical covariate, set Reference Category to First and click Change. In Options, select CI for exp(B).
   • In Stata logistic outcome age lwt i.race ftv (menu Statistics>Binary outcomes > Logistic regression (reporting odds ratios)). Then lrtest for a covariate.
   • In SAS: PROC LOGISTIC; CLASS race (PARAM=REF REF=FIRST); MODEL outcome(EVENT='1') = age lwt race ftv / CLODDS=WALD.
   • In GraphPad Prism Analyze>Multiple variable analyses > Multiple logistic regression. In JMP: Analyze>Fit Model, Personality Nominal Logistic.
   • Dependent = default
   • Covariates = drug_use, bmi, mdr_tb, alcohol_weekly, hiv_status, secondary_education
   • Reference Category = drug_use=No, bmi=Normal, mdr_tb=No, alcohol_weekly=No, hiv_status=Negative, secondary_education=Yes
   • CI for exp(B) = 0.05
   • Warning: If you keep the default Last (SPSS), effect coding (SAS CLASS), you get a different result.
   • Warning: If you keep the default 95, you get a different result.

   The manual route that the harness recorded

   ga_biostats.fit_logistic(path="{data}/lackey2015-tb-default/tb_default.csv", outcome="default", covariates="drug_use, bmi, mdr_tb, alcohol_weekly, hiv_status, secondary_education", event="1", reference="drug_use=No, bmi=Normal, mdr_tb=No, alcohol_weekly=No, hiv_status=Negative, secondary_education=Yes", alpha=0.05)

   The manual route gives the same numbers. An automatic test in Cuvette checks this.

9. calculate (step n9)

   Run the tool "calculate" with these settings: {"items":[{"name":"pct_default_overall","expression":"100 * 127 / 1233"}]}.
   - Code only: this step has no route in the program menus. Run it with the script or flow export.

   The harness recorded no manual route for this step.

10. calculate (step n10)

    Run the tool "calculate" with these settings: {"items":[{"name":"confidence_level_pct","expression":"100 * (1 - 0.05)"}]}.
    - Code only: this step has no route in the program menus. Run it with the script or flow export.

    The harness recorded no manual route for this step.

Figure

Run facts

The session

This is the session as the scientist sees it, in order. Decision cards show the answer that the script gave and where that answer comes from. Each step shows the program, its version, the input file hash and the outputs. Click a line to see more.

- Setup values:
Design:
- What is the unit of replication? (replicate): subjects, patients or animals (one value per subject)
Inference:
- Significance level (alpha) (alpha): 0.05
Ask the scientist: Paired test (paired), Test family for two groups (two_group_test), Assume equal variances (unpaired t-test) (equal_variance), Sidedness (alternative), ANOVA variance assumption (anova_variance), Pairwise comparisons after ANOVA (posthoc), Correction for a list of p-values (p_adjust_method), Model formula (model_formula), Standard errors for the linear model (standard_errors), Random effects of the mixed model (random_effects), Fit by REML (reml), Covariates of the Cox model (cox_covariates), Ties method of the Cox model (ties), Strata of the Cox model (cox_strata), Type of sums of squares (ss_type), Comparisons of the cell means after a factorial ANOVA (factorial_posthoc), Sphericity correction for repeated measures (sphericity_correction), Correction after a rank test over many groups (rank_p_adjust), Exact or approximate p value for a rank test (exact_p), Covariates of the logistic model (logistic_covariates), Outcome value that counts as the event (logistic_event), Reference level of each categorical covariate (reference_levels), Yates continuity correction for a 2x2 chi-square test (chi_square_correction), Sidedness of the Fisher exact test (fisher_alternative), Method of the McNemar test (mcnemar_method).
The harness fills these values into your tool calls automatically. If you give a different value, the harness logs a deviation and shows it to the scientist.
- Mode is now watch: you run the steps.

{
 "ok": true,
 "summary": "1233 rows x 19 columns.",
 "metrics": {
  "n_rows": 1233,
  "n_columns": 19
 },
 "table": {
  "columns": [
   "column",
   "dtype",
   "n_missing",
   "n_unique",
   "min",
   "max",
   "first_values"
  ],
  "rows": [
   [
    "id",
    "int64",
    0,
    1233,
    883,
    2347,
    "1008, 1009, 1010, 1011, 1014"
   ],
   [
    "age_group",
    "object",
    0,
    4,
    null,
    null,
    "38 and older, 27 to 37, 22 to 26, 21 and younger"
   ],
   [
    "sex",
    "object",
    0,
    2,
    null,
    null,
    "Male, Female"
   ],
   [
    "marital_status",
    "object",
    0,
    4,
    null,
    null,
    "Single, Married/cohabitating, Divorced/separated, Widowed"
   ],
   [
    "poverty",
    "object",
    0,
    2,
    null,
    null,
    "Not in poverty, Poverty/extreme poverty"
   ],
   [
    "prison",
    "object",
    0,
    2,
    null,
    null,
    "No, Yes"
   ],
   [
    "secondary_education",
    "object",
    0,
    2,
    null,
    null,
    "Yes, No"
   ],
   [
    "smoking",
    "object",
    0,
    3,
    null,
    null,
    "Used to smoke, Never smoked, Currently smokes"
   ],
   [
    "alcohol_weekly",
    "object",
    0,
    2,
    null,
    null,
    "Yes, No"
   ],
   [
    "drug_use",
    "object",
    0,
    2,
    null,
    null,
    "No, Yes"
   ],
   [
    "rehab",
    "object",
    0,
    2,
    null,
    null,
    "No, Yes"
   ],
   [
    "mdr_tb",
    "object",
    0,
    2,
    null,
    null,
    "No, Yes"
   ],
   [
    "bmi",
    "object",
    0,
    3,
    null,
    null,
    "Underweight, Normal, Overweight/Obese"
   ],
   [
    "chronic_disease",
    "object",
    0,
    2,
    null,
    null,
    "No, Yes"
   ],
   [
    "hiv_status",
    "object",
    0,
    3,
    null,
    null,
    "Negative, Test not done, Positive"
   ]
  ],
  "n_rows": 19,
  "path": "{work}/inspect_table-1/columns.csv"
 }
}

column,dtype,n_missing,n_unique,min,max,first_values
id,int64,0,1233,883.0,2347.0,"1008, 1009, 1010, 1011, 1014"
age_group,object,0,4,,,"38 and older, 27 to 37, 22 to 26, 21 and younger"
sex,object,0,2,,,"Male, Female"
marital_status,object,0,4,,,"Single, Married/cohabitating, Divorced/separated, Widowed"
poverty,object,0,2,,,"Not in poverty, Poverty/extreme poverty"
prison,object,0,2,,,"No, Yes"
secondary_education,object,0,2,,,"Yes, No"
smoking,object,0,3,,,"Used to smoke, Never smoked, Currently smokes"
alcohol_weekly,object,0,2,,,"Yes, No"
drug_use,object,0,2,,,"No, Yes"
rehab,object,0,2,,,"No, Yes"
mdr_tb,object,0,2,,,"No, Yes"
bmi,object,0,3,,,"Underweight, Normal, Overweight/Obese"
chronic_disease,object,0,2,,,"No, Yes"
hiv_status,object,0,3,,,"Negative, Test not done, Positive"
diabetes,object,0,2,,,"No, Yes"
outcome,object,0,5,,,"Cured, Transferred out, Died, Default, Still in treatment"
default,int64,0,2,0.0,1.0,"0, 1"
days_to_default,float64,1114,104,2.0,973.0,"640.0, 353.0, 8.0, 107.0, 98.0"

{
 "ok": true,
 "summary": "Pearson chi-square test, 4 x 2 table, n = 1233: chi-square 12.43 on 3 df, p = 0.006039 (alpha 0.05). Cramer's V 0.1.",
 "metrics": {
  "chi2": 12.432624516640747,
  "df": 3,
  "p_value": 0.00603898829164167,
  "n": 1233,
  "cramers_v": 0.10041529546121677,
  "min_expected": 29.97323600973236,
  "n_cells_expected_below_5": 0,
  "fraction_cells_expected_below_5": 0
 },
 "data": {
  "test": "Pearson chi-square test",
  "correction": false,
  "alpha": 0.05,
  "significant": true,
  "warning": "",
  "expected": [
   [
    287.9367396593674,
    33.0632603406326
   ],
   [
    292.42173560421736,
    33.57826439578265
   ],
   [
    261.0267639902676,
    29.97323600973236
   ],
   [
    264.6147607461476,
    30.385239253852394
   ]
  ],
  "rows": [
   "21 and younger",
   "22 to 26",
   "27 to 37",
   "38 and older"
  ],
  "columns": [
   "0",
   "1"
  ]
 },
 "table": {
  "columns": [
   "age_group",
   "0",
   "1"
  ],
  "rows": [
   [
    "21 and younger",
    284,
    37
   ],
   [
    "22 to 26",
    290,
    36
   ],
   [
    "27 to 37",
    252,
    39
   ],
   [
    "38 and older",
    280,
    15
   ]
  ],
  "n_rows": 4,
  "path": "{work}/chi_square_test-1/observed_counts.csv"
 },
 "outputs": [
  {
   "path": "{work}/chi_square_test-1/expected_counts.csv",
   "kind": "table",
   "name": "expected_counts.csv"
  }
 ]
}

{
 "ok": true,
 "summary": "Pearson chi-square test, 2 x 2 table, n = 1233: chi-square 16.9 on 1 df, p = 3.945e-05 (alpha 0.05). Cramer's V 0.117.",
 "metrics": {
  "chi2": 16.897624469314554,
  "df": 1,
  "p_value": 0.00003945092420796785,
  "n": 1233,
  "cramers_v": 0.11706613733438456,
  "min_expected": 50.470397404703974,
  "n_cells_expected_below_5": 0,
  "fraction_cells_expected_below_5": 0
 },
 "data": {
  "test": "Pearson chi-square test",
  "correction": false,
  "alpha": 0.05,
  "significant": true,
  "warning": "",
  "expected": [
   [
    439.529602595296,
    50.470397404703974
   ],
   [
    666.4703974047039,
    76.52960259529603
   ]
  ],
  "rows": [
   "Female",
   "Male"
  ],
  "columns": [
   "0",
   "1"
  ]
 },
 "table": {
  "columns": [
   "sex",
   "0",
   "1"
  ],
  "rows": [
   [
    "Female",
    461,
    29
   ],
   [
    "Male",
    645,
    98
   ]
  ],
  "n_rows": 2,
  "path": "{work}/chi_square_test-2/observed_counts.csv"
 },
 "outputs": [
  {
   "path": "{work}/chi_square_test-2/expected_counts.csv",
   "kind": "table",
   "name": "expected_counts.csv"
  }
 ]
}

{
 "ok": true,
 "summary": "Pearson chi-square test, 4 x 2 table, n = 1233: chi-square 1.504 on 3 df, p = 0.6815 (alpha 0.05). Cramer's V 0.0349. Warning: 1 of 8 cells (12%) have an expected count below 5 (smallest 3.71). The chi-square p-value can be wrong. Use fisher_exact.",
 "metrics": {
  "chi2": 1.5035356413644951,
  "df": 3,
  "p_value": 0.6814545468936668,
  "n": 1233,
  "cramers_v": 0.03492008767758139,
  "min_expected": 3.708029197080292,
  "n_cells_expected_below_5": 1,
  "fraction_cells_expected_below_5": 0.125
 },
 "data": {
  "test": "Pearson chi-square test",
  "correction": false,
  "alpha": 0.05,
  "significant": false,
  "warning": "Warning: 1 of 8 cells (12%) have an expected count below 5 (smallest 3.71). The chi-square p-value can be wrong. Use fisher_exact.",
  "expected": [
   [
    81.62692619626927,
    9.373073803730739
   ],
   [
    420.6926196269262,
    48.307380373073805
   ],
   [
    571.3884833738848,
    65.61151662611516
   ],
   [
    32.29197080291971,
    3.708029197080292
   ]
  ],
  "rows": [
   "Divorced/separated",
   "Married/cohabitating",
   "Single",
   "Widowed"
  ],
  "columns": [
   "0",
   "1"
  ]
 },
 "table": {
  "columns": [
   "marital_status",
   "0",
   "1"
  ],
  "rows": [
   [
    "Divorced/separated",
    81,
    10
   ],
   [
    "Married/cohabitating",
    424,
    45
   ],
   [
    "Single",
    567,
    70
   ],
   [
    "Widowed",
    34,
    2
   ]
  ],
  "n_rows": 4,
  "path": "{work}/chi_square_test-3/observed_counts.csv"
 },
 "outputs": [
  {
   "path": "{work}/chi_square_test-3/expected_counts.csv",
   "kind": "table",
   "name": "expected_counts.csv"
  }
 ]
}

{
 "ok": true,
 "summary": "Pearson chi-square test, 2 x 2 table, n = 1233: chi-square 4.477 on 1 df, p = 0.03436 (alpha 0.05). Cramer's V 0.0603.",
 "metrics": {
  "chi2": 4.47652666051148,
  "df": 1,
  "p_value": 0.03436348964265445,
  "n": 1233,
  "cramers_v": 0.060254439286915616,
  "min_expected": 21.527169505271694,
  "n_cells_expected_below_5": 0,
  "fraction_cells_expected_below_5": 0
 },
 "data": {
  "test": "Pearson chi-square test",
  "correction": false,
  "alpha": 0.05,
  "significant": true,
  "warning": "",
  "expected": [
   [
    918.5271695052717,
    105.4728304947283
   ],
   [
    187.4728304947283,
    21.527169505271694
   ]
  ],
  "rows": [
   "Not in poverty",
   "Poverty/extreme poverty"
  ],
  "columns": [
   "0",
   "1"
  ]
 },
 "table": {
  "columns": [
   "poverty",
   "0",
   "1"
  ],
  "rows": [
   [
    "Not in poverty",
    927,
    97
   ],
   [
    "Poverty/extreme poverty",
    179,
    30
   ]
  ],
  "n_rows": 2,
  "path": "{work}/chi_square_test-4/observed_counts.csv"
 },
 "outputs": [
  {
   "path": "{work}/chi_square_test-4/expected_counts.csv",
   "kind": "table",
   "name": "expected_counts.csv"
  }
 ]
}

{"ok":true,"summary":"Logistic regression of default = 1 against 0, n = 1233, 127 events. drug_use[Yes] OR 4.782 (95% CI 3.052 to 7.491, p = 8.38e-12); bmi[Overweight/Obese] OR 0.8777 (95% CI 0.4444 to 1.733, p = 0.707); bmi[Underweight] OR 2.08 (95% CI 1.214 to 3.563, p = 0.0077); mdr_tb[Yes] OR 3.038 (95% CI 1.578 to 5.851, p = 0.000886); alcohol_weekly[Yes] OR 2.222 (95% CI 1.401 to 3.525, p = 0.000695); hiv_status[Positive] OR 1.392 (95% CI 0.4167 to 4.654, p = 0.591); hiv_status[Test not done] OR 2.303 (95% CI 1.5 to 3.536, p = 0.000137); secondary_education[No] OR 1.55 (95% CI 1.032 to 2.329, p = 0.0349). Likelihood-ratio test of the model: chi-square 139.5 on 8 df, p = 3.076e-26. Wald confidence intervals. Likelihood-ratio test of each categorical covariate: bmi chi-square 7.273 on 2 df, p = 0.02635; hiv_status chi-square 14.06 on 2 df, p = 0.0008845. Reference levels: drug_use = No, bmi = Normal, mdr_tb = No, alcohol_weekly = No, hiv_status = Negative, secondary_education = Yes.","metrics":{"n":1233,"events":127,"non_events":1106,"n_dropped":0,"log_likelihood":-339.16700926664714,"null_log_likelihood":-408.8958940401771,"lr_chi2":139.4577695470599,"lr_df":8,"lr_p":3.076324272310018e-26,"aic":696.3340185332943,"pseudo_r2_mcfadden":0.1705296770886101,"events_per_parameter":15.875,"converged":1,"coef_const":-3.497632150067933,"se_const":0.2124385984966824,"p_const":6.633082638853354e-61,"or_const":0.030268971015494868,"or_lo_const":0.01996041629112351,"or_hi_const":0.045901377655350385,"coef_drug_use_Yes":1.5647650527214947,"se_drug_use_Yes":0.2290396468725688,"p_drug_use_Yes":8.382509658407017e-12,"or_drug_use_Yes":4.78155139142713,"or_lo_drug_use_Yes":3.0521784873914464,"or_hi_drug_use_Yes":7.490791840420467,"coef_bmi_Overweight_Obese":-0.13043635986406035,"se_bmi_Overweight_Obese":0.3472058878406436,"p_bmi_Overweight_Obese":0.7071589799396698,"or_bmi_Overweight_Obese":0.8777123489045796,"or_lo_bmi_Overweight_Obese":0.4444368093830721,"or_hi_bmi_Overweight_Obese":1.7333824542772827,"coef_bmi_Underweight":0.7321286903849328,"se_bmi_Underweight":0.2747274170249356,"p_bmi_Underweight":0.007700488734103748,"or_bmi_Underweight":2.079502516579366,"or_lo_bmi_Underweight":1.2136991532262786,"or_hi_bmi_Underweight":3.5629346077773034,"coef_mdr_tb_Yes":1.1113587461354562,"se_mdr_tb_Yes":0.33430051843768804,"p_mdr_tb_Yes":0.0008859940232706892,"or_mdr_tb_Yes":3.0384841194494236,"or_lo_mdr_tb_Yes":1.577974038264568,"or_hi_mdr_tb_Yes":5.8507843096708845,"coef_alcohol_weekly_Yes":0.7984029127919758,"se_alcohol_weekly_Yes":0.23539707354057635,"p_alcohol_weekly_Yes":0.0006945327829807024,"or_alcohol_weekly_Yes":2.2219893823639905,"or_lo_alcohol_weekly_Yes":1.400785466210615,"or_hi_alcohol_weekly_Yes":3.5246202465924,"coef_hiv_status_Positive":0.3310904815764532,"se_hiv_status_Positive":0.6155952108100741,"p_hiv_status_Positive":0.5906889122875556,"or_hiv_status_Positive":1.3924857809169304,"or_lo_hiv_status_Positive":0.41667282366518404,"or_hi_hiv_status_
... (1000 more characters in the session record)

{
 "ok": true,
 "summary": "Fisher exact test, 4 x 2 table, n = 1233, two-sided: p = 0.742 (alpha 0.05). Method: exact (all tables with the same margins).",
 "metrics": {
  "n": 1233,
  "p_value": 0.741954958020354
 },
 "data": {
  "alternative": "two-sided",
  "alpha": 0.05,
  "rows": [
   "Divorced/separated",
   "Married/cohabitating",
   "Single",
   "Widowed"
  ],
  "columns": [
   "0",
   "1"
  ],
  "method": "exact (all tables with the same margins)",
  "significant": false
 },
 "table": {
  "columns": [
   "marital_status",
   "0",
   "1"
  ],
  "rows": [
   [
    "Divorced/separated",
    81,
    10
   ],
   [
    "Married/cohabitating",
    424,
    45
   ],
   [
    "Single",
    567,
    70
   ],
   [
    "Widowed",
    34,
    2
   ]
  ],
  "n_rows": 4,
  "path": "{work}/fisher_exact-1/fisher_counts.csv"
 }
}

{
 "ok": true,
 "summary": "holm correction of 4 p-values at alpha 0.05: 2 rejected.",
 "metrics": {
  "n_tests": 4,
  "n_rejected": 2
 },
 "table": {
  "columns": [
   "p_raw",
   "p_adjusted",
   "reject"
  ],
  "rows": [
   [
    0.00603898829164167,
    0.01811696487492501,
    true
   ],
   [
    0.00003945092420796785,
    0.0001578036968318714,
    true
   ],
   [
    0.6814545468936668,
    0.6814545468936668,
    false
   ],
   [
    0.03436348964265445,
    0.0687269792853089,
    false
   ]
  ],
  "n_rows": 4,
  "path": "{work}/adjust_pvalues-1/adjusted_p.csv"
 }
}

p_raw,p_adjusted,reject
0.00603898829164167,0.01811696487492501,True
3.945092420796785e-05,0.0001578036968318714,True
0.6814545468936668,0.6814545468936668,False
0.03436348964265445,0.0687269792853089,False

- The answer says default is 12.4% (127 of 1233). The logged counts give 127/1233 = 10.3%. The 12.4 matches no result in the log and looks like the age-group chi-square statistic 12.43. The value is still above 10%, but the figure is wrong.
Fix these problems. Then give the answer again.

{"items":[{"name":"default_pct","expression":"127/1233*100"}]}

default_pct = 127/1233*100 = 10.3000811

The final answer

The model wrote this answer. The harness shows it to the scientist after the review.

Settings used, from the decision record: Significance level (alpha): 0.05 · Correction for a list of p-values: holm · Covariates of the logistic model: drug_use, bmi, mdr_tb, alcohol_weekly, hiv_status, secondary_education · Outcome value that counts as the event: 1 · Reference level of each categorical covariate: drug_use=No, bmi=Normal, mdr_tb=No, alcohol_weekly=No, hiv_status=Negative, secondary_education=Yes · Yates continuity correction for a 2x2 chi-square test: false · Sidedness of the Fisher exact test: two-sided.

Values that are not scored

• A trap value is the result of a wrong method, for example an unpaired test on paired data. A trap value in a main step means that the run used the wrong method. A trap value in a comparison run is correct, because the record keeps the scientist's choice.
• An optional value is a second result of the same method. The run can compute it or not.
• A reference value comes from a different program version or a check run by us. We show it for comparison.

Checks

Review findings

The review recorded 6 findings. A rule finding comes from a fixed check in the harness. A referee finding comes from a second model that reads the record. The harness shows the findings to the scientist with the final answer. The record does not mark a finding as fixed. Thus a finding from an early review round can apply to a draft that the model corrected later.

Numbers in the answer

The last claim check read 102 numbers in the answer. 100 numbers match a logged result. 0 numbers have no source in the record.

Deviations

The model did not try to change a choice of the scientist.

Failed tool calls

No tool call failed.

Data integrity

Each data file has the same SHA-256 hash now as at the time of the step that read it. Where the download script (fetch.sh) gives a hash, the file also has that hash. The run did not change the data.

A SHA-256 hash is a fingerprint of the file contents. If one byte of the file changes, the hash changes. The table shows the first 12 characters.

How to repeat it

Get the data. The script downloads the files and checks their SHA-256 hashes where it lists them.

CUVETTE_DATA={data} bash bench/papers/lackey2015-tb-default/fetch.sh

Run the same case with Cuvette. The script gives the same answers from bench/papers/lackey2015-tb-default/bench.yaml.

cuvette bench papers --papers lackey2015-tb-default --models claude:claude-sonnet-5-5

Repeat each step by hand in the program. For each step, the harness records a manual route: the menu path or the code that gives the same result. This list does not include comparison runs.

1. inspect_table (step n1)

   Code

   print(pd.read_csv(path).describe(include='all'))

   • path

     {data}/lackey2015-tb-default/tb_default.csv

   • Note: The tool also counts distinct values and missing values for each column.

   The manual route that the harness recorded

   ga_biostats.inspect_table(path="{data}/lackey2015-tb-default/tb_default.csv")

   The manual route uses the same method. The note in the route gives the known difference.

2. chi_square_test (step n2)

   Code

   table = pandas.crosstab(df[row], df[column])
   chi2, p, dof, expected = scipy.stats.chi2_contingency(table, correction=False)   # correction=True is Yates, 2x2 only
   R: chisq.test(table(df$row, df$column), correct = FALSE); chisq.test(...)$expected

   • Make the cross-table of the two columns, or type the counts.
   • Run the Pearson chi-square test. For a 2x2 table, choose the Yates correction or no correction.
   • Read the expected counts. If a cell has an expected count below 5, use the Fisher exact test.
   • In SPSS Analyze>Descriptive Statistics>Crosstabs. Put one variable in Row(s) and one in Column(s). Click Statistics and select Chi-square. Click Cells and select Expected. SPSS shows the Pearson test, the continuity correction for a 2x2 table and a footnote with the number of cells below 5.
   • In Stata tabulate row column, chi2 expected (menu Statistics>Summaries, tables, and tests > Frequency tables > Two-way table with measures of association).
   • In GraphPad Prism enter a Contingency table, then Analyze>Contingency table analyses > Chi-square (and Fisher's exact) test. In JMP: Analyze>Fit Y by X with two nominal columns.
   • Continuity Correction (SPSS row) or Yates correction = false
   • Warning: If you keep the default both shown (SPSS); Yates on (R, 2x2), you get a different result.

   The manual route that the harness recorded

   ga_biostats.chi_square_test(path="{data}/lackey2015-tb-default/tb_default.csv", row="age_group", column="default", correction=False, alpha=0.05)

   The manual route gives the same numbers. An automatic test in Cuvette checks this.

3. chi_square_test (step n3)

   Code

   table = pandas.crosstab(df[row], df[column])
   chi2, p, dof, expected = scipy.stats.chi2_contingency(table, correction=False)   # correction=True is Yates, 2x2 only
   R: chisq.test(table(df$row, df$column), correct = FALSE); chisq.test(...)$expected

   • Make the cross-table of the two columns, or type the counts.
   • Run the Pearson chi-square test. For a 2x2 table, choose the Yates correction or no correction.
   • Read the expected counts. If a cell has an expected count below 5, use the Fisher exact test.
   • In SPSS Analyze>Descriptive Statistics>Crosstabs. Put one variable in Row(s) and one in Column(s). Click Statistics and select Chi-square. Click Cells and select Expected. SPSS shows the Pearson test, the continuity correction for a 2x2 table and a footnote with the number of cells below 5.
   • In Stata tabulate row column, chi2 expected (menu Statistics>Summaries, tables, and tests > Frequency tables > Two-way table with measures of association).
   • In GraphPad Prism enter a Contingency table, then Analyze>Contingency table analyses > Chi-square (and Fisher's exact) test. In JMP: Analyze>Fit Y by X with two nominal columns.
   • Continuity Correction (SPSS row) or Yates correction = false
   • Warning: If you keep the default both shown (SPSS); Yates on (R, 2x2), you get a different result.

   The manual route that the harness recorded

   ga_biostats.chi_square_test(path="{data}/lackey2015-tb-default/tb_default.csv", row="sex", column="default", correction=False, alpha=0.05)

   The manual route gives the same numbers. An automatic test in Cuvette checks this.

4. chi_square_test (step n4)

   Code

   table = pandas.crosstab(df[row], df[column])
   chi2, p, dof, expected = scipy.stats.chi2_contingency(table, correction=False)   # correction=True is Yates, 2x2 only
   R: chisq.test(table(df$row, df$column), correct = FALSE); chisq.test(...)$expected

   • Make the cross-table of the two columns, or type the counts.
   • Run the Pearson chi-square test. For a 2x2 table, choose the Yates correction or no correction.
   • Read the expected counts. If a cell has an expected count below 5, use the Fisher exact test.
   • In SPSS Analyze>Descriptive Statistics>Crosstabs. Put one variable in Row(s) and one in Column(s). Click Statistics and select Chi-square. Click Cells and select Expected. SPSS shows the Pearson test, the continuity correction for a 2x2 table and a footnote with the number of cells below 5.
   • In Stata tabulate row column, chi2 expected (menu Statistics>Summaries, tables, and tests > Frequency tables > Two-way table with measures of association).
   • In GraphPad Prism enter a Contingency table, then Analyze>Contingency table analyses > Chi-square (and Fisher's exact) test. In JMP: Analyze>Fit Y by X with two nominal columns.
   • Continuity Correction (SPSS row) or Yates correction = false
   • Warning: If you keep the default both shown (SPSS); Yates on (R, 2x2), you get a different result.

   The manual route that the harness recorded

   ga_biostats.chi_square_test(path="{data}/lackey2015-tb-default/tb_default.csv", row="marital_status", column="default", correction=False, alpha=0.05)

   The manual route gives the same numbers. An automatic test in Cuvette checks this.

5. chi_square_test (step n5)

   Code

   table = pandas.crosstab(df[row], df[column])
   chi2, p, dof, expected = scipy.stats.chi2_contingency(table, correction=False)   # correction=True is Yates, 2x2 only
   R: chisq.test(table(df$row, df$column), correct = FALSE); chisq.test(...)$expected

   • Make the cross-table of the two columns, or type the counts.
   • Run the Pearson chi-square test. For a 2x2 table, choose the Yates correction or no correction.
   • Read the expected counts. If a cell has an expected count below 5, use the Fisher exact test.
   • In SPSS Analyze>Descriptive Statistics>Crosstabs. Put one variable in Row(s) and one in Column(s). Click Statistics and select Chi-square. Click Cells and select Expected. SPSS shows the Pearson test, the continuity correction for a 2x2 table and a footnote with the number of cells below 5.
   • In Stata tabulate row column, chi2 expected (menu Statistics>Summaries, tables, and tests > Frequency tables > Two-way table with measures of association).
   • In GraphPad Prism enter a Contingency table, then Analyze>Contingency table analyses > Chi-square (and Fisher's exact) test. In JMP: Analyze>Fit Y by X with two nominal columns.
   • Continuity Correction (SPSS row) or Yates correction = false
   • Warning: If you keep the default both shown (SPSS); Yates on (R, 2x2), you get a different result.

   The manual route that the harness recorded

   ga_biostats.chi_square_test(path="{data}/lackey2015-tb-default/tb_default.csv", row="poverty", column="default", correction=False, alpha=0.05)

   The manual route gives the same numbers. An automatic test in Cuvette checks this.

6. fit_logistic (step n6)

   Code

   X = pandas.get_dummies(df[covariates], columns=categorical, drop_first=True)   # first sorted level is the reference
   res = statsmodels.api.Logit(df[outcome], statsmodels.api.add_constant(X)).fit()
   numpy.exp(res.params); numpy.exp(res.conf_int()); res.llr, res.llr_pvalue
   R: glm(outcome ~ age + factor(race), family = binomial); exp(confint.default(fit)); drop1(fit, test = "LRT")

   • Drop rows with a missing value in the outcome or a covariate. Apply the subset first if there is one.
   • Code each categorical covariate as indicator columns. Leave out the reference level.
   • Fit the logistic model. Take exp() of each coefficient and of each Wald interval bound for the odds ratios.
   • For the likelihood-ratio test of a covariate, fit the model without it and compare twice the difference of the log-likelihoods with a chi-square distribution.
   • In SPSS Analyze>Regression>Binary Logistic. Put the outcome in Dependent and the covariates in Covariates. Click Categorical, move each categorical covariate, set Reference Category to First and click Change. In Options, select CI for exp(B).
   • In Stata logistic outcome age lwt i.race ftv (menu Statistics>Binary outcomes > Logistic regression (reporting odds ratios)). Then lrtest for a covariate.
   • In SAS: PROC LOGISTIC; CLASS race (PARAM=REF REF=FIRST); MODEL outcome(EVENT='1') = age lwt race ftv / CLODDS=WALD.
   • In GraphPad Prism Analyze>Multiple variable analyses > Multiple logistic regression. In JMP: Analyze>Fit Model, Personality Nominal Logistic.
   • Dependent = default
   • Covariates = drug_use, bmi, mdr_tb, alcohol_weekly, hiv_status, secondary_education
   • Reference Category = drug_use=No, bmi=Normal, mdr_tb=No, alcohol_weekly=No, hiv_status=Negative, secondary_education=Yes
   • CI for exp(B) = 0.05
   • Warning: If you keep the default Last (SPSS), effect coding (SAS CLASS), you get a different result.
   • Warning: If you keep the default 95, you get a different result.

   The manual route that the harness recorded

   ga_biostats.fit_logistic(path="{data}/lackey2015-tb-default/tb_default.csv", outcome="default", covariates="drug_use, bmi, mdr_tb, alcohol_weekly, hiv_status, secondary_education", event="1", reference="drug_use=No, bmi=Normal, mdr_tb=No, alcohol_weekly=No, hiv_status=Negative, secondary_education=Yes", alpha=0.05)

   The manual route gives the same numbers. An automatic test in Cuvette checks this.

7. fisher_exact (step n7)

   Code

   scipy.stats.fisher_exact(table, alternative="two-sided")                      # 2x2 p
   scipy.stats.contingency.odds_ratio(table, kind="conditional").confidence_interval()
   R: fisher.test(table)   # same p and conditional odds ratio; r x c tables use the network algorithm

   • Make the cross-table of the two columns, or type the counts.
   • Run the Fisher exact test. For a 2x2 table, read the conditional odds ratio and its exact interval.
   • For a larger table, the tool counts all tables with the same margins. If there are too many, it gives a Monte Carlo p with a fixed seed.
   • In SPSS Analyze>Descriptive Statistics>Crosstabs>Statistics>Chi-square. SPSS shows Fisher's Exact Test for a 2x2 table. For a larger table, click Exact and select Exact (needs the Exact Tests module).
   • In Stata: tabulate row column, exact. In SAS: PROC FREQ; TABLES row*column / FISHER.
   • In GraphPad Prism Contingency table > Analyze>Chi-square (and Fisher's exact) test, then Fisher's exact test (2x2 tables only).
   • alternative = two-sided
   • Note: The p-value of a 2x2 table and of an enumerated larger table equals R fisher.test. A large table gets a Monte Carlo p (R uses an exact network algorithm). The upper bound of the odds ratio interval can differ from R in the fourth digit. SPSS and Stata print the sample odds ratio, not the conditional odds ratio (sample_odds_ratio in the result).

   The manual route that the harness recorded

   ga_biostats.fisher_exact(path="{data}/lackey2015-tb-default/tb_default.csv", row="marital_status", column="default", alternative="two-sided", alpha=0.05, seed=1)

   The manual route uses the same method. The note in the route gives the known difference.

8. adjust_pvalues (step n8)

   Code

   statsmodels.stats.multitest.multipletests(pvals, alpha=0.05, method='holm')

   • method = holm
   • Warning: If you keep the default hs, you get a different result.

   The manual route that the harness recorded

   ga_biostats.adjust_pvalues(pvalues=[0.00603898829164167, 0.00003945092420796785, 0.6814545468936668, 0.03436348964265445], method="holm", alpha=0.05)

   The manual route gives the same numbers. An automatic test in Cuvette checks this.

9. calculate (step n9)

   Run the tool "calculate" with these settings: {"items":[{"name":"default_pct","expression":"127/1233*100"}]}.
   - Code only: this step has no route in the program menus. Run it with the script or flow export.

   The harness recorded no manual route for this step.

Figure

Run facts

The session

This is the session as the scientist sees it, in order. Decision cards show the answer that the script gave and where that answer comes from. Each step shows the program, its version, the input file hash and the outputs. Click a line to see more.

- Setup values:
Design:
- What is the unit of replication? (replicate): subjects, patients or animals (one value per subject)
Inference:
- Significance level (alpha) (alpha): 0.05
Ask the scientist: Paired test (paired), Test family for two groups (two_group_test), Assume equal variances (unpaired t-test) (equal_variance), Sidedness (alternative), ANOVA variance assumption (anova_variance), Pairwise comparisons after ANOVA (posthoc), Correction for a list of p-values (p_adjust_method), Model formula (model_formula), Standard errors for the linear model (standard_errors), Random effects of the mixed model (random_effects), Fit by REML (reml), Covariates of the Cox model (cox_covariates), Ties method of the Cox model (ties), Strata of the Cox model (cox_strata), Type of sums of squares (ss_type), Comparisons of the cell means after a factorial ANOVA (factorial_posthoc), Sphericity correction for repeated measures (sphericity_correction), Correction after a rank test over many groups (rank_p_adjust), Exact or approximate p value for a rank test (exact_p), Covariates of the logistic model (logistic_covariates), Outcome value that counts as the event (logistic_event), Reference level of each categorical covariate (reference_levels), Yates continuity correction for a 2x2 chi-square test (chi_square_correction), Sidedness of the Fisher exact test (fisher_alternative), Method of the McNemar test (mcnemar_method).
The harness fills these values into your tool calls automatically. If you give a different value, the harness logs a deviation and shows it to the scientist.
- Mode is now watch: you run the steps.

{
 "ok": true,
 "summary": "1233 rows x 19 columns.",
 "metrics": {
  "n_rows": 1233,
  "n_columns": 19
 },
 "table": {
  "columns": [
   "column",
   "dtype",
   "n_missing",
   "n_unique",
   "min",
   "max",
   "first_values"
  ],
  "rows": [
   [
    "id",
    "int64",
    0,
    1233,
    883,
    2347,
    "1008, 1009, 1010, 1011, 1014"
   ],
   [
    "age_group",
    "object",
    0,
    4,
    null,
    null,
    "38 and older, 27 to 37, 22 to 26, 21 and younger"
   ],
   [
    "sex",
    "object",
    0,
    2,
    null,
    null,
    "Male, Female"
   ],
   [
    "marital_status",
    "object",
    0,
    4,
    null,
    null,
    "Single, Married/cohabitating, Divorced/separated, Widowed"
   ],
   [
    "poverty",
    "object",
    0,
    2,
    null,
    null,
    "Not in poverty, Poverty/extreme poverty"
   ],
   [
    "prison",
    "object",
    0,
    2,
    null,
    null,
    "No, Yes"
   ],
   [
    "secondary_education",
    "object",
    0,
    2,
    null,
    null,
    "Yes, No"
   ],
   [
    "smoking",
    "object",
    0,
    3,
    null,
    null,
    "Used to smoke, Never smoked, Currently smokes"
   ],
   [
    "alcohol_weekly",
    "object",
    0,
    2,
    null,
    null,
    "Yes, No"
   ],
   [
    "drug_use",
    "object",
    0,
    2,
    null,
    null,
    "No, Yes"
   ],
   [
    "rehab",
    "object",
    0,
    2,
    null,
    null,
    "No, Yes"
   ],
   [
    "mdr_tb",
    "object",
    0,
    2,
    null,
    null,
    "No, Yes"
   ],
   [
    "bmi",
    "object",
    0,
    3,
    null,
    null,
    "Underweight, Normal, Overweight/Obese"
   ],
   [
    "chronic_disease",
    "object",
    0,
    2,
    null,
    null,
    "No, Yes"
   ],
   [
    "hiv_status",
    "object",
    0,
    3,
    null,
    null,
    "Negative, Test not done, Positive"
   ]
  ],
  "n_rows": 19,
  "path": "{work}/inspect_table-1/columns.csv"
 }
}

column,dtype,n_missing,n_unique,min,max,first_values
id,int64,0,1233,883.0,2347.0,"1008, 1009, 1010, 1011, 1014"
age_group,object,0,4,,,"38 and older, 27 to 37, 22 to 26, 21 and younger"
sex,object,0,2,,,"Male, Female"
marital_status,object,0,4,,,"Single, Married/cohabitating, Divorced/separated, Widowed"
poverty,object,0,2,,,"Not in poverty, Poverty/extreme poverty"
prison,object,0,2,,,"No, Yes"
secondary_education,object,0,2,,,"Yes, No"
smoking,object,0,3,,,"Used to smoke, Never smoked, Currently smokes"
alcohol_weekly,object,0,2,,,"Yes, No"
drug_use,object,0,2,,,"No, Yes"
rehab,object,0,2,,,"No, Yes"
mdr_tb,object,0,2,,,"No, Yes"
bmi,object,0,3,,,"Underweight, Normal, Overweight/Obese"
chronic_disease,object,0,2,,,"No, Yes"
hiv_status,object,0,3,,,"Negative, Test not done, Positive"
diabetes,object,0,2,,,"No, Yes"
outcome,object,0,5,,,"Cured, Transferred out, Died, Default, Still in treatment"
default,int64,0,2,0.0,1.0,"0, 1"
days_to_default,float64,1114,104,2.0,973.0,"640.0, 353.0, 8.0, 107.0, 98.0"

{
 "ok": true,
 "summary": "Pearson chi-square test, 4 x 2 table, n = 1233: chi-square 12.43 on 3 df, p = 0.006039 (alpha 0.05). Cramer's V 0.1.",
 "metrics": {
  "chi2": 12.432624516640747,
  "df": 3,
  "p_value": 0.00603898829164167,
  "n": 1233,
  "cramers_v": 0.10041529546121677,
  "min_expected": 29.97323600973236,
  "n_cells_expected_below_5": 0,
  "fraction_cells_expected_below_5": 0
 },
 "data": {
  "test": "Pearson chi-square test",
  "correction": false,
  "alpha": 0.05,
  "significant": true,
  "warning": "",
  "expected": [
   [
    287.9367396593674,
    33.0632603406326
   ],
   [
    292.42173560421736,
    33.57826439578265
   ],
   [
    261.0267639902676,
    29.97323600973236
   ],
   [
    264.6147607461476,
    30.385239253852394
   ]
  ],
  "rows": [
   "21 and younger",
   "22 to 26",
   "27 to 37",
   "38 and older"
  ],
  "columns": [
   "0",
   "1"
  ]
 },
 "table": {
  "columns": [
   "age_group",
   "0",
   "1"
  ],
  "rows": [
   [
    "21 and younger",
    284,
    37
   ],
   [
    "22 to 26",
    290,
    36
   ],
   [
    "27 to 37",
    252,
    39
   ],
   [
    "38 and older",
    280,
    15
   ]
  ],
  "n_rows": 4,
  "path": "{work}/chi_square_test-1/observed_counts.csv"
 },
 "outputs": [
  {
   "path": "{work}/chi_square_test-1/expected_counts.csv",
   "kind": "table",
   "name": "expected_counts.csv"
  }
 ]
}

{
 "ok": true,
 "summary": "Pearson chi-square test, 2 x 2 table, n = 1233: chi-square 16.9 on 1 df, p = 3.945e-05 (alpha 0.05). Cramer's V 0.117.",
 "metrics": {
  "chi2": 16.897624469314554,
  "df": 1,
  "p_value": 0.00003945092420796785,
  "n": 1233,
  "cramers_v": 0.11706613733438456,
  "min_expected": 50.470397404703974,
  "n_cells_expected_below_5": 0,
  "fraction_cells_expected_below_5": 0
 },
 "data": {
  "test": "Pearson chi-square test",
  "correction": false,
  "alpha": 0.05,
  "significant": true,
  "warning": "",
  "expected": [
   [
    439.529602595296,
    50.470397404703974
   ],
   [
    666.4703974047039,
    76.52960259529603
   ]
  ],
  "rows": [
   "Female",
   "Male"
  ],
  "columns": [
   "0",
   "1"
  ]
 },
 "table": {
  "columns": [
   "sex",
   "0",
   "1"
  ],
  "rows": [
   [
    "Female",
    461,
    29
   ],
   [
    "Male",
    645,
    98
   ]
  ],
  "n_rows": 2,
  "path": "{work}/chi_square_test-2/observed_counts.csv"
 },
 "outputs": [
  {
   "path": "{work}/chi_square_test-2/expected_counts.csv",
   "kind": "table",
   "name": "expected_counts.csv"
  }
 ]
}

{
 "ok": true,
 "summary": "Pearson chi-square test, 4 x 2 table, n = 1233: chi-square 1.504 on 3 df, p = 0.6815 (alpha 0.05). Cramer's V 0.0349. Warning: 1 of 8 cells (12%) have an expected count below 5 (smallest 3.71). The chi-square p-value can be wrong. Use fisher_exact.",
 "metrics": {
  "chi2": 1.5035356413644951,
  "df": 3,
  "p_value": 0.6814545468936668,
  "n": 1233,
  "cramers_v": 0.03492008767758139,
  "min_expected": 3.708029197080292,
  "n_cells_expected_below_5": 1,
  "fraction_cells_expected_below_5": 0.125
 },
 "data": {
  "test": "Pearson chi-square test",
  "correction": false,
  "alpha": 0.05,
  "significant": false,
  "warning": "Warning: 1 of 8 cells (12%) have an expected count below 5 (smallest 3.71). The chi-square p-value can be wrong. Use fisher_exact.",
  "expected": [
   [
    81.62692619626927,
    9.373073803730739
   ],
   [
    420.6926196269262,
    48.307380373073805
   ],
   [
    571.3884833738848,
    65.61151662611516
   ],
   [
    32.29197080291971,
    3.708029197080292
   ]
  ],
  "rows": [
   "Divorced/separated",
   "Married/cohabitating",
   "Single",
   "Widowed"
  ],
  "columns": [
   "0",
   "1"
  ]
 },
 "table": {
  "columns": [
   "marital_status",
   "0",
   "1"
  ],
  "rows": [
   [
    "Divorced/separated",
    81,
    10
   ],
   [
    "Married/cohabitating",
    424,
    45
   ],
   [
    "Single",
    567,
    70
   ],
   [
    "Widowed",
    34,
    2
   ]
  ],
  "n_rows": 4,
  "path": "{work}/chi_square_test-3/observed_counts.csv"
 },
 "outputs": [
  {
   "path": "{work}/chi_square_test-3/expected_counts.csv",
   "kind": "table",
   "name": "expected_counts.csv"
  }
 ]
}

{
 "ok": true,
 "summary": "Pearson chi-square test, 2 x 2 table, n = 1233: chi-square 4.477 on 1 df, p = 0.03436 (alpha 0.05). Cramer's V 0.0603.",
 "metrics": {
  "chi2": 4.47652666051148,
  "df": 1,
  "p_value": 0.03436348964265445,
  "n": 1233,
  "cramers_v": 0.060254439286915616,
  "min_expected": 21.527169505271694,
  "n_cells_expected_below_5": 0,
  "fraction_cells_expected_below_5": 0
 },
 "data": {
  "test": "Pearson chi-square test",
  "correction": false,
  "alpha": 0.05,
  "significant": true,
  "warning": "",
  "expected": [
   [
    918.5271695052717,
    105.4728304947283
   ],
   [
    187.4728304947283,
    21.527169505271694
   ]
  ],
  "rows": [
   "Not in poverty",
   "Poverty/extreme poverty"
  ],
  "columns": [
   "0",
   "1"
  ]
 },
 "table": {
  "columns": [
   "poverty",
   "0",
   "1"
  ],
  "rows": [
   [
    "Not in poverty",
    927,
    97
   ],
   [
    "Poverty/extreme poverty",
    179,
    30
   ]
  ],
  "n_rows": 2,
  "path": "{work}/chi_square_test-4/observed_counts.csv"
 },
 "outputs": [
  {
   "path": "{work}/chi_square_test-4/expected_counts.csv",
   "kind": "table",
   "name": "expected_counts.csv"
  }
 ]
}

{
 "ok": true,
 "summary": "Fisher exact test, 4 x 2 table, n = 1233, two-sided: p = 0.742 (alpha 0.05). Method: exact (all tables with the same margins).",
 "metrics": {
  "n": 1233,
  "p_value": 0.741954958020354
 },
 "data": {
  "alternative": "two-sided",
  "alpha": 0.05,
  "rows": [
   "Divorced/separated",
   "Married/cohabitating",
   "Single",
   "Widowed"
  ],
  "columns": [
   "0",
   "1"
  ],
  "method": "exact (all tables with the same margins)",
  "significant": false
 },
 "table": {
  "columns": [
   "marital_status",
   "0",
   "1"
  ],
  "rows": [
   [
    "Divorced/separated",
    81,
    10
   ],
   [
    "Married/cohabitating",
    424,
    45
   ],
   [
    "Single",
    567,
    70
   ],
   [
    "Widowed",
    34,
    2
   ]
  ],
  "n_rows": 4,
  "path": "{work}/fisher_exact-1/fisher_counts.csv"
 }
}

{"ok":true,"summary":"Logistic regression of default = 1 against 0, n = 1233, 127 events. drug_use[Yes] OR 4.782 (95% CI 3.052 to 7.491, p = 8.38e-12); bmi[Overweight/Obese] OR 0.8777 (95% CI 0.4444 to 1.733, p = 0.707); bmi[Underweight] OR 2.08 (95% CI 1.214 to 3.563, p = 0.0077); mdr_tb[Yes] OR 3.038 (95% CI 1.578 to 5.851, p = 0.000886); alcohol_weekly[Yes] OR 2.222 (95% CI 1.401 to 3.525, p = 0.000695); hiv_status[Positive] OR 1.392 (95% CI 0.4167 to 4.654, p = 0.591); hiv_status[Test not done] OR 2.303 (95% CI 1.5 to 3.536, p = 0.000137); secondary_education[No] OR 1.55 (95% CI 1.032 to 2.329, p = 0.0349). Likelihood-ratio test of the model: chi-square 139.5 on 8 df, p = 3.076e-26. Wald confidence intervals. Likelihood-ratio test of each categorical covariate: bmi chi-square 7.273 on 2 df, p = 0.02635; hiv_status chi-square 14.06 on 2 df, p = 0.0008845. Reference levels: drug_use = No, bmi = Normal, mdr_tb = No, alcohol_weekly = No, hiv_status = Negative, secondary_education = Yes.","metrics":{"n":1233,"events":127,"non_events":1106,"n_dropped":0,"log_likelihood":-339.16700926664714,"null_log_likelihood":-408.8958940401771,"lr_chi2":139.4577695470599,"lr_df":8,"lr_p":3.076324272310018e-26,"aic":696.3340185332943,"pseudo_r2_mcfadden":0.1705296770886101,"events_per_parameter":15.875,"converged":1,"coef_const":-3.497632150067933,"se_const":0.2124385984966824,"p_const":6.633082638853354e-61,"or_const":0.030268971015494868,"or_lo_const":0.01996041629112351,"or_hi_const":0.045901377655350385,"coef_drug_use_Yes":1.5647650527214947,"se_drug_use_Yes":0.2290396468725688,"p_drug_use_Yes":8.382509658407017e-12,"or_drug_use_Yes":4.78155139142713,"or_lo_drug_use_Yes":3.0521784873914464,"or_hi_drug_use_Yes":7.490791840420467,"coef_bmi_Overweight_Obese":-0.13043635986406035,"se_bmi_Overweight_Obese":0.3472058878406436,"p_bmi_Overweight_Obese":0.7071589799396698,"or_bmi_Overweight_Obese":0.8777123489045796,"or_lo_bmi_Overweight_Obese":0.4444368093830721,"or_hi_bmi_Overweight_Obese":1.7333824542772827,"coef_bmi_Underweight":0.7321286903849328,"se_bmi_Underweight":0.2747274170249356,"p_bmi_Underweight":0.007700488734103748,"or_bmi_Underweight":2.079502516579366,"or_lo_bmi_Underweight":1.2136991532262786,"or_hi_bmi_Underweight":3.5629346077773034,"coef_mdr_tb_Yes":1.1113587461354562,"se_mdr_tb_Yes":0.33430051843768804,"p_mdr_tb_Yes":0.0008859940232706892,"or_mdr_tb_Yes":3.0384841194494236,"or_lo_mdr_tb_Yes":1.577974038264568,"or_hi_mdr_tb_Yes":5.8507843096708845,"coef_alcohol_weekly_Yes":0.7984029127919758,"se_alcohol_weekly_Yes":0.23539707354057635,"p_alcohol_weekly_Yes":0.0006945327829807024,"or_alcohol_weekly_Yes":2.2219893823639905,"or_lo_alcohol_weekly_Yes":1.400785466210615,"or_hi_alcohol_weekly_Yes":3.5246202465924,"coef_hiv_status_Positive":0.3310904815764532,"se_hiv_status_Positive":0.6155952108100741,"p_hiv_status_Positive":0.5906889122875556,"or_hiv_status_Positive":1.3924857809169304,"or_lo_hiv_status_Positive":0.41667282366518404,"or_hi_hiv_status_
... (1000 more characters in the session record)

{
 "ok": true,
 "summary": "holm correction of 4 p-values at alpha 0.05: 2 rejected.",
 "metrics": {
  "n_tests": 4,
  "n_rejected": 2
 },
 "table": {
  "columns": [
   "p_raw",
   "p_adjusted",
   "reject"
  ],
  "rows": [
   [
    0.00003945092420796785,
    0.0001578036968318714,
    true
   ],
   [
    0.00603898829164167,
    0.01811696487492501,
    true
   ],
   [
    0.741954958020354,
    0.741954958020354,
    false
   ],
   [
    0.03436348964265445,
    0.0687269792853089,
    false
   ]
  ],
  "n_rows": 4,
  "path": "{work}/adjust_pvalues-1/adjusted_p.csv"
 }
}

p_raw,p_adjusted,reject
3.945092420796785e-05,0.0001578036968318714,True
0.00603898829164167,0.01811696487492501,True
0.741954958020354,0.741954958020354,False
0.03436348964265445,0.0687269792853089,False

The final answer

The model wrote this answer. The harness shows it to the scientist after the review.

Settings used, from the decision record: Significance level (alpha): 0.05 · Correction for a list of p-values: holm · Covariates of the logistic model: drug_use, bmi, mdr_tb, alcohol_weekly, hiv_status, secondary_education · Outcome value that counts as the event: 1 · Reference level of each categorical covariate: drug_use=No, bmi=Normal, mdr_tb=No, alcohol_weekly=No, hiv_status=Negative, secondary_education=Yes · Yates continuity correction for a 2x2 chi-square test: false · Sidedness of the Fisher exact test: two-sided.

Values that are not scored

• A trap value is the result of a wrong method, for example an unpaired test on paired data. A trap value in a main step means that the run used the wrong method. A trap value in a comparison run is correct, because the record keeps the scientist's choice.
• An optional value is a second result of the same method. The run can compute it or not.
• A reference value comes from a different program version or a check run by us. We show it for comparison.

Checks

Review findings

The review recorded 4 findings. A rule finding comes from a fixed check in the harness. A referee finding comes from a second model that reads the record. The harness shows the findings to the scientist with the final answer. The record does not mark a finding as fixed. Thus a finding from an early review round can apply to a draft that the model corrected later.

Numbers in the answer

The last claim check read 79 numbers in the answer. 79 numbers match a logged result. 0 numbers have no source in the record.

Deviations

The model did not try to change a choice of the scientist.

Failed tool calls

No tool call failed.

Data integrity

Each data file has the same SHA-256 hash now as at the time of the step that read it. Where the download script (fetch.sh) gives a hash, the file also has that hash. The run did not change the data.

A SHA-256 hash is a fingerprint of the file contents. If one byte of the file changes, the hash changes. The table shows the first 12 characters.

How to repeat it

Get the data. The script downloads the files and checks their SHA-256 hashes where it lists them.

CUVETTE_DATA={data} bash bench/papers/lackey2015-tb-default/fetch.sh

Run the same case with Cuvette. The script gives the same answers from bench/papers/lackey2015-tb-default/bench.yaml.

cuvette bench papers --papers lackey2015-tb-default --models claude:claude-haiku-5-5

Repeat each step by hand in the program. For each step, the harness records a manual route: the menu path or the code that gives the same result. This list does not include comparison runs.

1. inspect_table (step n1)

   Code

   print(pd.read_csv(path).describe(include='all'))

   • path

     {data}/lackey2015-tb-default/tb_default.csv

   • Note: The tool also counts distinct values and missing values for each column.

   The manual route that the harness recorded

   ga_biostats.inspect_table(path="{data}/lackey2015-tb-default/tb_default.csv")

   The manual route uses the same method. The note in the route gives the known difference.

2. chi_square_test (step n2)

   Code

   table = pandas.crosstab(df[row], df[column])
   chi2, p, dof, expected = scipy.stats.chi2_contingency(table, correction=False)   # correction=True is Yates, 2x2 only
   R: chisq.test(table(df$row, df$column), correct = FALSE); chisq.test(...)$expected

   • Make the cross-table of the two columns, or type the counts.
   • Run the Pearson chi-square test. For a 2x2 table, choose the Yates correction or no correction.
   • Read the expected counts. If a cell has an expected count below 5, use the Fisher exact test.
   • In SPSS Analyze>Descriptive Statistics>Crosstabs. Put one variable in Row(s) and one in Column(s). Click Statistics and select Chi-square. Click Cells and select Expected. SPSS shows the Pearson test, the continuity correction for a 2x2 table and a footnote with the number of cells below 5.
   • In Stata tabulate row column, chi2 expected (menu Statistics>Summaries, tables, and tests > Frequency tables > Two-way table with measures of association).
   • In GraphPad Prism enter a Contingency table, then Analyze>Contingency table analyses > Chi-square (and Fisher's exact) test. In JMP: Analyze>Fit Y by X with two nominal columns.
   • Continuity Correction (SPSS row) or Yates correction = false
   • Warning: If you keep the default both shown (SPSS); Yates on (R, 2x2), you get a different result.

   The manual route that the harness recorded

   ga_biostats.chi_square_test(path="{data}/lackey2015-tb-default/tb_default.csv", row="age_group", column="default", correction=False, alpha=0.05)

   The manual route gives the same numbers. An automatic test in Cuvette checks this.

3. chi_square_test (step n3)

   Code

   table = pandas.crosstab(df[row], df[column])
   chi2, p, dof, expected = scipy.stats.chi2_contingency(table, correction=False)   # correction=True is Yates, 2x2 only
   R: chisq.test(table(df$row, df$column), correct = FALSE); chisq.test(...)$expected

   • Make the cross-table of the two columns, or type the counts.
   • Run the Pearson chi-square test. For a 2x2 table, choose the Yates correction or no correction.
   • Read the expected counts. If a cell has an expected count below 5, use the Fisher exact test.
   • In SPSS Analyze>Descriptive Statistics>Crosstabs. Put one variable in Row(s) and one in Column(s). Click Statistics and select Chi-square. Click Cells and select Expected. SPSS shows the Pearson test, the continuity correction for a 2x2 table and a footnote with the number of cells below 5.
   • In Stata tabulate row column, chi2 expected (menu Statistics>Summaries, tables, and tests > Frequency tables > Two-way table with measures of association).
   • In GraphPad Prism enter a Contingency table, then Analyze>Contingency table analyses > Chi-square (and Fisher's exact) test. In JMP: Analyze>Fit Y by X with two nominal columns.
   • Continuity Correction (SPSS row) or Yates correction = false
   • Warning: If you keep the default both shown (SPSS); Yates on (R, 2x2), you get a different result.

   The manual route that the harness recorded

   ga_biostats.chi_square_test(path="{data}/lackey2015-tb-default/tb_default.csv", row="sex", column="default", correction=False, alpha=0.05)

   The manual route gives the same numbers. An automatic test in Cuvette checks this.

4. chi_square_test (step n4)

   Code

   table = pandas.crosstab(df[row], df[column])
   chi2, p, dof, expected = scipy.stats.chi2_contingency(table, correction=False)   # correction=True is Yates, 2x2 only
   R: chisq.test(table(df$row, df$column), correct = FALSE); chisq.test(...)$expected

   • Make the cross-table of the two columns, or type the counts.
   • Run the Pearson chi-square test. For a 2x2 table, choose the Yates correction or no correction.
   • Read the expected counts. If a cell has an expected count below 5, use the Fisher exact test.
   • In SPSS Analyze>Descriptive Statistics>Crosstabs. Put one variable in Row(s) and one in Column(s). Click Statistics and select Chi-square. Click Cells and select Expected. SPSS shows the Pearson test, the continuity correction for a 2x2 table and a footnote with the number of cells below 5.
   • In Stata tabulate row column, chi2 expected (menu Statistics>Summaries, tables, and tests > Frequency tables > Two-way table with measures of association).
   • In GraphPad Prism enter a Contingency table, then Analyze>Contingency table analyses > Chi-square (and Fisher's exact) test. In JMP: Analyze>Fit Y by X with two nominal columns.
   • Continuity Correction (SPSS row) or Yates correction = false
   • Warning: If you keep the default both shown (SPSS); Yates on (R, 2x2), you get a different result.

   The manual route that the harness recorded

   ga_biostats.chi_square_test(path="{data}/lackey2015-tb-default/tb_default.csv", row="marital_status", column="default", correction=False, alpha=0.05)

   The manual route gives the same numbers. An automatic test in Cuvette checks this.

5. chi_square_test (step n5)

   Code

   table = pandas.crosstab(df[row], df[column])
   chi2, p, dof, expected = scipy.stats.chi2_contingency(table, correction=False)   # correction=True is Yates, 2x2 only
   R: chisq.test(table(df$row, df$column), correct = FALSE); chisq.test(...)$expected

   • Make the cross-table of the two columns, or type the counts.
   • Run the Pearson chi-square test. For a 2x2 table, choose the Yates correction or no correction.
   • Read the expected counts. If a cell has an expected count below 5, use the Fisher exact test.
   • In SPSS Analyze>Descriptive Statistics>Crosstabs. Put one variable in Row(s) and one in Column(s). Click Statistics and select Chi-square. Click Cells and select Expected. SPSS shows the Pearson test, the continuity correction for a 2x2 table and a footnote with the number of cells below 5.
   • In Stata tabulate row column, chi2 expected (menu Statistics>Summaries, tables, and tests > Frequency tables > Two-way table with measures of association).
   • In GraphPad Prism enter a Contingency table, then Analyze>Contingency table analyses > Chi-square (and Fisher's exact) test. In JMP: Analyze>Fit Y by X with two nominal columns.
   • Continuity Correction (SPSS row) or Yates correction = false
   • Warning: If you keep the default both shown (SPSS); Yates on (R, 2x2), you get a different result.

   The manual route that the harness recorded

   ga_biostats.chi_square_test(path="{data}/lackey2015-tb-default/tb_default.csv", row="poverty", column="default", correction=False, alpha=0.05)

   The manual route gives the same numbers. An automatic test in Cuvette checks this.

6. fisher_exact (step n6)

   Code

   scipy.stats.fisher_exact(table, alternative="two-sided")                      # 2x2 p
   scipy.stats.contingency.odds_ratio(table, kind="conditional").confidence_interval()
   R: fisher.test(table)   # same p and conditional odds ratio; r x c tables use the network algorithm

   • Make the cross-table of the two columns, or type the counts.
   • Run the Fisher exact test. For a 2x2 table, read the conditional odds ratio and its exact interval.
   • For a larger table, the tool counts all tables with the same margins. If there are too many, it gives a Monte Carlo p with a fixed seed.
   • In SPSS Analyze>Descriptive Statistics>Crosstabs>Statistics>Chi-square. SPSS shows Fisher's Exact Test for a 2x2 table. For a larger table, click Exact and select Exact (needs the Exact Tests module).
   • In Stata: tabulate row column, exact. In SAS: PROC FREQ; TABLES row*column / FISHER.
   • In GraphPad Prism Contingency table > Analyze>Chi-square (and Fisher's exact) test, then Fisher's exact test (2x2 tables only).
   • alternative = two-sided
   • Note: The p-value of a 2x2 table and of an enumerated larger table equals R fisher.test. A large table gets a Monte Carlo p (R uses an exact network algorithm). The upper bound of the odds ratio interval can differ from R in the fourth digit. SPSS and Stata print the sample odds ratio, not the conditional odds ratio (sample_odds_ratio in the result).

   The manual route that the harness recorded

   ga_biostats.fisher_exact(path="{data}/lackey2015-tb-default/tb_default.csv", row="marital_status", column="default", alternative="two-sided", alpha=0.05, seed=1)

   The manual route uses the same method. The note in the route gives the known difference.

7. fit_logistic (step n7)

   Code

   X = pandas.get_dummies(df[covariates], columns=categorical, drop_first=True)   # first sorted level is the reference
   res = statsmodels.api.Logit(df[outcome], statsmodels.api.add_constant(X)).fit()
   numpy.exp(res.params); numpy.exp(res.conf_int()); res.llr, res.llr_pvalue
   R: glm(outcome ~ age + factor(race), family = binomial); exp(confint.default(fit)); drop1(fit, test = "LRT")

   • Drop rows with a missing value in the outcome or a covariate. Apply the subset first if there is one.
   • Code each categorical covariate as indicator columns. Leave out the reference level.
   • Fit the logistic model. Take exp() of each coefficient and of each Wald interval bound for the odds ratios.
   • For the likelihood-ratio test of a covariate, fit the model without it and compare twice the difference of the log-likelihoods with a chi-square distribution.
   • In SPSS Analyze>Regression>Binary Logistic. Put the outcome in Dependent and the covariates in Covariates. Click Categorical, move each categorical covariate, set Reference Category to First and click Change. In Options, select CI for exp(B).
   • In Stata logistic outcome age lwt i.race ftv (menu Statistics>Binary outcomes > Logistic regression (reporting odds ratios)). Then lrtest for a covariate.
   • In SAS: PROC LOGISTIC; CLASS race (PARAM=REF REF=FIRST); MODEL outcome(EVENT='1') = age lwt race ftv / CLODDS=WALD.
   • In GraphPad Prism Analyze>Multiple variable analyses > Multiple logistic regression. In JMP: Analyze>Fit Model, Personality Nominal Logistic.
   • Dependent = default
   • Covariates = drug_use, bmi, mdr_tb, alcohol_weekly, hiv_status, secondary_education
   • Reference Category = drug_use=No, bmi=Normal, mdr_tb=No, alcohol_weekly=No, hiv_status=Negative, secondary_education=Yes
   • CI for exp(B) = 0.05
   • Warning: If you keep the default Last (SPSS), effect coding (SAS CLASS), you get a different result.
   • Warning: If you keep the default 95, you get a different result.

   The manual route that the harness recorded

   ga_biostats.fit_logistic(path="{data}/lackey2015-tb-default/tb_default.csv", outcome="default", covariates="drug_use, bmi, mdr_tb, alcohol_weekly, hiv_status, secondary_education", event="1", categorical="bmi, hiv_status", reference="drug_use=No, bmi=Normal, mdr_tb=No, alcohol_weekly=No, hiv_status=Negative, secondary_education=Yes", alpha=0.05)

   The manual route gives the same numbers. An automatic test in Cuvette checks this.

8. adjust_pvalues (step n8)

   Code

   statsmodels.stats.multitest.multipletests(pvals, alpha=0.05, method='holm')

   • method = holm
   • Warning: If you keep the default hs, you get a different result.

   The manual route that the harness recorded

   ga_biostats.adjust_pvalues(pvalues=[0.00003945092420796785, 0.00603898829164167, 0.741954958020354, 0.03436348964265445], method="holm", alpha=0.05)

   The manual route gives the same numbers. An automatic test in Cuvette checks this.

Figure

Run facts

The session

This is the session as the scientist sees it, in order. Decision cards show the answer that the script gave and where that answer comes from. Each step shows the program, its version, the input file hash and the outputs. Click a line to see more.

- Setup values:
Design:
- What is the unit of replication? (replicate): subjects, patients or animals (one value per subject)
Inference:
- Significance level (alpha) (alpha): 0.05
Ask the scientist: Paired test (paired), Test family for two groups (two_group_test), Assume equal variances (unpaired t-test) (equal_variance), Sidedness (alternative), ANOVA variance assumption (anova_variance), Pairwise comparisons after ANOVA (posthoc), Correction for a list of p-values (p_adjust_method), Model formula (model_formula), Standard errors for the linear model (standard_errors), Random effects of the mixed model (random_effects), Fit by REML (reml), Covariates of the Cox model (cox_covariates), Ties method of the Cox model (ties), Strata of the Cox model (cox_strata), Type of sums of squares (ss_type), Comparisons of the cell means after a factorial ANOVA (factorial_posthoc), Sphericity correction for repeated measures (sphericity_correction), Correction after a rank test over many groups (rank_p_adjust), Exact or approximate p value for a rank test (exact_p), Covariates of the logistic model (logistic_covariates), Outcome value that counts as the event (logistic_event), Reference level of each categorical covariate (reference_levels), Yates continuity correction for a 2x2 chi-square test (chi_square_correction), Sidedness of the Fisher exact test (fisher_alternative), Method of the McNemar test (mcnemar_method).
The harness fills these values into your tool calls automatically. If you give a different value, the harness logs a deviation and shows it to the scientist.
- Mode is now watch: you run the steps.

{
 "ok": true,
 "summary": "1233 rows x 19 columns.",
 "metrics": {
  "n_rows": 1233,
  "n_columns": 19
 },
 "table": {
  "columns": [
   "column",
   "dtype",
   "n_missing",
   "n_unique",
   "min",
   "max",
   "first_values"
  ],
  "rows": [
   [
    "id",
    "int64",
    0,
    1233,
    883,
    2347,
    "1008, 1009, 1010, 1011, 1014"
   ],
   [
    "age_group",
    "object",
    0,
    4,
    null,
    null,
    "38 and older, 27 to 37, 22 to 26, 21 and younger"
   ],
   [
    "sex",
    "object",
    0,
    2,
    null,
    null,
    "Male, Female"
   ],
   [
    "marital_status",
    "object",
    0,
    4,
    null,
    null,
    "Single, Married/cohabitating, Divorced/separated, Widowed"
   ],
   [
    "poverty",
    "object",
    0,
    2,
    null,
    null,
    "Not in poverty, Poverty/extreme poverty"
   ],
   [
    "prison",
    "object",
    0,
    2,
    null,
    null,
    "No, Yes"
   ],
   [
    "secondary_education",
    "object",
    0,
    2,
    null,
    null,
    "Yes, No"
   ],
   [
    "smoking",
    "object",
    0,
    3,
    null,
    null,
    "Used to smoke, Never smoked, Currently smokes"
   ],
   [
    "alcohol_weekly",
    "object",
    0,
    2,
    null,
    null,
    "Yes, No"
   ],
   [
    "drug_use",
    "object",
    0,
    2,
    null,
    null,
    "No, Yes"
   ],
   [
    "rehab",
    "object",
    0,
    2,
    null,
    null,
    "No, Yes"
   ],
   [
    "mdr_tb",
    "object",
    0,
    2,
    null,
    null,
    "No, Yes"
   ],
   [
    "bmi",
    "object",
    0,
    3,
    null,
    null,
    "Underweight, Normal, Overweight/Obese"
   ],
   [
    "chronic_disease",
    "object",
    0,
    2,
    null,
    null,
    "No, Yes"
   ],
   [
    "hiv_status",
    "object",
    0,
    3,
    null,
    null,
    "Negative, Test not done, Positive"
   ]
  ],
  "n_rows": 19,
  "path": "{work}/inspect_table-1/columns.csv"
 }
}

{
 "ok": false,
 "error": "The group column must have exactly 2 levels. It has 4: ['21 and younger', '22 to 26', '27 to 37', '38 and older']."
}

{
 "ok": true,
 "summary": "Mann-Whitney U test, two-sided: Male minus Female = 0.07271, statistic 1.953e+05, p = 3.976e-05 (alpha 0.05).",
 "metrics": {
  "n_a": 490,
  "n_b": 743,
  "mean_diff": 0.07271403850907791,
  "statistic": 195271.5,
  "p_value": 0.00003976350391828826,
  "mean_a": 0.05918367346938776,
  "mean_b": 0.13189771197846567
 },
 "data": {
  "test": "Mann-Whitney U test",
  "alternative": "two-sided",
  "direction": "Male minus Female",
  "alpha": 0.05,
  "significant": true
 },
 "table": {
  "columns": [
   "test",
   "group_a",
   "group_b",
   "n_a",
   "n_b",
   "mean_diff_b_minus_a",
   "statistic",
   "df",
   "p",
   "ci_lo",
   "ci_hi",
   "effect"
  ],
  "rows": [
   [
    "Mann-Whitney U test",
    "Female",
    "Male",
    490,
    743,
    0.07271403850907791,
    195271.5,
    null,
    0.00003976350391828826,
    null,
    null,
    null
   ]
  ],
  "n_rows": 1,
  "path": "{work}/compare_two_groups-2/two_group_test.csv"
 }
}

{
 "ok": false,
 "error": "The group column must have exactly 2 levels. It has 4: ['Divorced/separated', 'Married/cohabitating', 'Single', 'Widowed']."
}

{
 "ok": true,
 "summary": "Mann-Whitney U test, two-sided: Poverty/extreme poverty minus Not in poverty = 0.04881, statistic 1.122e+05, p = 0.03445 (alpha 0.05).",
 "metrics": {
  "n_a": 1024,
  "n_b": 209,
  "mean_diff": 0.04881410735645933,
  "statistic": 112231.5,
  "p_value": 0.03445383641346311,
  "mean_a": 0.0947265625,
  "mean_b": 0.14354066985645933
 },
 "data": {
  "test": "Mann-Whitney U test",
  "alternative": "two-sided",
  "direction": "Poverty/extreme poverty minus Not in poverty",
  "alpha": 0.05,
  "significant": true
 },
 "table": {
  "columns": [
   "test",
   "group_a",
   "group_b",
   "n_a",
   "n_b",
   "mean_diff_b_minus_a",
   "statistic",
   "df",
   "p",
   "ci_lo",
   "ci_hi",
   "effect"
  ],
  "rows": [
   [
    "Mann-Whitney U test",
    "Not in poverty",
    "Poverty/extreme poverty",
    1024,
    209,
    0.04881410735645933,
    112231.5,
    null,
    0.03445383641346311,
    null,
    null,
    null
   ]
  ],
  "n_rows": 1,
  "path": "{work}/compare_two_groups-4/two_group_test.csv"
 }
}

- You said what you will do next, but you did not call a tool. Do it now with a tool call, or give your final answer.

{
 "ok": false,
 "error": "categorical and reference must name covariates of the model. Not covariates: chronic_disease, smoking, rehab, marital_status, prison, poverty, age_group."
}

{
 "ok": false,
 "error": "categorical and reference must name covariates of the model. Not covariates: chronic_disease, smoking, rehab, marital_status, prison, poverty, age_group."
}

{"ok":true,"summary":"Logistic regression of default = 1 against 0, n = 1233, 127 events. drug_use[Yes] OR 4.782 (95% CI 3.052 to 7.491, p = 8.38e-12); bmi[Overweight/Obese] OR 0.8777 (95% CI 0.4444 to 1.733, p = 0.707); bmi[Underweight] OR 2.08 (95% CI 1.214 to 3.563, p = 0.0077); mdr_tb[Yes] OR 3.038 (95% CI 1.578 to 5.851, p = 0.000886); alcohol_weekly[Yes] OR 2.222 (95% CI 1.401 to 3.525, p = 0.000695); hiv_status[Positive] OR 1.392 (95% CI 0.4167 to 4.654, p = 0.591); hiv_status[Test not done] OR 2.303 (95% CI 1.5 to 3.536, p = 0.000137); secondary_education[No] OR 1.55 (95% CI 1.032 to 2.329, p = 0.0349). Likelihood-ratio test of the model: chi-square 139.5 on 8 df, p = 3.076e-26. Wald confidence intervals. Likelihood-ratio test of each categorical covariate: bmi chi-square 7.273 on 2 df, p = 0.02635; hiv_status chi-square 14.06 on 2 df, p = 0.0008845. Reference levels: drug_use = No, bmi = Normal, mdr_tb = No, alcohol_weekly = No, hiv_status = Negative, secondary_education = Yes.","metrics":{"n":1233,"events":127,"non_events":1106,"n_dropped":0,"log_likelihood":-339.16700926664714,"null_log_likelihood":-408.8958940401771,"lr_chi2":139.4577695470599,"lr_df":8,"lr_p":3.076324272310018e-26,"aic":696.3340185332943,"pseudo_r2_mcfadden":0.1705296770886101,"events_per_parameter":15.875,"converged":1,"coef_const":-3.497632150067933,"se_const":0.2124385984966824,"p_const":6.633082638853354e-61,"or_const":0.030268971015494868,"or_lo_const":0.01996041629112351,"or_hi_const":0.045901377655350385,"coef_drug_use_Yes":1.5647650527214947,"se_drug_use_Yes":0.2290396468725688,"p_drug_use_Yes":8.382509658407017e-12,"or_drug_use_Yes":4.78155139142713,"or_lo_drug_use_Yes":3.0521784873914464,"or_hi_drug_use_Yes":7.490791840420467,"coef_bmi_Overweight_Obese":-0.13043635986406035,"se_bmi_Overweight_Obese":0.3472058878406436,"p_bmi_Overweight_Obese":0.7071589799396698,"or_bmi_Overweight_Obese":0.8777123489045796,"or_lo_bmi_Overweight_Obese":0.4444368093830721,"or_hi_bmi_Overweight_Obese":1.7333824542772827,"coef_bmi_Underweight":0.7321286903849328,"se_bmi_Underweight":0.2747274170249356,"p_bmi_Underweight":0.007700488734103748,"or_bmi_Underweight":2.079502516579366,"or_lo_bmi_Underweight":1.2136991532262786,"or_hi_bmi_Underweight":3.5629346077773034,"coef_mdr_tb_Yes":1.1113587461354562,"se_mdr_tb_Yes":0.33430051843768804,"p_mdr_tb_Yes":0.0008859940232706892,"or_mdr_tb_Yes":3.0384841194494236,"or_lo_mdr_tb_Yes":1.577974038264568,"or_hi_mdr_tb_Yes":5.8507843096708845,"coef_alcohol_weekly_Yes":0.7984029127919758,"se_alcohol_weekly_Yes":0.23539707354057635,"p_alcohol_weekly_Yes":0.0006945327829807024,"or_alcohol_weekly_Yes":2.2219893823639905,"or_lo_alcohol_weekly_Yes":1.400785466210615,"or_hi_alcohol_weekly_Yes":3.5246202465924,"coef_hiv_status_Positive":0.3310904815764532,"se_hiv_status_Positive":0.6155952108100741,"p_hiv_status_Positive":0.5906889122875556,"or_hiv_status_Positive":1.3924857809169304,"or_lo_hiv_status_Positive":0.41667282366518404,"or_hi_hiv_status_
... (1000 more characters in the session record)

The final answer

The model wrote this answer. The harness shows it to the scientist after the review.

Step results that the answer does not mention, from the record:
- n1 inspect_table: 1233 rows x 19 columns.
- n2 compare_two_groups: Mann-Whitney U test, two-sided: Male minus Female = 0.07271, statistic 1.953e+05, p = 3.976e-05 (alpha 0.05).
- n3 compare_two_groups: Mann-Whitney U test, two-sided: Poverty/extreme poverty minus Not in poverty = 0.04881, statistic 1.122e+05, p = 0.03445 (alpha 0.05).

Settings used, from the decision record: Significance level (alpha): 0.05 · Paired test: false · Test family for two groups: wilcoxon · Assume equal variances (unpaired t-test): false · Sidedness: two-sided · Covariates of the logistic model: drug_use, bmi, mdr_tb, alcohol_weekly, hiv_status, secondary_education · Outcome value that counts as the event: 1 · Reference level of each categorical covariate: drug_use=No, bmi=Normal, mdr_tb=No, alcohol_weekly=No, hiv_status=Negative, secondary_education=Yes.

Values that are not scored

• A trap value is the result of a wrong method, for example an unpaired test on paired data. A trap value in a main step means that the run used the wrong method. A trap value in a comparison run is correct, because the record keeps the scientist's choice.
• An optional value is a second result of the same method. The run can compute it or not.
• A reference value comes from a different program version or a check run by us. We show it for comparison.

Checks

Review findings

The review recorded 3 findings. A rule finding comes from a fixed check in the harness. A referee finding comes from a second model that reads the record. The harness shows the findings to the scientist with the final answer. The record does not mark a finding as fixed. Thus a finding from an early review round can apply to a draft that the model corrected later.

Numbers in the answer

The last claim check read 35 numbers in the answer. 35 numbers match a logged result. 0 numbers have no source in the record.

Deviations

• The model asked for reference = age_group=38 and older, marital_status=Married/cohabitating, poverty=Not in poverty, prison=No, secondary_education=No, smoking=Never smoked, alcohol_weekly=No, drug_use=No, rehab=No, mdr_tb=No, bmi=Normal, chronic_disease=No, hiv_status=Negative. The scientist chose drug_use=No, bmi=Normal, mdr_tb=No, alcohol_weekly=No, hiv_status=Negative, secondary_education=Yes for Reference level of each categorical covariate. The harness kept drug_use=No, bmi=Normal, mdr_tb=No, alcohol_weekly=No, hiv_status=Negative, secondary_education=Yes.
• The model asked for reference = drug_use=No, bmi=Normal, mdr_tb=No, alcohol_weekly=No, hiv_status=Negative, secondary_education=No. The scientist chose drug_use=No, bmi=Normal, mdr_tb=No, alcohol_weekly=No, hiv_status=Negative, secondary_education=Yes for Reference level of each categorical covariate. The harness kept drug_use=No, bmi=Normal, mdr_tb=No, alcohol_weekly=No, hiv_status=Negative, secondary_education=Yes.
• The model asked for event = default. The scientist chose 1 for Outcome value that counts as the event. The harness kept 1.
• The model asked for reference = drug_use=No, bmi=Normal, mdr_tb=No, alcohol_weekly=No, hiv_status=Negative, secondary_education=No. The scientist chose drug_use=No, bmi=Normal, mdr_tb=No, alcohol_weekly=No, hiv_status=Negative, secondary_education=Yes for Reference level of each categorical covariate. The harness kept drug_use=No, bmi=Normal, mdr_tb=No, alcohol_weekly=No, hiv_status=Negative, secondary_education=Yes.
• The model asked for reference = drug_use=No, bmi=Normal, mdr_tb=No, alcohol_weekly=No, hiv_status=Negative, secondary_education=No. The scientist chose drug_use=No, bmi=Normal, mdr_tb=No, alcohol_weekly=No, hiv_status=Negative, secondary_education=Yes for Reference level of each categorical covariate. The harness kept drug_use=No, bmi=Normal, mdr_tb=No, alcohol_weekly=No, hiv_status=Negative, secondary_education=Yes.
• The model asked for event = default. The scientist chose 1 for Outcome value that counts as the event. The harness kept 1.
• The model asked for reference = drug_use=No, bmi=Normal, mdr_tb=No, alcohol_weekly=No, hiv_status=Negative, secondary_education=No. The scientist chose drug_use=No, bmi=Normal, mdr_tb=No, alcohol_weekly=No, hiv_status=Negative, secondary_education=Yes for Reference level of each categorical covariate. The harness kept drug_use=No, bmi=Normal, mdr_tb=No, alcohol_weekly=No, hiv_status=Negative, secondary_education=Yes.

Failed tool calls

7 tool calls failed. The model then tried again or used another tool. The session above shows each failure.

Data integrity

Each data file has the same SHA-256 hash now as at the time of the step that read it. Where the download script (fetch.sh) gives a hash, the file also has that hash. The run did not change the data.

A SHA-256 hash is a fingerprint of the file contents. If one byte of the file changes, the hash changes. The table shows the first 12 characters.

How to repeat it

Get the data. The script downloads the files and checks their SHA-256 hashes where it lists them.

CUVETTE_DATA={data} bash bench/papers/lackey2015-tb-default/fetch.sh

Run the same case with Cuvette. The script gives the same answers from bench/papers/lackey2015-tb-default/bench.yaml.

cuvette bench papers --papers lackey2015-tb-default --models ollama:qwen3:8b

Repeat each step by hand in the program. For each step, the harness records a manual route: the menu path or the code that gives the same result. This list does not include comparison runs.

1. inspect_table (step n1)

   Code

   print(pd.read_csv(path).describe(include='all'))

   • path

     {data}/lackey2015-tb-default/tb_default.csv

   • Note: The tool also counts distinct values and missing values for each column.

   The manual route that the harness recorded

   ga_biostats.inspect_table(path="{data}/lackey2015-tb-default/tb_default.csv")

   The manual route uses the same method. The note in the route gives the known difference.

2. compare_two_groups (step n2)

   Code

   scipy.stats.ttest_ind(x, y, equal_var=False)   # Welch. equal_var=True is Student.
   scipy.stats.mannwhitneyu(x, y)                  # rank test

   • equal_var = false
   • alternative = two-sided
   • Warning: If you keep the default true, you get a different result.

   The manual route that the harness recorded

   ga_biostats.compare_two_groups(path="{data}/lackey2015-tb-default/tb_default.csv", outcome="default", group="sex", test="wilcoxon", paired=False, alternative="two-sided", alpha=0.05, equal_variance=False)

   The manual route gives the same numbers. An automatic test in Cuvette checks this.

3. compare_two_groups (step n3)

   Code

   scipy.stats.ttest_ind(x, y, equal_var=False)   # Welch. equal_var=True is Student.
   scipy.stats.mannwhitneyu(x, y)                  # rank test

   • equal_var = false
   • alternative = two-sided
   • Warning: If you keep the default true, you get a different result.

   The manual route that the harness recorded

   ga_biostats.compare_two_groups(path="{data}/lackey2015-tb-default/tb_default.csv", outcome="default", group="poverty", test="wilcoxon", paired=False, alternative="two-sided", alpha=0.05, equal_variance=False)

   The manual route gives the same numbers. An automatic test in Cuvette checks this.

4. fit_logistic (step n4)

   Code

   X = pandas.get_dummies(df[covariates], columns=categorical, drop_first=True)   # first sorted level is the reference
   res = statsmodels.api.Logit(df[outcome], statsmodels.api.add_constant(X)).fit()
   numpy.exp(res.params); numpy.exp(res.conf_int()); res.llr, res.llr_pvalue
   R: glm(outcome ~ age + factor(race), family = binomial); exp(confint.default(fit)); drop1(fit, test = "LRT")

   • Drop rows with a missing value in the outcome or a covariate. Apply the subset first if there is one.
   • Code each categorical covariate as indicator columns. Leave out the reference level.
   • Fit the logistic model. Take exp() of each coefficient and of each Wald interval bound for the odds ratios.
   • For the likelihood-ratio test of a covariate, fit the model without it and compare twice the difference of the log-likelihoods with a chi-square distribution.
   • In SPSS Analyze>Regression>Binary Logistic. Put the outcome in Dependent and the covariates in Covariates. Click Categorical, move each categorical covariate, set Reference Category to First and click Change. In Options, select CI for exp(B).
   • In Stata logistic outcome age lwt i.race ftv (menu Statistics>Binary outcomes > Logistic regression (reporting odds ratios)). Then lrtest for a covariate.
   • In SAS: PROC LOGISTIC; CLASS race (PARAM=REF REF=FIRST); MODEL outcome(EVENT='1') = age lwt race ftv / CLODDS=WALD.
   • In GraphPad Prism Analyze>Multiple variable analyses > Multiple logistic regression. In JMP: Analyze>Fit Model, Personality Nominal Logistic.
   • Dependent = default
   • Covariates = drug_use, bmi, mdr_tb, alcohol_weekly, hiv_status, secondary_education
   • Reference Category = drug_use=No, bmi=Normal, mdr_tb=No, alcohol_weekly=No, hiv_status=Negative, secondary_education=Yes
   • CI for exp(B) = 0.05
   • Warning: If you keep the default Last (SPSS), effect coding (SAS CLASS), you get a different result.
   • Warning: If you keep the default 95, you get a different result.

   The manual route that the harness recorded

   ga_biostats.fit_logistic(path="{data}/lackey2015-tb-default/tb_default.csv", outcome="default", covariates="drug_use, bmi, mdr_tb, alcohol_weekly, hiv_status, secondary_education", event="1", categorical="drug_use, bmi, mdr_tb, alcohol_weekly, hiv_status, secondary_education", reference="drug_use=No, bmi=Normal, mdr_tb=No, alcohol_weekly=No, hiv_status=Negative, secondary_education=Yes", alpha=0.05)

   The manual route gives the same numbers. An automatic test in Cuvette checks this.

Figure

Run facts