Skip to content

Predefined Folds

Fix the cross-validation yourself: write a fold ID into a column of your data and name that column in training.split_column. COREMat then runs its outer CV over exactly these folds, with no seed search.

Typical reasons:

  • Comparison on identical folds — another framework, a Matbench task, a paper's published split.
  • A structure group_column does not express — a hand-balanced leave-one-paper-out, alloy families, time blocks.
  • Small n — a hold-out would cost more data than it tells; everything goes into the CV.
  • Reproducibility — the folds live in the data, not in a seed.

Fold IDs only: evaluation by OOF

sample_id,yield_strength,fold
1,450,0
2,470,1
3,490,2
4,455,0

training:
  split_column: fold

Every row is in the CV and in the final fit. There is no hold-out, so there is no test score: the run is judged by its OOF predictions alone. The reports say so — "Predefined folds, no hold-out" — and show the OOF predictions where a test set would otherwise be. No test files are written (y_test_y_pred_rnd*.csv, test SHAP), and the SHAP ranking comes from the OOF rows.

Fold IDs plus a hold-out: the test marker

Mark the rows you want held out with test (any case), or with -1 in an integer column:

sample_id,yield_strength,fold
1,450,0
2,470,1
3,490,2
4,455,test

The marker rows are a real hold-out: they take no part in the CV and none in the final fit, and the test score is computed on them exactly as for a random split. The CV runs over the fold IDs of the remaining rows; at least two must be left.

If the marker rows have no target (NaN), the run is in Submission Mode.

Pitfalls

  • Two folds: a column holding only 0 and 1 is always read as a binary train/test split. For a 2-fold CV write 1 and 2.
  • Fold count: one fold per distinct ID — {0, 2, 5} is a 3-fold CV, not 6.
  • Not with group_column: the two are mutually exclusive.

See training.split_column for the full reference, including the binary format.