Getting Started

Throughout this software and documentation, we use three names for the main roles that data can play in model development. Data splitting is not intrinsically limited to two or three sets, but a three-way distinction is common. Terminology varies considerably across scientific disciplines, so we use the following convention:

• Training set (train set, training data, and, especially in chemometrics, calibration set): the data used to fit or calibrate a model, including estimation of model parameters and any data-dependent preprocessing.
• Validation set (validation data, development set, dev set, tuning set): the data used during model selection, for example to compare candidate models or choose hyperparameters. Cross-validation can be used instead of a separate validation set.
• Test set (test data, holdout set, evaluation set, independent test set): data that play no role in model fitting or model selection and are used only to estimate the generalisation performance of the selected modelling procedure.

These names are not universal. In some fields, particularly biomedical and applied sciences, validation set or external validation set may refer to what we call a test set. Conversely, test set is sometimes used for data used during model development. In this documentation, however, training, validation, and test always refer to the three roles defined above.

Basic API

Most interactions with DataSplits use partition. It takes the data, a splitting strategy, and any information required to construct the requested cohorts.

We use the Iris dataset for the first examples:

using DataSplitsusing RDatasetsusing Randomiris = dataset("datasets", "iris")X = iris[:, 1:4]y = iris.Speciessize(X), length(y)

((150, 4), 150)

A simple train/test split is:

res = partition(    X,    RandomSplit();    train = 0.8,    test = 0.2,    rng = Xoshiro(42),)length(trainindices(res)), length(testindices(res))

(120, 30)

partition returns indices describing the split rather than copies of the data. The corresponding subsets can be obtained with splitdata:

X_train, X_test = splitdata(res, X)size(X_train), size(X_test)

((120, 4), (30, 4))

DataSplits supports three main forms of partition:

# Train / testres = partition(data, alg; train, test, kwargs...)# Train / validation / testres = partition(    data,    test_alg,    val_alg;    train,    validation,    test,    kwargs...,)# Cross-validationcvs = partition(data, cv_alg; kwargs...)

For a three-way split, test_alg first separates the test cohort. val_alg then divides the remaining data into training and validation cohorts. This makes it possible to use different strategies for final evaluation and model selection.

Using splits with machine-learning packages

DataSplits returns indices or views rather than imposing a modelling framework. This makes the same split usable with packages such as MLUtils, Flux, and MLJ.

For arrays, DataSplits follows the Julia ML convention used by MLUtils: observations are stored along the last dimension. A supervised dataset can therefore be split and passed directly to an MLUtils.DataLoader:

using DataSplitsusing MLUtilsres = partition(    X,    RandomSplit();    train = 0.8,    test = 0.2,    rng = Xoshiro(42),)train_data = trainview(res, X, y)test_data = testview(res, X, y)train_loader = DataLoader(    train_data;    batchsize = 32,    shuffle = true,)test_loader = DataLoader(    test_data;    batchsize = 32,    shuffle = false,)

When multiple data sources are supplied, trainview and testview return tuples of corresponding views. These can be consumed directly by DataLoader, keeping predictors and targets aligned.

The resulting loader can be used directly in a Flux training loop:

using Fluxopt_state = Flux.setup(Adam(), model)Flux.train!(    loss,    model,    train_loader,    opt_state,)

Cross-validation results can be converted directly to the (train, test) row-index pairs expected by MLJ:

using MLJcvs = partition(    X,    StratifiedKFold(5);    target = y,)mach = machine(model, X, y)evaluate!(    mach;    resampling = rowpairs(cvs),    measure = accuracy,)

Cohort sizes

train, validation, and test are specified when calling partition, rather than when constructing the splitting strategy. They can be given in three forms:

• Fractions: floating-point values in (0, 1) that sum to 1.0.
• Percentages: integers that sum to 100.
• Counts: integers that sum to numobs(data).

For example:

# Fractionspartition(X, RandomSplit(); train = 0.8, test = 0.2)# Percentagespartition(X, RandomSplit(); train = 80, test = 20)# Absolute countspartition(X, RandomSplit(); train = 120, test = 30)  # requires numobs(X) == 150

The same convention applies to three-way splits:

threeway = partition(    X,    RandomSplit(),    RandomSplit();    train = 0.6,    validation = 0.2,    test = 0.2,    rng = Xoshiro(42),)(    train = length(trainindices(threeway)),    validation = length(valindices(threeway)),    test = length(testindices(threeway)),)

(train = 90, validation = 30, test = 30)

Cross-validation strategies generally do not take train and test keywords because fold sizes are determined by the strategy itself:

cvs = partition(X, KFold(5))

Resampling strategies such as ShuffleSplit are an exception: they construct repeated train/test splits and therefore accept cohort sizes for each repetition.

Working with a split

partition returns split objects containing indices. The corresponding observations can be materialised with splitdata or accessed lazily with splitview.

Materialised subsets

splitdata returns all cohorts defined by a split:

X_train, X_test = splitdata(res, X)y_train, y_test = splitdata(res, y)(size(X_train), length(y_train)), (size(X_test), length(y_test))

(((120, 4), 120), ((30, 4), 30))

For a three-way split, it returns training, validation, and test subsets:

Aligned data sources can be split using the same result:

y_train, y_val, y_test = splitdata(threeway, y)(    train = length(y_train),    validation = length(y_val),    test = length(y_test),)

(train = 90, validation = 30, test = 30)

Individual cohorts can instead be retrieved from several aligned data sources at once with traindata, valdata and testdata:

X_train, y_train = traindata(threeway, X, y)X_val, y_val = valdata(threeway, X, y)X_test, y_test = testdata(threeway, X, y)(    train = (size(X_train), length(y_train)),    validation = (size(X_val), length(y_val)),    test = (size(X_test), length(y_test)),)

(train = ((90, 4), 90), validation = ((30, 4), 30), test = ((30, 4), 30))

Lazy views

When copying the underlying data is unnecessary or expensive, use splitview:

X_train_view, X_test_view = splitview(res, X)size(X_train_view), size(X_test_view)

((120,), (30,))

The corresponding single-cohort helpers are trainview, valview, and testview:

X_train_view, y_train_view = trainview(res, X, y)size(X_train_view), length(y_train_view)

((120,), 120)

Cross-validation

Cross-validation strategies return a CrossValidationSplit containing one train/test split for each fold.

cvs = partition(X, KFold(5))length(folds(cvs))

5

The folds can be used directly with splitview:

for (X_tr, X_te) in splitview(cvs, X)    println(length(X_tr), " train / ", length(X_te), " test")end

120 train / 30 test
120 train / 30 test
120 train / 30 test
120 train / 30 test
120 train / 30 test

The same split can be applied to aligned feature and target data:

for fold in folds(cvs)    X_tr, y_tr = trainview(fold, X, y)    X_te, y_te = testview(fold, X, y)    println(length(X_tr), " train / ", length(X_te), " test")end

120 train / 30 test
120 train / 30 test
120 train / 30 test
120 train / 30 test
120 train / 30 test

trainview and testview also accept the whole CrossValidationSplit rather than a single fold. They then return one tuple of co-indexed views per fold, which is convenient when the folds are needed up front, for instance to hand each one to a data loader:

train_folds = trainview(cvs, X, y)X_fold1, y_fold1 = train_folds[1](folds = length(train_folds), first_fold = (length(X_fold1), length(y_fold1)))

(folds = 5, first_fold = (120, 120))

The materialising counterparts traindata and testdata behave the same way but copy the observations instead of viewing them.

When performing model selection with cross-validation, any data-dependent preprocessing must also be fitted within the training portion of each fold.

Additional information

Some splitting strategies require information in addition to the data being split. These inputs are passed as keyword arguments to partition:

• target — response, property, or class labels, used by strategies such as SPXYSplit and StratifiedKFold.
• time — timestamps or another temporal ordering variable, used by strategies such as TimeSplit, TimeSeriesSplit, and BlockedCV.
• groups — group membership, used by strategies such as GroupShuffleSplit and GroupKFold.

For example:

partition(X, StratifiedKFold(5); target = classes)partition(X, GroupKFold(5); groups = patient_ids)partition(X, TimeSeriesSplit(5); time = timestamps)

Single-input shorthand

If the data being split are themselves the values required by the strategy, the corresponding keyword can be omitted.

For example, when splitting a vector of timestamps:

partition(    timestamps,    TimeSplit();    train = 0.7,    test = 0.3,)

Likewise, group identifiers can be split directly:

partition(    patient_ids,    GroupShuffleSplit();    train = 0.8,    test = 0.2,)

Data formats

data can be a matrix, a vector, a Tables.jl-compatible container such as a DataFrame, or any custom type implementing the MLUtils observation interface.

For matrices, DataSplits follows the Julia machine-learning convention:

columns are samples and rows are features

Tables such as DataFrames instead store observations in rows. For example:

X_rows = Matrix(iris[:, 1:4])X_matrix = permutedims(X_rows)(size(X_rows), size(X_matrix))

((150, 4), (4, 150))

The first matrix has 150 rows × 4 features, whereas the matrix passed to matrix-oriented DataSplits strategies has 4 features × 150 observations.

If your data store samples in rows, transpose or permute them before passing them to strategies that operate on feature matrices:

For Tables.jl-compatible inputs, rows are interpreted as observations. DataSplits performs the required conversion internally when a strategy needs a feature-by-observation matrix.

All returned indices refer to observations and are positive integers in 1:N.

For custom containers or arrays with non-standard indexing, prefer splitdata and splitview over indexing with the returned indices manually.

Reproducibility

Strategies that involve randomisation, such as RandomSplit, ShuffleSplit, and GroupShuffleSplit, accept an rng keyword.

Using the same seeded random-number generator state reproduces the same split:

split_a = partition(    X,    RandomSplit();    train = 0.8,    test = 0.2,    rng = Xoshiro(42),)split_b = partition(    X,    RandomSplit();    train = 0.8,    test = 0.2,    rng = Xoshiro(42),)split_c = partition(    X,    RandomSplit();    train = 0.8,    test = 0.2,    rng = Xoshiro(43),)(trainindices(split_a) == trainindices(split_b),trainindices(split_a) == trainindices(split_c))

(true, false)

Using an explicit random-number generator makes the source of randomness visible in scripts, examples, and reproducible analyses.

Accessing indices directly

When you need the indices themselves, use the public accessor functions rather than reading fields of the returned split object directly:

(    train = length(trainindices(threeway)),    validation = length(valindices(threeway)),    test = length(testindices(threeway)),)

(train = 90, validation = 30, test = 30)

Cross-validation results can be inspected with folds.

These accessors form part of the public API and should be preferred over relying on the internal representation of split objects.

Where to go next

The examples above cover the common DataSplits workflow:

1. Choose a splitting strategy,
2. Call partition,
3. Obtain data subsets or views,
4. Fit, select, and evaluate models using the appropriate cohorts.

For guidance on choosing an appropriate strategy, see Why Splitting Matters.

For the complete behaviour of partition, result types, and helper functions, continue to the Core API Reference.