Cross-validation
Cross-validation in the COBRA subsystem is used primarily to evaluate aggregation hyperparameters such as:
on the aggregation/calibration dataset.
It is separate from the earlier train/calibration split described in Data Splitting.
The current source provides three registered cross-validation strategies:
implemented under:
and resolved through:
Where cross-validation sits
COBRA training has two different partitioning levels.
First, the data can be separated into:
Then cross-validation operates inside the calibration side while tuning COBRA hyperparameters.
flowchart TD
D["Training data"]
S["Train / calibration split"]
K["X_k, y_k"]
L["X_l, y_l"]
E["Fit base estimators"]
P["Build calibration prediction space"]
CV["Cross-validation folds on calibration data"]
O["Optimize bandwidth / alpha / beta"]
D --> S
S --> K
S --> L
K --> E
E --> P
L --> P
P --> CV
L --> CV
CV --> OSo:
and:
are different operations in the current architecture.
Why COBRA cross-validation is used
The source-level objective is not to refit the original base estimators for every fold.
Instead, COBRA first constructs the calibration representation and its pairwise distance matrix.
For a candidate hyperparameter setting, it:
- adapts the stored distance matrix;
- applies the kernel;
- extracts validation-to-training kernel blocks;
- aggregates the calibration targets;
- evaluates a loss;
- averages the fold losses.
Conceptually:
BaseCrossValidator
All built-in cross-validation classes inherit:
The abstract interface defines:
and:
The split() method yields:
objects.
SplitIndices
Each fold is represented by:
The fields are:
train_idx
indices used as the fold's aggregation-training subset
eval_idx
indices used as the fold's validation subset
fold_id
integer fold identifier
For the built-in cross-validation classes:
starts at:
and increases sequentially.
CVFactory
Cross-validation implementations are registered with:
which inherits the package's common:
Create a CV strategy with:
from kfc_procedure.cobra.core.cv import (
CVFactory,
)
cv = CVFactory.create(
"kfold",
n_splits=5,
random_state=42,
)
Inspect available strategies
The current source registers:
where:
are aliases for the same class.
KFoldCV
The default cross-validator used by the current main COBRA estimators is:
registered as:
Its constructor is:
K-fold behavior
The implementation:
- creates all sample indices;
- optionally shuffles them;
- distributes them round-robin across
n_splitslists; - uses each list once as validation data;
- concatenates the other lists as training data.
The source does not delegate to scikit-learn's KFold.
It implements its own fold assignment.
K-fold index construction
First:
Then:
If:
is true:
Round-robin fold assignment
The source constructs:
and then:
So observations are distributed cyclically across folds.
This differs in implementation detail from simply cutting a shuffled index array into contiguous chunks.
Example K-fold assignment
Suppose:
The current algorithm produces:
because assignment is based on:
K-fold train and validation indices
For each fold:
and:
Then the training indices are sorted:
The validation indices are not explicitly sorted after shuffling.
Consequence of sorting only training indices
With:
the validation array retains the order produced by the shuffled round-robin assignment.
The training array is sorted back into ascending original-index order.
This does not change which observations belong to each set, but it is an exact behavior of the current implementation.
K-fold reproducibility
KFoldCV uses:
rather than:
With the same input order, n_splits, and seed, the fold assignment is
reproducible.
Disable K-fold shuffling
Then the original row order is distributed round-robin among folds.
This does not produce contiguous validation blocks.
For temporal data, use TimeSeriesCV rather than assuming
KFoldCV(shuffle=False) creates chronological windows.
get_n_splits()
KFoldCV defines:
and returns:
The optional x and y parameters are ignored.
K-fold direct example
from kfc_procedure.cobra.core.cv import (
KFoldCV,
)
cv = KFoldCV(
n_splits=5,
shuffle=True,
random_state=42,
)
for fold in cv.split(
X,
y,
):
print(
fold.fold_id,
fold.train_idx,
fold.eval_idx,
)
StratifiedKFoldCV
The source also provides:
registered as:
Its purpose is to distribute each class across folds.
The constructor is:
There is no separate:
parameter.
The class always shuffles class-specific indices internally.
Stratified construction
The implementation converts:
with:
then creates:
Next it builds a dictionary:
mapping each observed label to the list of row indices belonging to that class.
Per-class split
For each class:
Then part i for every class is appended to fold i.
So every fold receives approximately:
observations from class \(c\), subject to integer division.
Stratified example
Suppose:
Then each fold receives approximately:
after the within-class shuffling.
This is the source mechanism behind its class-distribution preservation.
Stratified train and validation indices
After building all folds:
and:
Unlike KFoldCV, the current stratified implementation does not sort the
training indices before yielding them.
Stratified reproducibility
StratifiedKFoldCV uses:
which differs from the RandomState API used by KFoldCV.
A fixed seed makes the source implementation reproducible, but the exact
permutation sequence should not be assumed to match KFoldCV or scikit-learn
splitters using a different RNG API.
TimeSeriesCV
Temporal cross-validation is implemented by:
registered under:
Its constructor is:
It does not shuffle data and does not accept random_state.
Time-series design
The source uses expanding training windows.
For each split:
Conceptually:
fold 0
train: [------]
valid: [--]
fold 1
train: [----------]
valid: [--]
fold 2
train: [--------------]
valid: [--]
Time-series step size
The implementation computes:
So if:
is provided and truthy, it becomes the step size.
Otherwise:
Time-series fold boundaries
For fold:
the source computes:
Training indices are:
and validation indices are:
Time-series example
For:
the default step is:
The folds are:
fold 0:
train 0:10
valid 10:20
fold 1:
train 0:20
valid 20:30
fold 2:
train 0:30
valid 30:40
fold 3:
train 0:40
valid 40:50
fold 4:
train 0:50
valid 50:60
Explicit time-series test_size
For:
the source uses:
for both the growth increment and validation-window width.
For example:
subject to clipping at n.
Current validation limitations
The built-in CV classes do not perform comprehensive constructor validation.
For example, the source does not explicitly reject:
when the objects are created.
This can produce empty folds or lower-level NumPy errors later.
KFold with too many folds
If:
some round-robin folds remain empty.
Those empty validation folds are still yielded.
The source contains no explicit minimum fold-size validation.
KFold with one split
With:
the training-set construction attempts to concatenate all folds except the single validation fold.
That produces an empty list for:
which can raise a NumPy error.
The current source does not intercept this case with a custom message.
Stratified folds with rare classes
If a class has fewer samples than:
then:
creates empty class-specific parts for some folds.
A fold may therefore lack that class.
The source does not reject this situation.
So "stratified" should be understood as distributing available class examples as evenly as the implementation can, not guaranteeing every class appears in every fold.
Time-series empty windows
If:
is too small relative to:
then:
can become:
That creates empty train and validation arrays.
Likewise, a large explicit test_size can cause later validation windows to be
empty once:
The current class does not validate these conditions.
COBRA defaults to K-fold
Although three CV strategies are registered, the current main COBRA estimators hard-code:
inside component resolution.
This is true for:
Important API consequence
The public constructors expose:
but do not expose:
as constructor arguments.
Therefore, in the current high-level APIs, you can change the number of folds but not select:
through ordinary estimator configuration.
Current API boundary
StratifiedKFoldCV and TimeSeriesCV are registered and can be used
directly through CVFactory, but the current primary COBRA estimators
construct kfold internally.
Using a different CV strategy in those estimators requires extending or modifying the source.
n_cv
The public parameter controlling fold count is:
with current default:
for:
During component resolution it becomes:
Fitted CV object
After component resolution, the cross-validator is stored as:
For the current main estimators:
should be:
unless source code has been changed.
Stored folds
The concrete folds used during optimization are materialized as a list:
For GradientCOBRA:
For MixCOBRA:
For CombinedClassifier:
GradientCOBRA's unusual y argument
GradientCOBRA calls:
rather than:
as the second argument.
With the current default:
this makes no difference because KFoldCV.split() ignores y.
Current source detail
If GradientCOBRA were changed to use a cross-validator whose fold
construction depends on y, this call would matter because the supplied
second argument is the normalized prediction-space matrix rather than the
regression target vector.
The current high-level source avoids that issue by always resolving
kfold.
CombinedClassifier does not use stratified CV by default
Although:
is available in the registry, CombinedClassifier currently creates:
rather than:
for its bandwidth optimization.
Therefore classification CV inside CombinedClassifier does not explicitly
preserve class proportions.
This is separate from the package's top-level KFC split, which has its own classification stratification logic.
How GradientCOBRA uses folds
GradientCOBRA first stores a square calibration distance matrix:
For a candidate bandwidth:
it sets:
then computes:
Fold-specific kernel block
For each stored fold:
the validation-to-training kernel block is:
If:
then:
Fold prediction in GradientCOBRA
The fold training targets are:
and validation targets:
Predictions are produced by:
Then the fold error is:
Mean GradientCOBRA CV error
Errors are appended:
and the objective returns:
So optimization minimizes the mean loss across folds.
GradientCOBRA CV fallback
Inside cross-validation, GradientCOBRA explicitly passes:
to the regression aggregator.
This differs from final prediction, where the source uses:
for the fitted model.
Therefore candidate bandwidths are evaluated under a zero-weight fallback of:
during CV.
MixCOBRA cross-validation
MixCOBRA uses the same basic fold structure but has two objective forms.
With:
it evaluates one:
parameter.
With:
it evaluates:
jointly.
MixCOBRA one-parameter CV
The objective:
sets:
then transforms:
and applies the kernel.
The same fold extraction pattern is used:
MixCOBRA two-parameter CV
The objective:
receives:
and updates:
The two stored distance matrices are fused by:
before kernel application.
Each fold is then evaluated with the same target aggregation process.
MixCOBRA CV fallback
Like GradientCOBRA, MixCOBRA uses:
inside its regression cross-validation objectives.
The fold losses are averaged with:
CombinedClassifier cross-validation
CombinedClassifier also starts from a square calibration distance matrix.
For a candidate:
it computes a kernel matrix and extracts:
Fold-level classification prediction
For each validation row:
the source checks:
if np.sum(w) <= 0:
pred = self.global_majority_class_
else:
pred = self.aggregator_.aggregate(
y_train,
w,
)
Predictions are collected into an array and compared with:
using:
CombinedClassifier fallback during CV
If a validation query receives a non-positive kernel-weight sum, the fallback is:
This differs from the regression CV fallback of:
CombinedClassifier loss default
The current constructor default is:
even though the estimator is a classifier.
Therefore, unless changed, the CV objective applies the registered MSE loss to the numeric class predictions and targets.
This is a current source default, not a generic classification recommendation.
For supported alternatives, see:
CV and optimization
Cross-validation itself does not choose the final hyperparameter.
It defines the objective function consumed by the optimizer.
For example:
candidate bandwidth
↓
mean cross-validation loss
↓
optimizer compares candidates
↓
selected bandwidth_
The optimizer layer is documented separately in:
Grid search and folds
With the default grid optimization, every candidate bandwidth is evaluated by calling the CV objective.
For GradientCOBRA and CombinedClassifier, the default candidate list is:
when no custom bandwidth list is supplied.
So with:
the CV objective conceptually performs:
although the expensive pairwise distance matrix is precomputed once before those evaluations.
Why the distance matrix is precomputed
The source does not recompute prediction-space distances for every fold and every candidate parameter.
Instead:
or the MixCOBRA distance matrices are created once.
Each candidate parameter only changes the adapter transformation and kernel weights.
This makes CV operate on:
rather than repeatedly reconstructing the full distance computation.
Inspect the fitted folds
After fitting:
Normally:
with the default:
Inspect fold sizes
for fold in model.cv_folds_:
print(
"fold:",
fold.fold_id,
"train:",
len(
fold.train_idx
),
"validation:",
len(
fold.eval_idx
),
)
Inspect exact indices
These indices refer to rows of the calibration arrays:
not to the original unsplit full dataset.
Map fold indices to calibration targets
fold = model.cv_folds_[0]
y_train_fold = model.y_l_[
fold.train_idx
]
y_val_fold = model.y_l_[
fold.eval_idx
]
This reproduces the target subsets used by the CV objective.
Reproduce the default K-fold object
For a fitted model configured with:
the equivalent current CV constructor is:
from kfc_procedure.cobra.core.cv import (
KFoldCV,
)
cv = KFoldCV(
n_splits=5,
shuffle=True,
random_state=42,
)
Then:
matches the same K-fold logic for MixCOBRA and CombinedClassifier.
For GradientCOBRA, its source call supplies:
as the second argument, but KFoldCV ignores that value.
Using stratified CV directly
Although current high-level COBRA estimators do not expose it as a constructor choice, the CV class itself can be used directly.
from kfc_procedure.cobra.core.cv import (
StratifiedKFoldCV,
)
cv = StratifiedKFoldCV(
n_splits=5,
random_state=42,
)
for fold in cv.split(
X,
y,
):
...
This can be useful when extending the estimator source or testing fold construction independently.
Inspect class balance
import numpy as np
for fold in cv.split(
X,
y,
):
labels, counts = np.unique(
y[
fold.eval_idx
],
return_counts=True,
)
print(
fold.fold_id,
dict(
zip(
labels,
counts,
)
),
)
This shows the approximate class distribution achieved by the source implementation.
Using time-series CV directly
from kfc_procedure.cobra.core.cv import (
TimeSeriesCV,
)
cv = TimeSeriesCV(
n_splits=5,
)
for fold in cv.split(
X,
y,
):
print(
fold.train_idx,
fold.eval_idx,
)
The input order is treated as chronological order.
The class itself does not inspect timestamps or sort rows by a time column.
Important time-order requirement
If temporal rows are not already ordered correctly before calling:
the source does not reorder them.
Chronology is assumed to be encoded by row order.
Custom cross-validator
A custom CV strategy can subclass:
and register with:
For example:
import numpy as np
from kfc_procedure.cobra.core.cv import (
BaseCrossValidator,
CVFactory,
)
from kfc_procedure.cobra.core.types import (
SplitIndices,
)
@CVFactory.register(
"two_fold_fixed"
)
class TwoFoldFixed(
BaseCrossValidator
):
def split(
self,
x,
y,
*,
groups=None,
):
n = len(x)
cut = n // 2
first = np.arange(
cut
)
second = np.arange(
cut,
n,
)
yield SplitIndices(
train_idx=first,
eval_idx=second,
fold_id=0,
)
yield SplitIndices(
train_idx=second,
eval_idx=first,
fold_id=1,
)
def get_n_splits(
self,
):
return 2
Current custom-CV integration limitation
Registering a custom strategy makes it available through:
but the current:
constructors still hard-code:
inside their component resolution.
Therefore registration alone does not make a custom CV strategy selectable via a high-level estimator argument.
The estimator source would need a configurable:
path to expose that extension fully.
Groups parameter
BaseCrossValidator.split() includes:
in its abstract interface.
However:
do not implement group-based splitting logic.
Their concrete split() signatures also do not all preserve the abstract
keyword-only groups parameter.
So group-aware cross-validation is not implemented by the current built-ins.
Data splitting vs cross-validation
A useful comparison:
| Layer | Purpose | Current default |
|---|---|---|
| data splitter | create X_k/y_k and X_l/y_l |
OverlapSplitter |
| cross-validator | create optimization folds within calibration data | KFoldCV |
| optimizer | choose bandwidth / alpha / beta from CV objective | grid |
These layers should not be treated as interchangeable.
KFC C-Step and CV
When KFC wraps:
it passes the F-Step matrix with:
This bypasses COBRA's first-level train/calibration split.
However, the wrapped estimator still resolves its internal:
and builds:
over the supplied KFC C-Step training matrix.
So:
bypasses data splitting but does not bypass COBRA hyperparameter cross-validation.
KFC example: GradientCOBRA
KFC's C-Step wrapper runs conceptually:
Inside GradientCOBRA:
P_l
becomes calibration prediction space
KFoldCV
splits rows of P_l into CV folds
bandwidth optimizer
minimizes mean fold loss
So the C-Step aggregation data is itself cross-validated for bandwidth selection.
Classification KFC example
For:
the KFC F-Step class-prediction matrix becomes:
inside CombinedClassifier.
Then ordinary K-fold CV is used to optimize the classifier's bandwidth.
The current wrapper does not switch to StratifiedKFoldCV.
Reproducibility
For the main COBRA estimators, the same:
is passed to:
Therefore changing the estimator's random_state can change:
in addition to any other random components of the model.
Cross-validation does not refit base estimators
The current CV objective works from:
It does not call:
inside every fold.
That means the fold "training" indices refer to the aggregation stage, not to freshly refitted base learners.
This is important when interpreting the CV design.
No out-of-fold base-prediction construction inside CV
Base estimators are fitted before COBRA hyperparameter cross-validation.
The source does not use each cv_folds_ split to regenerate base-estimator
predictions out of fold.
Instead, the already-constructed calibration prediction-space geometry is partitioned for aggregation evaluation.
Debugging cross-validation
Inspect the CV object
Inspect the number of configured folds
Inspect materialized folds
Check validation coverage
For K-fold:
import numpy as np
all_validation = np.concatenate([
fold.eval_idx
for fold
in model.cv_folds_
])
print(
np.sort(
all_validation
)
)
With a normal valid K-fold configuration, every calibration row should occur once in the concatenated validation indices.
Check for empty folds
for fold in model.cv_folds_:
if len(
fold.eval_idx
) == 0:
print(
"empty validation fold:",
fold.fold_id,
)
This is especially useful if:
is large relative to the calibration sample count.
Check fold loss manually
For GradientCOBRA, one candidate bandwidth can be evaluated directly:
This reuses:
and the already-fitted component state.
Current implementation summary
| CV | Registry | Default folds | Shuffle | RNG | Main behavior |
|---|---|---|---|---|---|
KFoldCV |
kfold |
5 | Yes | RandomState |
shuffled round-robin folds |
StratifiedKFoldCV |
stratified_kfold |
5 | internally | default_rng |
split each class with array_split |
TimeSeriesCV |
time_series, tscv |
5 | No | none | expanding chronological windows |
Current high-level estimator behavior
| Estimator | CV used internally | Public fold parameter | Public CV strategy parameter |
|---|---|---|---|
GradientCOBRA |
kfold |
n_cv |
No |
MixCOBRARegressor |
kfold |
n_cv |
No |
CombinedClassifier |
kfold |
n_cv |
No |
Important source caveats
| Area | Current behavior |
|---|---|
| CV default | custom KFoldCV, not sklearn KFold |
| fold assignment | round-robin after optional shuffle |
n_splits validation |
not explicitly implemented |
| classification default CV | ordinary K-fold, not stratified |
| time-series order | assumes rows already chronological |
| group-aware CV | not implemented |
| custom CV registry | supported |
| high-level custom CV selection | not exposed |
| CV operates on | calibration/aggregation rows |
| base estimators refitted per fold | No |
| regression CV zero-weight fallback | 0.0 |
| classification CV zero-weight fallback | global majority class |
Quick reference
| Question | Current source answer |
|---|---|
| What is the default CV? | KFoldCV |
| How many folds by default? | 5 |
| Is K-fold shuffled? | Yes |
| Which seed is used? | estimator random_state |
| Is classification CV stratified? | No |
| Is stratified CV implemented? | Yes, but not selected by main estimators |
| Is time-series CV implemented? | Yes, but not selected by main estimators |
Can cv= be passed to GradientCOBRA? |
No |
| Are base estimators refitted each fold? | No |
| What does CV optimize? | COBRA aggregation hyperparameters |
| Where are folds stored? | cv_folds_ |
Mental model
COBRA cross-validation does not rebuild the entire learning pipeline for every fold. It reuses the calibration geometry and asks which aggregation hyperparameters predict held-out calibration rows best.
The current high-level estimators all use the package's custom shuffled
KFoldCV, while stratified and temporal strategies exist as registered
components that are not yet exposed through the main estimator constructors.