Data Splitting
COBRA-style estimators separate base-estimator fitting data from aggregation/calibration data.
The current source represents these two roles as:
This separation is handled primarily by:
in:
and by the splitter implementations under:
The main splitters are:
Why COBRA uses two datasets
In the normal feature-based workflow, base estimators are first fitted on:
Those fitted estimators then generate predictions on:
The resulting prediction representation is paired with:
and used by the aggregation procedure.
Conceptually:
flowchart LR
X["Full training data"]
S["Data split"]
XK["X_k, y_k"]
XL["X_l, y_l"]
E["Fit base estimators"]
P["Predictions on X_l"]
A["COBRA aggregation / optimization"]
X --> S
S --> XK
S --> XL
XK --> E
E --> P
XL --> P
P --> A
XL --> AThis distinction is used by:
when they are fitted from ordinary feature matrices.
TrainingContext
The resolved datasets are stored in:
defined in:
Its fields are:
@dataclass(slots=True)
class TrainingContext:
X_k: np.ndarray | None
y_k: np.ndarray | None
X_l: np.ndarray
y_l: np.ndarray
as_predictions: bool
Meaning of the fields
| Field | Purpose |
|---|---|
X_k |
data used to fit base estimators |
y_k |
targets for base-estimator fitting |
X_l |
aggregation/calibration input |
y_l |
aggregation/calibration targets |
as_predictions |
whether the supplied matrix is already prediction space |
When:
the training side is intentionally absent:
because no base estimators need to be fitted.
Split indices
Concrete splitters return:
with:
The dataclass is:
@dataclass(slots=True)
class SplitIndices:
train_idx: np.ndarray
eval_idx: np.ndarray
fold_id: int | None = None
For ordinary train/calibration splitting:
remains None.
The splitter interface
All splitters inherit:
and are expected to implement:
returning a:
object.
The base abstraction describes splitters as index generators rather than data transformers.
They operate on sample indices and do not modify the feature matrix or target vector directly.
Splitter registry
Splitters are registered through:
The current built-in names are:
where:
are aliases for:
and:
creates:
Inspect available splitters
from kfc_procedure.cobra.core.splitters import (
SplitterFactory,
)
print(
SplitterFactory.available()
)
The current registry should contain:
after the splitter package has been imported.
Default COBRA split
The shared resolver uses:
by default.
If no explicit splitter is passed to:
the source creates:
SplitterFactory.create(
"split_overlap",
split_ratio=split_ratio,
overlap=overlap,
random_state=random_state,
)
Therefore the default automatic COBRA split is not
RandomHoldoutSplitter.
It is:
with:
unless different values are supplied.
OverlapSplitter
OverlapSplitter is registered as:
Its constructor is:
It can create disjoint or overlapping training/calibration index sets.
Parameter validation
The constructor requires:
Otherwise:
It also requires:
Otherwise:
Finally:
If not, the implementation first prints:
and then raises:
Overlap split geometry
For:
observations, the implementation computes:
and:
where:
Then:
So the region:
belongs to both subsets.
Visual model
0 n
|------------------------------------------------|
split_ratio = r
k1 k2
|-----------|
| overlap |
|-----------|
train:
|---------------------------|
evaluation:
|------------------------------|
The training subset ends at:
while the evaluation subset begins at:
No-overlap case
With:
the boundaries are:
Therefore:
and the partitions are disjoint.
This is the default behavior used by the COBRA estimators.
Example: 50/50 split
For:
the implementation gives approximately:
subject to integer truncation when the sample count does not divide exactly.
Example: overlap
For:
the boundaries are:
and:
So:
giving:
Approximate subset sizes
Ignoring integer truncation:
and:
The shared region is approximately:
Because the implementation uses Python int(), exact counts can differ by one
or more samples from these continuous expressions for small datasets.
split_ratio is not simply the final train fraction when overlap is used
With:
the final training set extends beyond:
by approximately:
Likewise, the evaluation set begins before the split-ratio boundary by the same amount.
So:
does not create a 50-sample training subset and a 50-sample evaluation subset with 20 shared samples.
The implementation creates approximately:
for n=100.
Shuffling
OverlapSplitter defaults to:
Before computing subsets, it creates:
and passes them through:
Random generator
When shuffling is enabled, the implementation uses:
Therefore:
controls the permutation produced by the overlap splitter.
Disable shuffling
You can create:
Then:
contains the first portion of the original row order and:
contains the remaining portion.
Warning
The high-level COBRA fit() methods do not currently expose a
shuffle argument for the default splitter.
Their automatic split path constructs the default overlap splitter with its own default:
Target values are not used by OverlapSplitter
The splitter's split() method receives:
but the implementation uses only:
to obtain the sample count.
It does not use:
when generating the partition.
Therefore the automatic overlap split is not stratified.
No group-aware splitting in the concrete implementation
BaseDataSplitter.split() includes an optional abstract:
parameter.
However, the current concrete:
implementations do not expose a groups argument in their own method
signatures.
So the current built-ins do not implement group-aware partitioning.
RandomHoldoutSplitter
The second built-in strategy is:
registered as:
Its constructor is:
Implementation
It creates:
and calls:
train_test_split(
indices,
test_size=self.calibration_size,
random_state=self.random_state,
shuffle=True,
)
The result is returned as:
Holdout partitions are disjoint
Unlike OverlapSplitter with:
the random holdout strategy produces disjoint subsets.
No observation is placed in both:
by train_test_split.
Holdout calibration size
The intended constructor domain is:
The source stores:
without an explicit manual interval check in __init__().
Validation of unsupported values is therefore left to:
when split() is called.
Holdout is also not stratified
Although split() receives y, it does not pass:
to train_test_split().
The actual call uses:
train_test_split(
indices,
test_size=self.calibration_size,
random_state=self.random_state,
shuffle=True,
)
Therefore classification label balance is not explicitly preserved by the current holdout implementation.
Direct splitter usage
Overlap splitter
from kfc_procedure.cobra.core.splitters import (
OverlapSplitter,
)
splitter = OverlapSplitter(
split_ratio=0.5,
overlap=0.1,
random_state=42,
)
indices = splitter.split(
X,
y,
)
print(
indices.train_idx
)
print(
indices.eval_idx
)
Holdout splitter
from kfc_procedure.cobra.core.splitters import (
RandomHoldoutSplitter,
)
splitter = RandomHoldoutSplitter(
calibration_size=0.4,
random_state=42,
)
indices = splitter.split(
X,
y,
)
Factory usage
Create an overlap splitter:
from kfc_procedure.cobra.core.splitters import (
SplitterFactory,
)
splitter = SplitterFactory.create(
"split_overlap",
split_ratio=0.5,
overlap=0.1,
random_state=42,
)
Create a holdout splitter:
The alias:
creates the same class.
resolve_training_context()
Most COBRA estimators use the shared:
function.
Its relevant signature is:
resolve_training_context(
X,
y,
*,
X_l=None,
y_l=None,
pred_features=None,
as_predictions=False,
splitter=None,
split_ratio=0.5,
overlap=0.0,
random_state=None,
)
The function supports several distinct data-resolution modes.
Mode 1 — precomputed prediction data
The first branch is:
if as_predictions:
return TrainingContext(
X_k=None,
y_k=None,
X_l=np.asarray(X),
y_l=np.asarray(y),
as_predictions=True,
)
So when:
no split is performed.
The entire supplied matrix becomes:
and the entire target vector becomes:
Example
produces conceptually:
The splitter arguments are bypassed in this mode.
Mode 2 — explicit calibration data
If both:
are supplied, the resolver does not create a split.
Instead:
become:
and the explicitly supplied data become:
Example
produces:
Both explicit arrays are required
Providing only:
or only:
raises:
This check occurs before automatic splitting.
Mode 3 — pred_features
The resolver also contains a special:
branch.
If present, it checks:
and returns:
TrainingContext(
X_k=np.asarray(X),
y_k=np.asarray(y),
X_l=pred_features,
y_l=np.asarray(y),
as_predictions=False,
)
No automatic train/calibration split occurs in this branch.
Important current MixCOBRA caveat
MixCOBRARegressor.fit() passes its:
argument into this resolver.
However, after resolution, the current MixCOBRA code still treats:
as an input to the fitted base estimators in the normal
as_predictions=False path.
Therefore the source does not currently use this resolver branch as a direct, clean precomputed-prediction path.
See:
for the source-level behavior.
Mode 4 — automatic splitting
If none of the earlier modes applies, the resolver performs an automatic split.
If:
it creates:
through:
SplitterFactory.create(
"split_overlap",
split_ratio=split_ratio,
overlap=overlap,
random_state=random_state,
)
Then:
and the arrays are sliced by those indices.
Returned arrays
The final context is:
TrainingContext(
X_k=np.asarray(X)[train_idx],
y_k=np.asarray(y)[train_idx],
X_l=np.asarray(X)[eval_idx],
y_l=np.asarray(y)[eval_idx],
as_predictions=False,
)
So all overlap behavior is reflected literally in the data arrays.
If a sample index occurs in both index sets, that observation occurs in both
X_k and X_l.
Input validation
At the beginning of:
the source calls:
from scikit-learn.
Explicit:
data are also validated with:
before they are placed in the TrainingContext.
GradientCOBRA splitting
The current GradientCOBRA.fit() signature contains:
It calls:
resolve_training_context(
X,
y,
X_l=X_l,
y_l=y_l,
as_predictions=as_predictions,
split_ratio=split_ratio,
overlap=overlap,
random_state=self.random_state,
)
So GradientCOBRA exposes:
through its public fit() method.
CombinedClassifier splitting
The current classifier signature is:
It passes these values to the same resolver.
The default:
is accepted by OverlapSplitter because Python False behaves numerically as
zero.
So the default classifier split is effectively:
MixCOBRA splitting
The current MixCOBRARegressor.fit() signature contains:
fit(
X,
y,
X_l=None,
y_l=None,
split_ratio=0.5,
overlap=0.0,
pred_features=None,
as_predictions=False,
)
It therefore exposes:
to the shared resolver.
The high-level COBRA estimators do not expose splitter=
Although:
accepts a custom:
object, the current public fit() signatures of:
do not include a:
argument.
Therefore normal estimator usage cannot currently select:
instead of:
through a public fit parameter.
Current API boundary
The splitter factory is implemented and usable directly, but the main
estimator fit() methods currently route automatic splitting through the
default overlap splitter.
Using another splitter inside those workflows requires source extension or
manually preparing X_k/y_k and X_l/y_l.
Using holdout behavior with an estimator
Because public COBRA fit() methods accept explicit calibration data, you can
manually split first.
from kfc_procedure.cobra.core.splitters import (
RandomHoldoutSplitter,
)
splitter = RandomHoldoutSplitter(
calibration_size=0.4,
random_state=42,
)
indices = splitter.split(
X,
y,
)
X_k = X[
indices.train_idx
]
y_k = y[
indices.train_idx
]
X_l = X[
indices.eval_idx
]
y_l = y[
indices.eval_idx
]
Then:
This uses the holdout split you constructed rather than the estimator's automatic overlap-split path.
Reproducibility
The default overlap splitter uses:
while RandomHoldoutSplitter delegates its seed to scikit-learn's:
So both splitters support reproducible partitions when a fixed
random_state is supplied.
COBRA estimator seed
The high-level estimators pass:
into resolve_training_context().
Therefore:
with automatic splitting uses the seed:
for the default overlap permutation.
The same applies to the other COBRA estimators that pass their
random_state into the resolver.
Reproduce an automatic split
To reproduce the default automatic split outside a COBRA estimator:
from kfc_procedure.cobra.core.splitters import (
OverlapSplitter,
)
splitter = OverlapSplitter(
split_ratio=0.5,
overlap=0.0,
random_state=42,
)
indices = splitter.split(
X,
y,
)
For the same sample ordering and seed, this follows the same splitter logic used by the resolver.
Inspect train/evaluation overlap
train = set(
indices.train_idx.tolist()
)
evaluation = set(
indices.eval_idx.tolist()
)
shared = train.intersection(
evaluation
)
print(
len(shared)
)
With:
expected:
With nonzero overlap, the value should be approximately:
subject to integer truncation.
Inspect split sizes
And:
For OverlapSplitter, the two subset sizes can sum to more than the total
dataset size because shared observations are counted twice.
Example split table
For:
the implementation computes:
and:
Therefore:
and:
This directly illustrates the source algorithm.
Automatic splitting is not cross-validation
The train/calibration splitter is distinct from the cross-validation machinery used later for COBRA hyperparameter optimization.
The initial split determines:
Then the COBRA estimator can create CV folds within the calibration side for bandwidth or parameter optimization.
So these are two different data-partitioning levels:
See:
for the second level.
Relationship to KFC Procedure splitting
The top-level KFC Procedure has its own internal split logic.
KFCProcedure.fit() uses scikit-learn:
train_test_split(
X,
y,
test_size=0.5,
random_state=self.random_state,
stratify=(
y
if self.task == "classification"
else None
),
)
This is separate from the COBRA splitter framework described on this page.
Therefore:
and:
should not be assumed to use identical logic.
Notably:
- KFC classification uses
stratify=y; - the current COBRA
OverlapSplitterdoes not stratify.
KFC C-Step and as_predictions=True
When COBRA estimators are wrapped inside the KFC C-Step, the wrappers fit them with:
on the already-created F-Step matrix.
In that mode:
returns immediately and performs no additional COBRA train/calibration split.
For example:
internally calls:
so:
is treated directly as the aggregation prediction space.
Avoiding an additional split
If you already prepared separate estimator-training and aggregation datasets, use:
instead of letting the COBRA estimator split the combined data again.
For example:
This bypasses the automatic OverlapSplitter.
Split-ratio examples
70/30 disjoint
Approximately:
subject to integer truncation.
50/50 disjoint
Approximately:
50/50 center with 20% overlap
Approximately:
This follows the source's symmetric overlap around the split-ratio boundary.
What overlap changes statistically
From the source alone, overlap changes only which sample indices are included in each subset.
The implementation does not attach special weights or labels to shared observations.
A shared observation simply occurs in both:
and:
Any statistical consequence follows from that reuse; the splitter itself adds no correction.
What the splitters do not do
The current built-in splitter implementations do not perform:
feature scaling
target transformation
stratification
group-aware splitting
time-series ordering constraints
cross-validation
class balancing
sample weighting
Their job is limited to constructing train/evaluation sample indices.
Custom splitter
The extension interface is:
A custom splitter should return:
Example:
import numpy as np
from kfc_procedure.cobra.core.splitters import (
BaseDataSplitter,
SplitterFactory,
)
from kfc_procedure.cobra.core.types import (
SplitIndices,
)
@SplitterFactory.register(
"first_half"
)
class FirstHalfSplitter(
BaseDataSplitter
):
def split(
self,
x,
y,
*,
groups=None,
):
n = len(x)
cut = n // 2
return SplitIndices(
train_idx=np.arange(
cut
),
eval_idx=np.arange(
cut,
n,
),
)
Then:
can resolve it directly.
Using a custom splitter with resolve_training_context
The resolver itself supports:
For example:
However, as noted earlier, the current high-level COBRA fit() methods do not
expose this parameter directly.
Debugging data splitting
Check the automatic split parameters
and record the values passed to:
during fit().
Reproduce indices
splitter = OverlapSplitter(
split_ratio=0.5,
overlap=0.0,
random_state=model.random_state,
)
indices = splitter.split(
X,
y,
)
Check overlap
Check all samples are covered
For OverlapSplitter:
Because of the construction:
all shuffled indices are covered.
Check class proportions manually
Because the built-in COBRA splitters are not stratified:
import numpy as np
print(
np.unique(
y[
indices.train_idx
],
return_counts=True,
)
)
print(
np.unique(
y[
indices.eval_idx
],
return_counts=True,
)
)
This is useful for classification diagnostics.
Current source caveats
| Area | Current behavior |
|---|---|
| default automatic splitter | split_overlap |
default split_ratio |
0.5 |
| default overlap | 0.0 / False |
| overlap shuffling | True |
| overlap RNG | np.random.default_rng |
| holdout backend | sklearn train_test_split |
| stratification | not implemented by built-in COBRA splitters |
| group-aware split | not implemented by concrete built-ins |
public estimator splitter= argument |
not exposed |
explicit X_l, y_l |
supported |
as_predictions=True |
bypasses splitting |
pred_features branch |
present; current MixCOBRA integration has caveat |
| KFC top-level split | separate implementation |
Quick reference
| Mode | X_k/y_k |
X_l/y_l |
Automatic splitter used? |
|---|---|---|---|
as_predictions=True |
None |
supplied X/y |
No |
explicit X_l/y_l |
supplied X/y |
supplied calibration data | No |
pred_features |
supplied X/y |
pred_features/y |
No |
| automatic | splitter training rows | splitter evaluation rows | Yes |
Mental model
COBRA splitting decides which observations train the base estimators and which observations define the aggregation problem.
The current automatic path uses OverlapSplitter, which can either create
disjoint subsets or deliberately reuse a controlled region in both roles.