Custom Aggregation Data
The COBRA estimators in kfc_procedure can use a user-supplied aggregation
or calibration dataset instead of automatically splitting one dataset into
training and calibration subsets.
The relevant fit arguments are:
and they are supported by:
through the shared:
helper.
This advanced mode lets you decide exactly which observations are used for:
and which observations are used for:
Core idea
In normal COBRA fitting, the package can split:
into:
for fitting base estimators and:
for constructing the prediction-space aggregation problem.
With explicit aggregation data, you provide those two roles yourself.
Conceptually:
X_k, y_k
↓
fit base estimators
↓
X_l
↓
base-estimator predictions
↓
prediction-space calibration matrix
↓
distance / kernel / aggregation optimization
↑
y_l
Explicit aggregation mode
The shared resolver uses explicit mode when both:
and:
are supplied.
The source branch is:
if X_l is not None and y_l is not None:
X_l, y_l = check_X_y(
X_l,
y_l,
)
return TrainingContext(
X_k=np.asarray(X),
y_k=np.asarray(y),
X_l=np.asarray(X_l),
y_l=np.asarray(y_l),
as_predictions=False,
)
Therefore the main X and y arguments become:
while your explicit aggregation arrays become:
Meaning of the four arrays
| Array | Current source role |
|---|---|
X |
base-estimator training features |
y |
base-estimator training targets |
X_l |
aggregation/calibration features |
y_l |
aggregation/calibration targets |
The resulting internal context is:
for the ordinary feature-based path.
No automatic first-level split
When explicit:
are supplied, the resolver returns before the automatic splitter branch.
Therefore:
do not generate another train/calibration split.
Your supplied arrays define the split directly.
Both arrays are required
The resolver checks:
if (
X_l is None
) != (
y_l is None
):
raise ValueError(
"Both 'X_l' and 'y_l' must be provided together."
)
So these are invalid:
and:
Both calibration features and calibration targets must be supplied together.
Input validation
The primary training data are first validated with:
The explicit aggregation data are separately validated with:
So each pair must have internally matching sample counts.
Separate sample counts are allowed
The source does not require:
because the two datasets serve different roles.
For example:
is structurally valid.
Feature count must still be compatible
Although the resolver validates each pair separately, it does not explicitly check that:
before returning the context.
In ordinary feature mode, the fitted base estimators are trained on:
and later asked to predict:
Therefore the feature dimensions must be compatible with those estimators.
A mismatch normally fails later through the underlying estimator's
predict() validation.
Why custom aggregation data can be useful
The source supports explicit aggregation data without imposing a particular reason for using it.
Examples include:
a dedicated calibration dataset
a chronological holdout
a manually stratified split
a group-aware split prepared outside the package
a split produced by another library
a reproducible experiment with fixed train/calibration rows
The package accepts the supplied arrays and skips its own automatic first-level splitter.
GradientCOBRA
GradientCOBRA.fit() exposes:
So explicit aggregation data can be supplied directly.
GradientCOBRA example
from kfc_procedure.cobra import (
GradientCOBRA,
)
model = GradientCOBRA(
random_state=42,
)
model.fit(
X_train,
y_train,
X_l=X_cal,
y_l=y_cal,
)
The internal context becomes:
GradientCOBRA training flow
With explicit aggregation data, the source does:
then:
So the base estimators are fitted on:
and prediction-space coordinates are generated on:
GradientCOBRA calibration targets
The source sets:
So the final zero-weight fallback is based on the explicit calibration targets:
not on the base-estimator training targets:
GradientCOBRA cross-validation
After building the calibration prediction space, GradientCOBRA creates:
These CV indices refer only to rows of:
and the normalized prediction-space matrix built from those calibration rows.
So the explicit aggregation dataset becomes the population used for COBRA hyperparameter cross-validation.
GradientCOBRA normalization detail
The current source computes:
self.normalize_constant_ = (
compute_normalization_constant(
y=y,
norm_constant=self.norm_constant,
scale_factor=30.0,
M=prediction_space.shape[1],
)
)
Notice that it passes the original fit argument:
rather than:
Consequence with explicit aggregation data
If you call:
then:
prediction space
is generated from X_cal
global_mean_
is computed from y_cal
normalization constant
is computed from y_train.
Current source detail
GradientCOBRA's prediction-space normalization scalar is derived from the
main y argument, even when explicit y_l calibration targets are
supplied.
This page documents the current implementation rather than assuming the
normalization should instead use y_l.
GradientCOBRA full example
from sklearn.model_selection import (
train_test_split,
)
from kfc_procedure.cobra import (
GradientCOBRA,
)
X_train, X_cal, y_train, y_cal = (
train_test_split(
X,
y,
test_size=0.25,
random_state=42,
)
)
model = GradientCOBRA(
random_state=42,
)
model.fit(
X_train,
y_train,
X_l=X_cal,
y_l=y_cal,
)
y_pred = model.predict(
X_test
)
The external train_test_split() defines the first-level separation.
GradientCOBRA does not split those arrays again before fitting base estimators and constructing calibration prediction space.
CombinedClassifier
CombinedClassifier.fit() exposes:
Explicit calibration data follow the same resolver path.
CombinedClassifier example
from kfc_procedure.cobra import (
CombinedClassifier,
)
classifier = CombinedClassifier(
random_state=42,
)
classifier.fit(
X_train,
y_train,
X_l=X_cal,
y_l=y_cal,
)
CombinedClassifier estimator fitting
In ordinary feature mode:
self.classes_ = np.unique(
self.y_k_
)
self.estimators_ = self._fit_estimators(
self.X_k_,
self.y_k_,
)
self.pred_l_ = self._load_predictions(
self.X_l_
)
Therefore:
CombinedClassifier classes
With explicit calibration data, classes_ are derived from:
because:
So:
rather than:
Global majority class
The fallback class is derived separately from:
using:
classes, counts = np.unique(
self.y_l_,
return_counts=True,
)
self.global_majority_class_ = (
classes[
np.argmax(
counts
)
]
)
Therefore:
Missing calibration classes
This distinction matters if:
contains a class that is absent from:
or vice versa.
The source does not add a dedicated validation that both target arrays contain exactly the same class set.
A user-supplied explicit classification split should therefore preserve the class coverage needed by the experiment.
CombinedClassifier cross-validation
The classifier creates folds with:
So bandwidth optimization occurs entirely within the explicit calibration dataset.
The current high-level implementation still uses its ordinary internal
KFoldCV; supplying custom aggregation data does not change the CV strategy.
CombinedClassifier default split is bypassed
When X_l and y_l are supplied, these arguments become irrelevant to the
first-level split:
The explicit arrays take precedence.
Classification example with external stratification
The built-in COBRA automatic split is not stratified.
If you want a stratified first-level split, prepare it externally.
from sklearn.model_selection import (
train_test_split,
)
X_train, X_cal, y_train, y_cal = (
train_test_split(
X,
y,
test_size=0.3,
stratify=y,
random_state=42,
)
)
Then:
This uses your stratified split instead of the package's automatic
OverlapSplitter.
MixCOBRARegressor
MixCOBRARegressor.fit() exposes:
fit(
X,
y,
X_l=None,
y_l=None,
split_ratio=0.5,
overlap=0.0,
pred_features=None,
as_predictions=False,
)
Explicit calibration data work through the same resolver.
MixCOBRA example
from kfc_procedure.cobra import (
MixCOBRARegressor,
)
model = MixCOBRARegressor(
random_state=42,
)
model.fit(
X_train,
y_train,
X_l=X_cal,
y_l=y_cal,
)
MixCOBRA base estimators
With:
the source runs:
then:
So the role split is:
MixCOBRA input-space calibration data
MixCOBRA also retains:
because it compares both:
The explicit X_cal therefore serves two roles:
MixCOBRA normalization detail
The current source computes:
self.normalize_constant_x_ = (
compute_normalization_constant(
X,
norm_constant=self.norm_constant_x,
scale_factor=5.0,
M=prediction_space.shape[1],
)
)
and:
self.normalize_constant_y_ = (
compute_normalization_constant(
y,
norm_constant=self.norm_constant_y,
scale_factor=50.0,
M=prediction_space.shape[1],
)
)
Consequence with explicit aggregation data
For:
the source computes:
but applies them to:
Current source geometry
The stored arrays become:
and:
This is the exact current implementation.
MixCOBRA calibration target fallback
Like GradientCOBRA:
So the final regression fallback comes from:
rather than:
MixCOBRA cross-validation
The source creates folds over:
so:
are optimized on the explicit aggregation/calibration dataset.
Automatic splitting vs explicit aggregation
Compare:
Automatic
The package chooses:
Explicit
You choose those arrays.
The second form bypasses the package's automatic first-level split.
Explicit aggregation vs precomputed predictions
These are different modes.
Explicit aggregation features
means:
fit internal estimators on X_train
predict X_cal internally
build prediction space from those predictions.
Precomputed prediction mode
means:
See:
Explicit aggregation and as_predictions=True
Because the resolver checks:
before it checks:
the precomputed branch takes precedence.
For example:
causes the resolver to return:
and the explicit:
arguments are not used by the resolver.
Branch precedence
Do not combine as_predictions=True with explicit X_l/y_l expecting
both modes to be active.
In the current resolver, precomputed-prediction mode returns first.
Explicit aggregation and pred_features
The resolver checks explicit:
before:
So if both are provided while:
the explicit aggregation branch wins and:
is ignored by the resolver.
Resolver precedence
The current order is:
1. validate X/y
2. if as_predictions:
prediction mode
3. validate X_l/y_l pair
4. if X_l and y_l:
explicit aggregation mode
5. if pred_features:
pred_features mode
6. otherwise:
automatic split mode
This order is important when advanced arguments are combined.
Explicit aggregation does not disable COBRA CV
Supplying:
bypasses only the first-level automatic train/calibration split.
The estimator still performs:
calibration-space construction
distance computation
cross-validation folds
hyperparameter optimization.
So the explicit calibration set is itself partitioned internally during COBRA hyperparameter tuning.
Use a dedicated calibration dataset
A common structure is:
training set
-> fit base estimators
calibration set
-> fit COBRA aggregation geometry / tune aggregation parameters
test set
-> final evaluation
The package supports the first two roles directly through:
while the test set remains outside fit() and is passed later to:
Three-way split example
from sklearn.model_selection import (
train_test_split,
)
X_train_cal, X_test, y_train_cal, y_test = (
train_test_split(
X,
y,
test_size=0.2,
random_state=42,
)
)
X_train, X_cal, y_train, y_cal = (
train_test_split(
X_train_cal,
y_train_cal,
test_size=0.25,
random_state=42,
)
)
This gives approximately:
Then:
Chronological aggregation data
For time-ordered data, you can prepare an earlier training block and a later calibration block yourself.
X_train = X[:train_end]
y_train = y[:train_end]
X_cal = X[
train_end:cal_end
]
y_cal = y[
train_end:cal_end
]
Then:
The COBRA estimator does not reshuffle this first-level split because explicit aggregation data bypass the default splitter.
Internal CV still matters
The current main COBRA estimators still build their normal internal:
over the calibration dataset.
Therefore using a chronological explicit aggregation set does not make the internal hyperparameter CV time-series-aware.
The current high-level estimators do not expose a cv= argument.
See:
for that limitation.
Group-aware external splitting
The current built-in splitter framework does not implement group-aware partitioning.
You can prepare a group-aware split externally and pass the resulting arrays as explicit aggregation data.
For example, use an external splitter to produce:
then:
The estimator uses those rows directly.
Class-balanced calibration data
For classification, explicit data can be useful when you need the first-level calibration set to preserve class coverage.
The current CombinedClassifier automatic overlap splitter is not stratified.
Preparing:
externally gives you control over that first-level class distribution.
Class coverage checks
A practical source-aware diagnostic is:
import numpy as np
print(
np.unique(
y_train,
return_counts=True,
)
)
print(
np.unique(
y_cal,
return_counts=True,
)
)
This matters because:
and:
are currently derived from different target arrays in feature-based CombinedClassifier mode.
Inspect resolved data after fitting
GradientCOBRA:
MixCOBRA:
CombinedClassifier:
Verify your explicit arrays were used
For NumPy arrays:
import numpy as np
print(
np.array_equal(
model.X_k_,
X_train,
)
)
print(
np.array_equal(
model.X_l_,
X_cal,
)
)
The resolver converts values with:
so object identity is not the important check; compare values and shapes.
Inspect calibration prediction space
GradientCOBRA:
CombinedClassifier:
MixCOBRA:
Inspect calibration CV folds
Those indices are relative to:
not to the original full dataset.
Mapping CV rows back to your calibration set
fold = model.cv_folds_[0]
X_cal_train_fold = model.X_l_[
fold.train_idx
]
X_cal_val_fold = model.X_l_[
fold.eval_idx
]
This lets you inspect exactly which rows in your custom calibration set are used for one internal optimization fold.
Random state and explicit aggregation data
Supplying X_l/y_l removes randomness from the first-level COBRA split because
that split no longer occurs.
However, random_state can still affect:
base estimators
internal KFoldCV shuffling
optimizer-related components
other estimator-specific randomness.
So an explicit aggregation set does not make the entire model deterministic by itself.
split_ratio is ignored for explicit data
This call:
still uses your supplied:
The split_ratio value is never reached by the resolver's automatic-split
branch.
overlap is ignored for explicit data
Likewise:
does not cause rows to be copied between your explicit arrays.
If you want overlap between training and calibration data, you must construct
that overlap yourself before calling fit().
Intentional overlap
The source does not prevent the same observation from appearing in both:
and:
For example:
is accepted structurally.
The package does not track row identity and does not warn about reused observations.
Any statistical consequences of that overlap are the caller's responsibility.
No disjointness validation
The resolver validates shape consistency but does not test:
So "custom aggregation data" means the package trusts the split you provide.
Calibration size
The source does not impose a package-specific minimum calibration sample count before returning the explicit context.
However, later components can require enough rows for:
For example, n_cv=5 with a very small calibration set can produce empty or
degenerate folds in the current custom KFold implementation.
Check calibration size before fitting
A calibration set should contain enough samples for the requested internal CV structure.
The current source does not automatically reduce n_cv to match a small
calibration set.
Regression fallback depends on calibration targets
Both GradientCOBRA and MixCOBRA compute:
So changing the custom aggregation dataset changes not only:
but also the zero-weight fallback value.
Classification fallback depends on calibration targets
CombinedClassifier computes:
from:
So a different calibration class distribution can change the fallback class used when a query has no positive kernel-weight mass.
Calibration data affect optimization
The optimized parameters are selected from cross-validation performed on the calibration set.
Therefore changing:
can change:
The source does not treat explicit aggregation data as a passive holdout; they actively define the aggregation model.
Calibration data are part of the fitted estimator
Attributes such as:
remain on the fitted estimator.
So the calibration data are not used only transiently during fit.
They define the reference set against which future query distances are computed.
Prediction compares against calibration representation
For GradientCOBRA:
For CombinedClassifier:
For MixCOBRA, query representations are compared with the stored normalized calibration spaces.
Thus your explicit aggregation dataset becomes the model's prediction reference population.
Difference from a conventional validation set
A conventional validation set is often used only for model selection and then discarded.
In the current COBRA implementation, the aggregation/calibration data also serve as the stored reference observations used by the final prediction mechanism.
So:
are more than a temporary evaluation set.
They are part of the fitted aggregation model.
Saving custom split indices
For reproducibility, it is often useful to save the row indices that produced:
For example:
Then the same explicit aggregation configuration can be reconstructed later.
The package does not save original full-dataset indices automatically.
DataFrames
If you pass pandas DataFrames and Series, the resolver uses:
and:
so the fitted internal context is NumPy-based.
Index labels are not retained as sample identifiers.
If row identity matters, keep the original DataFrame index or split indices externally.
Example: externally prepared holdout splitter
from kfc_procedure.cobra.core.splitters import (
RandomHoldoutSplitter,
)
splitter = RandomHoldoutSplitter(
calibration_size=0.3,
random_state=42,
)
indices = splitter.split(
X,
y,
)
Then:
model.fit(
X[
indices.train_idx
],
y[
indices.train_idx
],
X_l=X[
indices.eval_idx
],
y_l=y[
indices.eval_idx
],
)
This is how you can use a splitter other than the high-level estimator's hard-coded default first-level path.
Example: custom index logic
train_idx = np.arange(
0,
800,
)
cal_idx = np.arange(
800,
1000,
)
model.fit(
X[
train_idx
],
y[
train_idx
],
X_l=X[
cal_idx
],
y_l=y[
cal_idx
],
)
The package accepts the resulting arrays without needing to know how those indices were selected.
Custom aggregation data with KFC Procedure
The top-level KFC Procedure does not currently expose:
through its public:
interface.
KFC performs its own internal split, then its C-Step receives the internal aggregation prediction matrix.
Therefore the custom aggregation-data interface documented here applies
directly to the standalone COBRA estimators, not to the top-level
KFCProcedure.fit() API.
KFC C-Step wrappers
Inside KFC, wrappers such as:
receive the already-created F-Step matrix and call the wrapped estimator with:
That is a different advanced pathway from supplying raw feature-based
X_l/y_l to standalone COBRA.
Custom aggregation vs KFC internal split
Standalone COBRA:
Top-level KFC:
If you need full control over the KFC internal split, the current public KFC
fit interface does not expose that as X_l/y_l.
Common mistake: using test data as aggregation data unintentionally
Because X_l/y_l become part of the fitted model and influence
hyperparameter optimization, they are not untouched final-evaluation data.
If you need an independent test set, keep it separate and call:
only after fitting.
Common mistake: mismatched preprocessing
The base estimators are fitted on:
and asked to predict:
If those arrays were produced with inconsistent external preprocessing, the package does not reconcile them.
Ensure they use the same feature schema and transformation conventions.
Common mistake: different column order
Even if:
the semantics can still differ if columns were reordered.
The package does not store feature names after conversion to NumPy.
Common mistake: too-small classification calibration set
A small:
can omit classes or create unstable internal K-fold composition.
The source does not automatically repair class coverage.
Inspect:
before fitting.
Common mistake: expecting split_ratio to modify explicit data
Once:
are present, the resolver does not use:
Changing those parameters has no effect on your custom first-level split.
Common mistake: combining X_l/y_l with as_predictions=True
Because as_predictions=True is checked first, your explicit calibration
arrays are bypassed.
Choose one mode deliberately:
or:
Debugging explicit aggregation mode
Check mode
Expected:
for raw-feature explicit X_l/y_l mode.
Check stored arrays
Check calibration target summary
Regression:
Classification:
Check internal estimator count
In feature-based explicit aggregation mode, internal base estimators are still fitted.
Check calibration prediction-space shape
GradientCOBRA or MixCOBRA:
CombinedClassifier:
Check CV uses calibration rows
for non-empty validation folds.
Validation checklist
Before calling fit():
For ordinary feature mode:
For numeric regression:
assert np.isfinite(
X_train
).all()
assert np.isfinite(
X_cal
).all()
assert np.isfinite(
y_train
).all()
assert np.isfinite(
y_cal
).all()
These checks are useful because the package's component-level validation is not uniform.
Current source comparison
| Estimator | Explicit X_l/y_l |
Base estimators fit on | Aggregation representation built on |
|---|---|---|---|
GradientCOBRA |
Yes | X/y |
predictions on X_l |
MixCOBRARegressor |
Yes | X/y |
X_l + predictions on X_l |
CombinedClassifier |
Yes | X/y |
hard predictions on X_l |
Calibration-derived fitted state
| Estimator | Calibration-derived state |
|---|---|
GradientCOBRA |
X_l_, y_l_, Y_l_norm_, distance_matrix_, cv_folds_, global_mean_ |
MixCOBRARegressor |
X_l_, y_l_, normalized X/Y spaces, distance matrices, cv_folds_, global_mean_ |
CombinedClassifier |
X_l_, y_l_, pred_l_, distance_matrix_, cv_folds_, global_majority_class_ |
Branch precedence quick reference
| Inputs | Resolver mode |
|---|---|
as_predictions=True |
precomputed prediction mode |
X_l + y_l |
explicit aggregation mode |
pred_features only |
pred_features mode |
| none of the above | automatic split mode |
only one of X_l / y_l |
error |
Quick reference
| Goal | Current source usage |
|---|---|
| supply dedicated calibration features | X_l=X_cal, y_l=y_cal |
| bypass automatic overlap split | provide both X_l and y_l |
| keep internal base estimators | use explicit X_l/y_l with as_predictions=False |
| skip internal base estimators | use as_predictions=True instead |
| use external stratified first-level split | prepare arrays externally, then pass X_l/y_l |
| use external chronological first-level split | prepare arrays externally, then pass X_l/y_l |
| inspect resolved training rows | X_k_, y_k_ |
| inspect resolved calibration rows | X_l_, y_l_ |
| inspect internal calibration CV | cv_folds_ |
Mental model
Custom aggregation data lets you decide which observations train the model experts and which observations become the reference population for COBRA aggregation.
with:
The explicit X_l/y_l path bypasses the automatic first-level split, but it
does not bypass the COBRA distance, cross-validation, optimization, or final
reference-set behavior.