Loss Functions
Loss functions connect COBRA predictions to the optimization objective.
During cross-validation, a COBRA estimator:
- builds predictions for the validation rows;
- compares those predictions with the validation targets;
- evaluates a registered loss;
- averages the fold losses;
- asks the optimizer to minimize that value.
The current source provides six built-in losses:
with aliases for several of them.
The implementations live under:
kfc_procedure/cobra/core/losses/
├── base.py
├── mse.py
├── mae.py
├── huber.py
├── log_loss.py
├── hinge.py
└── quantile.py
and are registered through:
Losses in the COBRA pipeline
The loss layer appears after fold-level aggregation:
flowchart LR
P["Candidate parameters"]
K["Kernel weights"]
A["Aggregator"]
YH["Fold predictions"]
Y["Validation targets"]
L["Loss"]
O["Optimizer"]
P --> K --> A --> YH
YH --> L
Y --> L
L --> O
For one cross-validation fold, the objective has the form:
The optimizer minimizes the average of these fold losses.
Built-in losses
| Registry name | Aliases | Class | Primary source formula |
|---|---|---|---|
mse |
l2, squared_error |
MSELoss |
mean squared error |
mae |
l1 |
MAELoss |
mean absolute error |
huber |
— | HuberLoss |
quadratic/linear robust loss |
log_loss |
cross_entropy |
LogLoss |
binary cross-entropy |
hinge |
— | HingeLoss |
mean hinge loss |
quantile |
— | QuantileLoss |
pinball / quantile loss |
BaseLoss
Every built-in loss inherits:
The abstract interface is intentionally small:
A loss object is therefore callable:
and must return one scalar float.
LossFactory
Losses are created through:
which inherits the package's common:
Example:
Then:
Inspect registered losses
The current source registers names including:
subject to normal module loading.
Mean squared error
MSELoss is registered as:
The source formula is:
Implementation
The implementation is:
y_true = np.asarray(
y_true
)
y_pred = np.asarray(
y_pred
)
return float(
np.mean(
(
y_true
-
y_pred
) ** 2
)
)
No additional weighting, clipping, or regularization is applied.
Direct example
import numpy as np
from kfc_procedure.cobra.core.losses import (
MSELoss,
)
y_true = np.array([
1.0,
2.0,
3.0,
])
y_pred = np.array([
1.2,
1.8,
2.5,
])
loss = MSELoss()
value = loss(
y_true,
y_pred,
)
print(
value
)
MSE behavior
Squaring the residual means larger absolute errors contribute disproportionately more to the objective.
For residual:
the contribution is:
The source module describes MSE as the most common regression loss and notes that it strongly penalizes large errors.
Mean absolute error
MAELoss is registered as:
The source formula is:
Implementation
y_true = np.asarray(
y_true
)
y_pred = np.asarray(
y_pred
)
return float(
np.mean(
np.abs(
y_true
-
y_pred
)
)
)
Direct example
from kfc_procedure.cobra.core.losses import (
MAELoss,
)
loss = MAELoss()
value = loss(
y_true,
y_pred,
)
The source describes MAE as more robust to outliers than MSE because it uses absolute rather than squared error.
Huber loss
HuberLoss combines quadratic behavior for small residuals with linear
behavior for large residuals.
It is registered as:
Constructor
The parameter:
controls the transition point.
The current source does not validate that delta is positive.
Huber formula
Let:
The implementation computes:
and:
Then:
The final loss is the mean across samples.
Implementation
error = (
y_true
-
y_pred
)
abs_error = np.abs(
error
)
quadratic = (
0.5
*
error ** 2
)
linear = (
self.delta
*
(
abs_error
-
0.5
*
self.delta
)
)
return float(
np.mean(
np.where(
abs_error
<=
self.delta,
quadratic,
linear,
)
)
)
Configure Huber loss
High-level COBRA estimators pass:
to LossFactory.create().
Example:
This creates:
Quantile loss
QuantileLoss is registered as:
and is intended for quantile regression.
Its constructor is:
Source formula
Let:
The source computes:
The returned loss is the mean of those values.
Equivalent piecewise form:
Implementation
error = (
y_true
-
y_pred
)
return float(
np.mean(
np.maximum(
self.tau
*
error,
(
self.tau
-
1
)
*
error,
)
)
)
tau
The source documentation states:
However, the constructor currently performs no explicit interval validation.
So values outside:
are not rejected when the loss object is constructed.
Source contract vs enforcement
The module documents tau as a quantile level in (0, 1), but the current
implementation simply stores the supplied value.
Median-like quantile loss
With:
the formula gives symmetric linear penalties on positive and negative residuals.
The resulting value is proportional to absolute error:
So the default quantile loss is not numerically identical to MAELoss; it is
half its value for the same residuals.
Binary log loss
LogLoss is registered as:
The source formula is:
This is a binary cross-entropy formula.
Prediction clipping
Before evaluating logs, the source clips predictions:
This protects against:
at exactly 0 or 1.
Implementation
Expected log-loss inputs
The source module describes this loss as being used for probabilistic classification.
The formula expects values conceptually like:
The implementation does not itself verify:
beyond NumPy's normal broadcasting behavior.
Multiclass limitation
The current LogLoss implementation is not a general multiclass
cross-entropy implementation.
It does not:
one-hot encode labels
index class probabilities
sum over class dimensions
accept class labels as a `classes` argument
It directly evaluates the binary formula above.
Therefore it should not be interpreted as a source-supported multiclass cross-entropy implementation.
Hinge loss
HingeLoss is registered as:
The module explicitly states:
Its formula is:
Implementation
y_true = np.asarray(
y_true
)
y_pred = np.asarray(
y_pred
)
return float(
np.mean(
np.maximum(
0.0,
1.0
-
y_true
*
y_pred,
)
)
)
Expected hinge inputs
The source describes the loss as common in SVM-style classification and expects:
The implementation does not convert labels from:
to:
and does not convert hard class labels into decision margins.
It uses whatever arrays are passed directly into:
Important classifier integration detail
CombinedClassifier produces hard class predictions during its
cross-validation objective.
It does not produce probability vectors or decision-function margins for the loss layer.
Therefore classification losses such as:
are not automatically supplied the conventional inputs implied by their source modules.
This is important when configuring the current classifier.
Default loss in COBRA estimators
The current constructors use:
by default in all three main estimators:
That includes the classification estimator:
Regression defaults
For:
and:
the default objective is therefore ordinary mean squared error.
This matches their numeric regression outputs.
CombinedClassifier default
The current CombinedClassifier constructor also specifies:
Its cross-validation loop produces hard predicted class labels and evaluates:
So with the default, class labels are numerically compared using squared error.
Current source behavior
CombinedClassifier does not default to classification error,
log loss, or hinge loss.
Its current default is the same registered MSELoss used by the regression
estimators.
Consequence for encoded class labels
Because MSE is applied directly to class labels, the numeric encoding matters.
For example:
contributes:
But:
contributes:
So with class labels:
the default classifier objective treats the numeric distance between class codes as meaningful.
That is a direct consequence of the current source default.
String labels with MSE
If classification targets are strings such as:
then the default:
cannot perform:
on those string arrays.
So the default CombinedClassifier loss path assumes numerically subtractable labels.
The loss layer itself does not encode categorical labels before evaluation.
Loss resolution
The high-level estimators all resolve the configured loss through:
and store the result in:
Inspect the fitted loss
After component resolution or fitting:
For the default:
Configure MAE
Alias:
Both resolve to:
Configure MSE
These names all resolve to MSELoss:
Configure Huber
Configure quantile loss
The current source simply forwards:
to the constructor.
How GradientCOBRA uses the loss
For every stored cross-validation fold, GradientCOBRA creates fold predictions:
Then:
The fold errors are collected and returned as:
GradientCOBRA objective
For bandwidth \(h\), the cross-validation objective is conceptually:
The configured loss determines the definition of:
The optimizer then searches for a parameter value minimizing:
How MixCOBRA uses the loss
MixCOBRA follows the same pattern.
For one-parameter mode:
For two-parameter mode:
In both cases the fold objective calls:
How CombinedClassifier uses the loss
For each validation row, CombinedClassifier first obtains a hard class prediction.
Then:
Thus the loss receives:
rather than probability vectors.
log_loss with CombinedClassifier
Although:
is registered and can be requested through the factory, the current CombinedClassifier CV path does not call:
inside its objective.
Therefore LogLoss receives hard predicted labels if selected there.
For binary numeric classes 0/1, clipping transforms hard predictions into:
before the logarithms.
That is valid numerically but is not the same input as calibrated probability predictions.
hinge with CombinedClassifier
Likewise, HingeLoss expects:
and conventional hinge use expects margin-like predictions.
The current classifier CV objective instead passes hard predicted labels.
No label conversion or margin calculation is performed by the loss class or the classifier objective.
Loss functions do not know the task
LossFactory registrations do not carry regression or classification
categories in the current source.
The factory itself therefore does not prevent configurations such as:
or:
The package resolves the requested registered class and lets the arrays and formula determine the resulting behavior.
Loss validation scope
The built-in losses are intentionally lightweight.
They do not share a common validation layer for:
Most implementations simply call:
and apply the formula.
Broadcasting
Because the losses use NumPy expressions directly, differently shaped arrays can sometimes broadcast rather than immediately raise an error.
For example:
and:
can produce an (n, n) broadcasted result in some formulas.
The current source does not call a shared:
or shape-equality validator inside the loss classes.
Warning
Ensure y_true and y_pred have the intended matching shape before
evaluating a built-in loss.
Empty arrays
The current loss classes do not explicitly reject empty arrays.
Calling:
on an empty result can return:
with a NumPy runtime warning.
No custom ValueError is implemented for empty input.
Non-finite values
The losses do not generally sanitize:
in the inputs.
LogLoss clips numeric prediction values but does not replace NaN.
So non-finite values can propagate to the returned loss.
No sample weights
The built-in BaseLoss interface accepts only:
There is no:
argument.
Therefore the current built-in cross-validation objectives compute ordinary unweighted means over samples within each fold.
Mean of fold losses
The estimators compute:
where each item is already a mean loss for one fold.
Therefore the final CV objective is a mean of fold means.
It is not explicitly weighted by fold size.
With the current K-fold implementation, fold sizes are usually close, but can
differ by one sample when the calibration sample count is not divisible by
n_cv.
Direct loss comparison
import numpy as np
from kfc_procedure.cobra.core.losses import (
LossFactory,
)
y_true = np.array([
1.0,
2.0,
3.0,
])
y_pred = np.array([
1.2,
1.8,
2.5,
])
for name in [
"mse",
"mae",
"huber",
"quantile",
]:
loss = LossFactory.create(
name
)
print(
name,
loss(
y_true,
y_pred,
),
)
Inspect aliases
mse = LossFactory.create(
"mse"
)
l2 = LossFactory.create(
"l2"
)
squared = LossFactory.create(
"squared_error"
)
print(
type(mse).__name__,
type(l2).__name__,
type(squared).__name__,
)
All three should resolve to:
Binary log-loss example
import numpy as np
from kfc_procedure.cobra.core.losses import (
LogLoss,
)
y_true = np.array([
0,
1,
1,
0,
])
p = np.array([
0.1,
0.8,
0.7,
0.2,
])
loss = LogLoss()
print(
loss(
y_true,
p,
)
)
This matches the binary formula implemented in the source.
Hinge example
from kfc_procedure.cobra.core.losses import (
HingeLoss,
)
y_true = np.array([
-1,
1,
1,
-1,
])
margin = np.array([
-1.2,
0.8,
1.4,
-0.6,
])
loss = HingeLoss()
print(
loss(
y_true,
margin,
)
)
The loss class itself does not create those margin values; they must already be supplied by the caller.
Quantile example
from kfc_procedure.cobra.core.losses import (
QuantileLoss,
)
loss = QuantileLoss(
tau=0.9,
)
value = loss(
y_true,
y_pred,
)
This evaluates the exact pinball formula used by the current source.
Custom loss
A custom loss only needs to subclass:
and implement:
For example:
import numpy as np
from kfc_procedure.cobra.core.losses import (
BaseLoss,
LossFactory,
)
@LossFactory.register(
"root_mse"
)
class RootMSELoss(
BaseLoss
):
def __call__(
self,
y_true,
y_pred,
):
y_true = np.asarray(
y_true
)
y_pred = np.asarray(
y_pred
)
return float(
np.sqrt(
np.mean(
(
y_true
-
y_pred
) ** 2
)
)
)
Then:
can resolve it through the ordinary factory path.
Custom loss with parameters
@LossFactory.register(
"scaled_mae"
)
class ScaledMAELoss(
BaseLoss
):
def __init__(
self,
scale=1.0,
):
self.scale = scale
def __call__(
self,
y_true,
y_pred,
):
return float(
self.scale
*
np.mean(
np.abs(
np.asarray(
y_true
)
-
np.asarray(
y_pred
)
)
)
)
Use:
Loss values and optimizer direction
The current optimizers are supplied objective functions that represent losses, and their search logic minimizes those objectives.
Therefore a custom loss should follow the same convention:
If a metric is naturally "higher is better," convert it into a minimization objective before registering it as a loss.
Loss vs aggregator
The aggregator creates predictions.
The loss scores them.
For regression:
For classification:
The loss does not influence how aggregation itself is computed.
It influences which hyperparameters the optimizer prefers.
Loss vs final prediction
After the optimal bandwidth or other adapter parameter has been chosen, the loss is not called for every ordinary prediction.
The final prediction path uses:
The loss is primarily part of the training-time optimization objective.
Loss vs cross-validation
The loss is applied separately inside every stored fold.
For a candidate parameter vector \(\theta\):
Changing the loss changes:
without changing the distance or kernel implementation itself.
Debugging the configured loss
Inspect configured name
Inspect constructor parameters
Inspect resolved object
Evaluate manually
Debugging shape issues
For the current scalar-per-sample losses, the safest shape is typically:
unless intentionally using a matrix-valued probability representation with a custom loss.
Debugging classification MSE
With CombinedClassifier:
If the default is still:
remember that the numeric class codes directly affect the cross-validation objective.
Debugging log loss
Before calling the built-in LogLoss manually:
The loss clips predictions internally to:
but does not transform hard class labels into calibrated probabilities.
Debugging quantile configuration
The source will report whatever value was passed, including values outside the
documented (0, 1) range.
Current implementation summary
| Loss | Aliases | Parameter | Source expectation |
|---|---|---|---|
mse |
l2, squared_error |
none | numeric predictions |
mae |
l1 |
none | numeric predictions |
huber |
— | delta=1.0 |
numeric residuals |
quantile |
— | tau=0.5 |
quantile regression |
log_loss |
cross_entropy |
none | binary probabilities |
hinge |
— | none | labels {-1,+1} and margin-like predictions |
High-level defaults
| Estimator | Current default loss |
|---|---|
GradientCOBRA |
mse |
MixCOBRARegressor |
mse |
CombinedClassifier |
mse |
Important current-source caveats
| Area | Current behavior |
|---|---|
| shared input validation | minimal |
| shape equality validation | not explicit |
| sample weighting | unsupported |
| empty-array validation | not explicit |
| finite-value sanitization | generally none |
Huber delta validation |
none |
Quantile tau validation |
none |
| log loss | binary formula only |
| hinge label conversion | none |
| CombinedClassifier objective input | hard class predictions |
| CombinedClassifier default loss | MSE |
| multiclass categorical loss | no dedicated built-in |
| fold-size weighting | mean of fold means |
Quick reference
| Goal | Current source option |
|---|---|
| ordinary squared regression error | mse |
| absolute regression error | mae |
| robust quadratic/linear regression error | huber |
| asymmetric quantile objective | quantile |
| binary probability cross-entropy | log_loss |
| SVM-style margin loss | hinge |
| custom task-specific scoring | subclass BaseLoss and register it |
Mental model
The loss is the signal that tells the COBRA optimizer whether one aggregation parameter setting predicts held-out calibration rows better than another.
The current source keeps the loss interface deliberately small. That makes it easy to extend, but it also means task compatibility, target encoding, and prediction semantics are largely the caller's responsibility.