Weighted Mean

WeightedMeanCombiner is a supervised regression combiner used by the KFC C-Step.

It learns a linear relationship between the F-Step prediction matrix and the true regression target.

The implementation is registered under:

weighted_mean

and lives in:

kfc_procedure/core/combiner/regression/weighted_mean.py

The source describes it as:

Weighted mean combiner (regression).

Learns optimal linear weights using Ordinary Least Squares (OLS).

Core idea

Assume the F-Step produces a prediction matrix

\[ X \in \mathbb{R}^{n\times K}, \]

where:

\(n\) is the number of observations;
\(K\) is the number of prediction columns.

In KFC, those columns correspond to divergence-specific prediction families.

The source models the target as

\[ y \approx Xw, \]

where:

\[ w \in \mathbb{R}^{K} \]

contains the learned coefficients.

Conceptually:

F-Step predictions
        │
        ▼
┌──────────────────────────────┐
│ euclidean   gkl   logistic   │
├──────────────────────────────┤
│   12.4      12.8     12.1    │
│   20.7      21.0     20.4    │
│    8.3       8.1      8.5    │
└──────────────────────────────┘
        │
        ▼
LinearRegression
        │
        ▼
learned coefficients
        │
        ▼
final regression prediction

Constructor

The current implementation is:

WeightedMeanCombiner(
    fit_intercept=False,
)

The constructor creates:

self.model = LinearRegression(
    fit_intercept=fit_intercept
)

So the only combiner-specific constructor parameter is:

fit_intercept

with default:

False

Default model

With the default:

fit_intercept=False

the learned rule is approximately:

\[ \widehat y = \sum_{k=1}^{K}w_kX_k. \]

If:

fit_intercept=True

then the model becomes:

\[ \widehat y = b + \sum_{k=1}^{K}w_kX_k. \]

Important: this is not a constrained weighted average

The name weighted_mean can suggest a convex weighted average, but the current source does not impose such constraints.

The learned coefficients are produced by ordinary:

LinearRegression

and are not constrained to satisfy:

\[ w_k \ge 0 \]

or:

\[ \sum_{k=1}^{K}w_k = 1. \]

Therefore coefficients may be:

negative
greater than one
or sum to a value other than one

A more precise description of the current implementation is:

OLS linear combiner over F-Step prediction columns

rather than a mathematically constrained weighted mean.

Fit behavior

The implementation is:

def fit(
    self,
    X,
    y,
):
    X = np.asarray(X)

    if X.ndim != 2:
        raise ValueError(
            f"Expected 2D array, got {X.shape}"
        )

    self.model.fit(
        X,
        y,
    )

    return self

There are three important behaviors here:

input is converted with np.asarray();
X must be two-dimensional;
the internal LinearRegression is fitted directly on X and y.

Required input shape

The expected shape is:

(n_samples, n_prediction_columns)

In KFC:

(n_samples, n_divergences)

For example, with four divergences:

X.shape == (
    n_samples,
    4,
)

Invalid one-dimensional input

This is invalid:

X.shape == (
    n_samples,
)

because the source explicitly checks:

X.ndim != 2

and raises:

ValueError:
Expected 2D array, got ...

Even if you have only one prediction source, the matrix should still have shape:

(n_samples, 1)

rather than:

(n_samples,)

Prediction behavior

WeightedMeanCombiner.combine() is:

def combine(
    self,
    X,
):
    X = np.asarray(X)

    return self.model.predict(X)

Because BaseCombiner.predict() delegates to:

self.combine(X)

both of these are equivalent:

combiner.combine(P_test)

and:

combiner.predict(P_test)

Direct example

import numpy as np

from kfc_procedure.core.combiner.regression import (
    WeightedMeanCombiner,
)


P_train = np.array([
    [10.0, 10.5,  9.8],
    [12.0, 11.8, 12.3],
    [18.0, 18.4, 17.9],
    [25.0, 24.7, 25.2],
])

y_train = np.array([
    10.2,
    12.1,
    18.3,
    25.0,
])


combiner = WeightedMeanCombiner(
    fit_intercept=False,
)

combiner.fit(
    P_train,
    y_train,
)


P_test = np.array([
    [14.0, 14.2, 13.9],
    [20.0, 19.8, 20.3],
])

y_pred = combiner.predict(
    P_test
)

print(
    y_pred
)

Inspect the learned coefficients

The fitted scikit-learn model is available at:

combiner.model

The learned coefficients are:

combiner.model.coef_

For example:

print(
    combiner.model.coef_
)

If there are three prediction columns, the shape is typically:

(3,)

Inspect the intercept

print(
    combiner.model.intercept_
)

With:

fit_intercept=False

scikit-learn's fitted intercept is effectively zero.

With:

fit_intercept=True

the model estimates an intercept from the training data.

Interpreting coefficients

Suppose the F-Step column order is:

euclidean
gkl
logistic

and the learned coefficients are:

[
    0.50,
    -0.20,
    0.90,
]

With no intercept, the learned aggregation rule is:

\[ \widehat y = 0.50p_{\text{euclidean}} - 0.20p_{\text{gkl}} + 0.90p_{\text{logistic}}. \]

This example shows why the implementation should not be interpreted as a convex average.

KFC prediction-column order

The F-Step prediction matrix is ordered according to:

model.fstep_.models_

You can inspect the column names with:

divergence_names = list(
    model.fstep_.models_.keys()
)

print(
    divergence_names
)

Then pair them with the learned coefficients:

weights = model.cstep_.strategy_.model.coef_

for name, weight in zip(
    divergence_names,
    weights,
):
    print(
        name,
        weight,
    )

This is the recommended way to interpret which divergence-specific predictions receive positive or negative linear coefficients.

Using an intercept

To fit an intercept:

combiner = WeightedMeanCombiner(
    fit_intercept=True,
)

Then the model is:

\[ \widehat y = b + Xw. \]

Inspect:

print(
    combiner.model.intercept_
)

and:

print(
    combiner.model.coef_
)

KFC integration

Conceptually, the registry name is:

weighted_mean

and the intended KFC configuration is:

from kfc_procedure import KFCRegressor

model = KFCRegressor(
    divergences=[
        "euclidean",
        "gkl",
    ],
    local_model="ridge",
    combiner="weighted_mean",
    combiner_params={
        "fit_intercept": False,
    },
)

However, the current C-Step implementation introduces an important constructor compatibility issue.

Current C-Step `random_state` issue

When C-Step resolves a string combiner, the source does:

params = dict(
    self.combiner_params
)

if "random_state" not in params:
    params["random_state"] = self.random_state

and then:

CombinerFactory.create(
    name,
    **params
)

That means a string-based weighted_mean configuration is constructed conceptually as:

WeightedMeanCombiner(
    fit_intercept=False,
    random_state=...,
)

But the actual constructor is only:

def __init__(
    self,
    fit_intercept=False,
):
    ...

It does not accept:

random_state

Result

The current string-based KFC path can raise:

TypeError:
WeightedMeanCombiner.__init__() got an unexpected keyword argument 'random_state'

This can happen even if the forwarded value is:

None

because the unsupported keyword is still present.

Current source limitation

With the current source version, weighted_mean is affected by C-Step's unconditional random_state parameter injection.

This is an implementation issue in C-Step parameter forwarding, not an issue with WeightedMeanCombiner itself.

Current workaround

Pass a pre-instantiated combiner object instead of the string name.

from kfc_procedure import KFCRegressor

from kfc_procedure.core.combiner.regression import (
    WeightedMeanCombiner,
)


model = KFCRegressor(
    divergences=[
        "euclidean",
        "gkl",
    ],
    local_model="ridge",
    combiner=WeightedMeanCombiner(
        fit_intercept=False,
    ),
    random_state=42,
)

C-Step checks:

if not isinstance(
    self.combiner,
    str,
):
    return self.combiner

so the already-created object is used directly and the unsupported random_state keyword is not injected into its constructor.

How KFC trains weighted mean

KFC internally separates the provided training data into two subsets:

D_k
D_l

The K-Step and F-Step are fitted using:

D_k

Then:

D_l

is passed through those fitted stages to create:

P_l

The C-Step then fits:

WeightedMeanCombiner.fit(
    P_l,
    y_l,
)

Conceptually:

flowchart LR
    DK["Dₖ"]
    K["K-Step"]
    F["F-Step"]

    DL["Dₗ"]
    P["Prediction matrix Pₗ"]
    W["WeightedMeanCombiner"]
    Y["Learned linear combiner"]

    DK --> K --> F
    DL --> P
    F --> P
    P --> W
    DL --> W
    W --> Y

The weighted combiner is therefore trained on divergence-level predictions generated for the internal aggregation subset.

Reconstruct the C-Step input

After fitting:

clusters = model.kstep_.predict(
    X_test
)

P_test = model.fstep_.predict(
    X_test,
    clusters,
)

print(
    P_test.shape
)

Then:

y_pred = model.cstep_.predict(
    P_test
)

uses the fitted weighted linear model.

Inspect the fitted KFC combiner

After fitting:

strategy = model.cstep_.strategy_

Then:

print(
    type(strategy).__name__
)

should be:

WeightedMeanCombiner

when that strategy is used.

The internal linear regression model is:

strategy.model

Inspect KFC coefficients

weights = strategy.model.coef_

print(
    weights
)

Map the coefficients to divergence names:

divergence_names = list(
    model.fstep_.models_.keys()
)

for name, weight in zip(
    divergence_names,
    weights,
):
    print(
        f"{name}: {weight}"
    )

Inspect the KFC intercept

print(
    strategy.model.intercept_
)

Its interpretation depends on:

fit_intercept

used when creating the combiner.

Example coefficient table

import pandas as pd

divergence_names = list(
    model.fstep_.models_.keys()
)

weights = model.cstep_.strategy_.model.coef_

table = pd.DataFrame({
    "divergence": divergence_names,
    "coefficient": weights,
})

print(
    table
)

Conceptually:

    divergence    coefficient
0   euclidean        0.62
1   gkl              0.18
2   logistic        -0.11
3   is               0.44

Again, these are OLS coefficients, not normalized ensemble probabilities or convex weights.

Weighted mean vs ordinary mean

MeanCombiner uses:

\[ \widehat y = \frac{1}{K} \sum_k X_k. \]

Every column contributes equally.

WeightedMeanCombiner instead learns:

\[ \widehat y = Xw \]

or:

\[ \widehat y = b+Xw. \]

So the coefficient assigned to each column is learned from data.

Weighted mean vs stacking regressor

Both use supervised C-Step fitting.

The default implementations differ as follows:

Property	Weighted Mean	Stacking Regressor
Internal model	`LinearRegression`	default `LinearRegression`
Default intercept	`False`	sklearn default, normally `True`
Custom meta-model	No	Yes
Fitted estimator location	`model`	`meta_model_`
Clones configured estimator	No	Yes
Purpose	OLS coefficient combiner	general meta-regression abstraction

With their defaults, both are linear, but they are not configured identically.

For more detail, see:

Stacking

Weighted mean vs GradientCOBRA

WeightedMeanCombiner learns one global linear relationship:

\[ \widehat y = Xw+b. \]

GradientCOBRACombiner instead passes the F-Step matrix into a kernel-based prediction-space aggregation procedure.

So their aggregation mechanisms are structurally different:

weighted_mean
    prediction matrix
        ↓
    global OLS fit
        ↓
    final value

versus:

gradientcobra
    prediction matrix
        ↓
    distance / kernel consensus
        ↓
    local weighted aggregation
        ↓
    final value

For the algorithmic details, see:

GradientCOBRA

Weighted mean with one divergence

If KFC uses only:

divergences=[
    "euclidean"
]

then the prediction matrix has shape:

(n_samples, 1)

Weighted mean still works as a linear calibration model:

\[ \widehat y = wp \]

or with an intercept:

\[ \widehat y = b+wp. \]

So even with one divergence, the learned mapping can rescale or shift the single F-Step prediction column.

Weighted mean with several divergences

With:

divergences=[
    "euclidean",
    "gkl",
    "logistic",
    "is",
]

the combiner learns four coefficients:

\[ w = ( w_{\text{euclidean}}, w_{\text{gkl}}, w_{\text{logistic}}, w_{\text{is}} ). \]

The source does not perform feature selection or regularization inside this combiner.

All behavior comes from standard LinearRegression.

Multicollinearity

Because WeightedMeanCombiner delegates directly to sklearn.linear_model.LinearRegression, the source itself adds no special handling for strongly correlated prediction columns.

For example, two divergences may produce very similar F-Step predictions.

The current combiner does not:

regularize coefficients
drop duplicate columns
normalize weights
enforce sparsity

Any numerical behavior in such cases comes from the underlying LinearRegression implementation.

No internal cross-validation

WeightedMeanCombiner does not perform cross-validation.

Its complete fitting logic is:

self.model.fit(
    X,
    y,
)

There is no internal search over:

fit_intercept
regularization strength
subset of columns
weight constraints

If you need a more flexible learned C-Step, use a custom stacking meta-model or a COBRA strategy.

No normalization inside the combiner

The source does not normalize or standardize the prediction columns before calling:

LinearRegression.fit()

The F-Step prediction matrix is used as supplied.

So if prediction columns have different numerical behavior, the current combiner itself does not rescale them first.

No explicit finite-value validation

WeightedMeanCombiner.fit() explicitly validates only:

X is two-dimensional

It does not manually check:

NaN
infinity
X/y sample mismatch

Those errors, when relevant, are left to scikit-learn's LinearRegression.

A useful upstream check is:

import numpy as np

print(
    np.isfinite(
        P_train
    ).all()
)

Fit-before-predict behavior

The combiner itself does not maintain its own _is_fitted flag.

It relies on the wrapped scikit-learn model.

If:

combiner.predict(X)

is called before:

combiner.fit(X, y)

then LinearRegression.predict() is responsible for reporting that the model is not fitted.

Direct use outside KFC

WeightedMeanCombiner can aggregate predictions from any regression models, not only KFC divergence families.

For example:

P_train = np.column_stack([
    pred_model_a_train,
    pred_model_b_train,
    pred_model_c_train,
])

P_test = np.column_stack([
    pred_model_a_test,
    pred_model_b_test,
    pred_model_c_test,
])

combiner = WeightedMeanCombiner(
    fit_intercept=False,
)

combiner.fit(
    P_train,
    y_train,
)

final_prediction = combiner.predict(
    P_test
)

The class only assumes that X is a two-dimensional prediction matrix.

Custom constrained weighting

The current implementation does not provide constrained weights.

If you need:

\[ w_k\geq0 \]

and:

\[ \sum_kw_k=1, \]

that behavior is not supported by WeightedMeanCombiner as currently written.

A custom BaseCombiner implementation would be needed for those constraints.

For example, the custom combiner would have to solve an explicitly constrained optimization problem rather than using ordinary LinearRegression.

Debugging weighted mean

1. Inspect matrix shape

print(
    P_train.shape
)

Expected:

(n_samples, n_prediction_columns)

2. Check finite values

print(
    np.isfinite(
        P_train
    ).all()
)

3. Inspect fitted coefficients

strategy = model.cstep_.strategy_

print(
    strategy.model.coef_
)

4. Inspect intercept

print(
    strategy.model.intercept_
)

5. Inspect divergence order

print(
    list(
        model.fstep_.models_.keys()
    )
)

6. Check the current constructor error

If KFC raises:

unexpected keyword argument 'random_state'

while using:

combiner="weighted_mean"

use:

combiner=WeightedMeanCombiner(...)

instead of the string path with the current source version.

Quick reference

Property	Current behavior
Registry name	`weighted_mean`
Task	regression
Internal model	`LinearRegression`
Default `fit_intercept`	`False`
Requires target in `fit()`	Yes
Input requirement	2D prediction matrix
Weight positivity enforced	No
Sum-to-one enforced	No
Internal normalization	No
Internal cross-validation	No
Regularization	No
Learned coefficients	`model.coef_`
Learned intercept	`model.intercept_`
Current string KFC path	affected by C-Step `random_state` injection

Mental model

Weighted mean learns one global OLS coefficient for each F-Step prediction column.

\[ \boxed{ \begin{bmatrix} p_1(x) & p_2(x) & \cdots & p_K(x) \end{bmatrix} \rightarrow \text{LinearRegression} \rightarrow \widehat y(x) } \]

Despite the class name, the current source implements unconstrained linear regression rather than a normalized weighted average.

Weighted Mean

Core idea

Constructor

Default model

Important: this is not a constrained weighted average

Fit behavior

Required input shape

Invalid one-dimensional input

Prediction behavior

Direct example

Inspect the learned coefficients

Inspect the intercept

Interpreting coefficients

KFC prediction-column order

Using an intercept

KFC integration

Current C-Step random_state issue

Result

Current workaround

How KFC trains weighted mean

Reconstruct the C-Step input

Inspect the fitted KFC combiner

Inspect KFC coefficients

Inspect the KFC intercept

Example coefficient table

Weighted mean vs ordinary mean

Weighted mean vs stacking regressor

Weighted mean vs GradientCOBRA

Weighted mean with one divergence

Weighted mean with several divergences

Multicollinearity

No internal cross-validation

No normalization inside the combiner

No explicit finite-value validation

Fit-before-predict behavior

Direct use outside KFC

Custom constrained weighting

Debugging weighted mean

1. Inspect matrix shape

2. Check finite values

3. Inspect fitted coefficients

4. Inspect intercept

5. Inspect divergence order

6. Check the current constructor error

Quick reference

Mental model

Current C-Step `random_state` issue