Regularizers

The mixturelib.regularizers contains classes:

class mixturelib.regularizers.RegularizeFunc(ListOfModels=None, R=<function RegularizeFunc.<lambda>>, epoch=100, device='cpu')[source]

The class of regularization to create any relationship between prior means. The relationship between the parameters is set by using the link function.

In the M-step solves next optimisation problem \(\sum_{k=1}^{num\_models}\left[-\frac{1}{2}w_k^0A_k^{-1}w_k^0+ w_k^0A_k^{-1}\mathsf{E}w_k\right] + R(W^0) \to \infty\).

Warning

All local models must be Linear model for the regression task. Also can be used mixturelib.local_models.EachModelLinear.

Warning

Link function represent a likelihood. This function will be maximizing during optimisation.

This Regularizer make correction on the M-step for each Linear Model.

Parameters:
  • ListOfModels (list) – A list of local models to be regularized.
  • device (string) – The device for pytorch. Can be ‘cpu’ or ‘gpu’. Default ‘cpu’.
  • R (function) – The link function between prior means for all local models. The function must be scalar with type FloatTensor.
  • epoch (int) – The number of epoch for solving optimisation problem in the M-step.

Example:

>>> _ = torch.random.manual_seed(42) # Set random seed for repeatability
>>>
>>> w = torch.randn(2, 1) # Generate real parameter vector
>>> X = torch.randn(10, 2) # Generate features data
>>> Z = torch.ones(10, 1) # Set that all data correspond to this model
>>> Y = X@w + 0.1*torch.randn(10, 1) # Generate target data with noise 0.1
>>>
>>> first_model = EachModelLinear(
...     input_dim=2, 
...     A=torch.tensor([1., 1.]),
...     w=torch.tensor([[0.], [0.]])) # Init first local model
>>> second_model = EachModelLinear(
...     input_dim=2,
...     A=torch.tensor([1., 1.]),
...     w=torch.tensor([[1.], [1.]])) # Init second local model
>>> hyper_parameters = {
...     'alpha': torch.tensor([1., 1e-10])} # Set regularization parameter
>>>
>>> first_model.w_0, first_model.W # First prior and paramaters before
(tensor([[0.],
         [0.]]),
tensor([[1.3314e-06],
        [8.6398e-06]]))
>>> second_model.w_0, second_model.W # Second prior and paramaters before
(tensor([[1.],
         [1.]]),
 tensor([[1.0000],
         [1.0000]]))
>>>
>>> Rg = RegularizeModel(
...     ListOfModels=[first_model, second_model],
...     R = lambda x: -(x**2).sum()) # Set regulariser
>>> _ = Rg.M_step(X, Y, Z, hyper_parameters) # Regularize
>>>
>>> first_model.w_0, first_model.W # First prior and paramaters after
(tensor([[4.8521e-06],
         [6.7789e-06]]),
 tensor([[1.3314e-06],
         [8.6398e-06]]))
>>> second_model.w_0, second_model.W # Second prior and paramaters after
(tensor([[0.9021],
         [0.9021]]),
 tensor([[1.0000],
         [1.0000]]))
E_step(X, Y, Z, HyperParameters)[source]

The method does nothing.

Parameters:
  • X (FloatTensor) – The tensor of shape num_elements \(\times\) num_feature.
  • Y (FloatTensor) – The tensor of shape num_elements \(\times\) num_answers.
  • Z (FloatTensor) – The tensor of shape num_elements \(\times\) num_models.
  • HyperParameters (dict) – The dictionary of all hyper parametrs. Where key is string and value is FloatTensor.
M_step(X, Y, Z, HyperParameters)[source]

Make some regularization on the M-step.

Solves next optimisation problem \(\sum_{k=1}^{num\_models}\left[-\frac{1}{2}w_k^0A_k^{-1}w_k^0+ w_k^0A_k^{-1}\mathsf{E}w_k\right] + R(W^0) \to \infty\).

Parameters:
  • X (FloatTensor) – The tensor of shape num_elements \(\times\) num_feature.
  • Y (FloatTensor) – The tensor of shape num_elements \(\times\) num_answers.
  • Z (FloatTensor) – The tensor of shape num_elements \(\times\) num_models.
  • HyperParameters (dict) – The dictionary of all hyper parametrs. Where key is string and value is FloatTensor.
class mixturelib.regularizers.RegularizeModel(ListOfModels=None, device='cpu')[source]

The class of regularization to create a relationship between prior means. The relationship between the parameters in this case, is that the mean distributions should be equal.

Warning

All local models must be Linear model for the regression task. Also can be used mixturelib.local_models.EachModelLinear.

This Regularizer make correction on the M-step for each Linear Model.

Parameters:
  • ListOfModels (list) – A list of local models to be regularized.
  • device (string) – The device for pytorch. Can be ‘cpu’ or ‘gpu’. Default ‘cpu’.

Example:

>>> _ = torch.random.manual_seed(42) # Set random seed for repeatability
>>>
>>> w = torch.randn(2, 1) # Generate real parameter vector
>>> X = torch.randn(10, 2) # Generate features data
>>> Z = torch.ones(10, 1) # Set that all data correspond to this model
>>> Y = X@w + 0.1*torch.randn(10, 1) # Generate target data with noise 0.1
>>>
>>> first_model = EachModelLinear(
...     input_dim=2, 
...     A=torch.tensor([1., 1.]),
...     w=torch.tensor([[0.], [0.]])) # Init first local model
>>> second_model = EachModelLinear(
...     input_dim=2,
...     A=torch.tensor([1., 1.]),
...     w=torch.tensor([[1.], [1.]])) # Init second local model
>>> hyper_parameters = {
...     'alpha': torch.tensor([1., 1e-10])} # Set regularization parameter
>>>
>>> first_model.w_0, first_model.W # First prior and paramaters before
(tensor([[0.],
         [0.]]),
tensor([[1.3314e-06],
        [8.6398e-06]]))
>>> second_model.w_0, second_model.W # Second prior and paramaters before
(tensor([[1.],
         [1.]]),
 tensor([[1.0000],
         [1.0000]]))
>>>
>>> Rg = RegularizeModel(
...     ListOfModels=[first_model, second_model]) # Set regulariser
>>> _ = Rg.M_step(X, Y, Z, hyper_parameters) # Regularize
>>>
>>> first_model.w_0, first_model.W # First prior and paramaters after
(tensor([[0.3333],
         [0.5000]]),
 tensor([[1.3314e-06],
         [8.6398e-06]]))
>>> second_model.w_0, second_model.W # Second prior and paramaters after
(tensor([[0.6667],
         [0.5000]]),
 tensor([[1.0000],
         [1.0000]]))
E_step(X, Y, Z, HyperParameters)[source]

The method does nothing.

Parameters:
  • X (FloatTensor) – The tensor of shape num_elements \(\times\) num_feature.
  • Y (FloatTensor) – The tensor of shape num_elements \(\times\) num_answers.
  • Z (FloatTensor) – The tensor of shape num_elements \(\times\) num_models.
  • HyperParameters (dict) – The dictionary of all hyper parametrs. Where key is string and value is FloatTensor.
M_step(X, Y, Z, HyperParameters)[source]

Make some regularization on the M-step.

For all local model from ListOfModels with prior, make regularization \(w^0_k = \left[A_k^{-1} + (num\_models-1)\alpha\right] \left(A_k^{-1}\mathsf{E}w_k + \alpha\sum_{k'\not=k}w_k'\right)\)

Warning

HyperParameters must contain alpha hyperparameter.

Parameters:
  • X (FloatTensor) – The tensor of shape num_elements \(\times\) num_feature.
  • Y (FloatTensor) – The tensor of shape num_elements \(\times\) num_answers.
  • Z (FloatTensor) – The tensor of shape num_elements \(\times\) num_models.
  • HyperParameters (dict) – The dictionary of all hyper parametrs. Where key is string and value is FloatTensor.
class mixturelib.regularizers.Regularizers[source]

Base class for all regulizers.

E_step(X, Y, Z, HyperParameters)[source]

Make some regularization on the E-step.

Parameters:
  • X (FloatTensor) – The tensor of shape num_elements \(\times\) num_feature.
  • Y (FloatTensor) – The tensor of shape num_elements \(\times\) num_answers.
  • Z (FloatTensor) – The tensor of shape num_elements \(\times\) 1.
  • HyperParameters (dict) – The dictionary of all hyper parametrs. Where key is string and value is FloatTensor.
M_step(X, Y, Z, HyperParameters)[source]

Make some regularization on the M-step.

Parameters:
  • X (FloatTensor) – The tensor of shape num_elements \(\times\) num_feature.
  • Y (FloatTensor) – The tensor of shape num_elements \(\times\) num_answers.
  • Z (FloatTensor) – The tensor of shape num_elements \(\times\) 1.
  • HyperParameters (dict) – The dictionary of all hyper parametrs. Where key is string and value is FloatTensor.