Regularizers¶
The mixturelib.regularizers
contains classes:
mixturelib.regularizers.Regularizers
mixturelib.regularizers.RegularizeModel
mixturelib.regularizers.RegularizeFunc
-
class
mixturelib.regularizers.
RegularizeFunc
(ListOfModels=None, R=<function RegularizeFunc.<lambda>>, epoch=100, device='cpu')[source]¶ The class of regularization to create any relationship between prior means. The relationship between the parameters is set by using the link function.
In the M-step solves next optimisation problem \(\sum_{k=1}^{num\_models}\left[-\frac{1}{2}w_k^0A_k^{-1}w_k^0+ w_k^0A_k^{-1}\mathsf{E}w_k\right] + R(W^0) \to \infty\).
Warning
All local models must be Linear model for the regression task. Also can be used
mixturelib.local_models.EachModelLinear
.Warning
Link function represent a likelihood. This function will be maximizing during optimisation.
This Regularizer make correction on the M-step for each Linear Model.
Parameters: - ListOfModels (list) – A list of local models to be regularized.
- device (string) – The device for pytorch. Can be ‘cpu’ or ‘gpu’. Default ‘cpu’.
- R (function) – The link function between prior means for all local models. The function must be scalar with type FloatTensor.
- epoch (int) – The number of epoch for solving optimisation problem in the M-step.
Example:
>>> _ = torch.random.manual_seed(42) # Set random seed for repeatability >>> >>> w = torch.randn(2, 1) # Generate real parameter vector >>> X = torch.randn(10, 2) # Generate features data >>> Z = torch.ones(10, 1) # Set that all data correspond to this model >>> Y = X@w + 0.1*torch.randn(10, 1) # Generate target data with noise 0.1 >>> >>> first_model = EachModelLinear( ... input_dim=2, ... A=torch.tensor([1., 1.]), ... w=torch.tensor([[0.], [0.]])) # Init first local model >>> second_model = EachModelLinear( ... input_dim=2, ... A=torch.tensor([1., 1.]), ... w=torch.tensor([[1.], [1.]])) # Init second local model >>> hyper_parameters = { ... 'alpha': torch.tensor([1., 1e-10])} # Set regularization parameter >>> >>> first_model.w_0, first_model.W # First prior and paramaters before (tensor([[0.], [0.]]), tensor([[1.3314e-06], [8.6398e-06]])) >>> second_model.w_0, second_model.W # Second prior and paramaters before (tensor([[1.], [1.]]), tensor([[1.0000], [1.0000]])) >>> >>> Rg = RegularizeModel( ... ListOfModels=[first_model, second_model], ... R = lambda x: -(x**2).sum()) # Set regulariser >>> _ = Rg.M_step(X, Y, Z, hyper_parameters) # Regularize >>> >>> first_model.w_0, first_model.W # First prior and paramaters after (tensor([[4.8521e-06], [6.7789e-06]]), tensor([[1.3314e-06], [8.6398e-06]])) >>> second_model.w_0, second_model.W # Second prior and paramaters after (tensor([[0.9021], [0.9021]]), tensor([[1.0000], [1.0000]]))
-
E_step
(X, Y, Z, HyperParameters)[source]¶ The method does nothing.
Parameters: - X (FloatTensor) – The tensor of shape num_elements \(\times\) num_feature.
- Y (FloatTensor) – The tensor of shape num_elements \(\times\) num_answers.
- Z (FloatTensor) – The tensor of shape num_elements \(\times\) num_models.
- HyperParameters (dict) – The dictionary of all hyper parametrs. Where key is string and value is FloatTensor.
-
M_step
(X, Y, Z, HyperParameters)[source]¶ Make some regularization on the M-step.
Solves next optimisation problem \(\sum_{k=1}^{num\_models}\left[-\frac{1}{2}w_k^0A_k^{-1}w_k^0+ w_k^0A_k^{-1}\mathsf{E}w_k\right] + R(W^0) \to \infty\).
Parameters: - X (FloatTensor) – The tensor of shape num_elements \(\times\) num_feature.
- Y (FloatTensor) – The tensor of shape num_elements \(\times\) num_answers.
- Z (FloatTensor) – The tensor of shape num_elements \(\times\) num_models.
- HyperParameters (dict) – The dictionary of all hyper parametrs. Where key is string and value is FloatTensor.
-
class
mixturelib.regularizers.
RegularizeModel
(ListOfModels=None, device='cpu')[source]¶ The class of regularization to create a relationship between prior means. The relationship between the parameters in this case, is that the mean distributions should be equal.
Warning
All local models must be Linear model for the regression task. Also can be used
mixturelib.local_models.EachModelLinear
.This Regularizer make correction on the M-step for each Linear Model.
Parameters: - ListOfModels (list) – A list of local models to be regularized.
- device (string) – The device for pytorch. Can be ‘cpu’ or ‘gpu’. Default ‘cpu’.
Example:
>>> _ = torch.random.manual_seed(42) # Set random seed for repeatability >>> >>> w = torch.randn(2, 1) # Generate real parameter vector >>> X = torch.randn(10, 2) # Generate features data >>> Z = torch.ones(10, 1) # Set that all data correspond to this model >>> Y = X@w + 0.1*torch.randn(10, 1) # Generate target data with noise 0.1 >>> >>> first_model = EachModelLinear( ... input_dim=2, ... A=torch.tensor([1., 1.]), ... w=torch.tensor([[0.], [0.]])) # Init first local model >>> second_model = EachModelLinear( ... input_dim=2, ... A=torch.tensor([1., 1.]), ... w=torch.tensor([[1.], [1.]])) # Init second local model >>> hyper_parameters = { ... 'alpha': torch.tensor([1., 1e-10])} # Set regularization parameter >>> >>> first_model.w_0, first_model.W # First prior and paramaters before (tensor([[0.], [0.]]), tensor([[1.3314e-06], [8.6398e-06]])) >>> second_model.w_0, second_model.W # Second prior and paramaters before (tensor([[1.], [1.]]), tensor([[1.0000], [1.0000]])) >>> >>> Rg = RegularizeModel( ... ListOfModels=[first_model, second_model]) # Set regulariser >>> _ = Rg.M_step(X, Y, Z, hyper_parameters) # Regularize >>> >>> first_model.w_0, first_model.W # First prior and paramaters after (tensor([[0.3333], [0.5000]]), tensor([[1.3314e-06], [8.6398e-06]])) >>> second_model.w_0, second_model.W # Second prior and paramaters after (tensor([[0.6667], [0.5000]]), tensor([[1.0000], [1.0000]]))
-
E_step
(X, Y, Z, HyperParameters)[source]¶ The method does nothing.
Parameters: - X (FloatTensor) – The tensor of shape num_elements \(\times\) num_feature.
- Y (FloatTensor) – The tensor of shape num_elements \(\times\) num_answers.
- Z (FloatTensor) – The tensor of shape num_elements \(\times\) num_models.
- HyperParameters (dict) – The dictionary of all hyper parametrs. Where key is string and value is FloatTensor.
-
M_step
(X, Y, Z, HyperParameters)[source]¶ Make some regularization on the M-step.
For all local model from ListOfModels with prior, make regularization \(w^0_k = \left[A_k^{-1} + (num\_models-1)\alpha\right] \left(A_k^{-1}\mathsf{E}w_k + \alpha\sum_{k'\not=k}w_k'\right)\)
Warning
HyperParameters must contain alpha hyperparameter.
Parameters: - X (FloatTensor) – The tensor of shape num_elements \(\times\) num_feature.
- Y (FloatTensor) – The tensor of shape num_elements \(\times\) num_answers.
- Z (FloatTensor) – The tensor of shape num_elements \(\times\) num_models.
- HyperParameters (dict) – The dictionary of all hyper parametrs. Where key is string and value is FloatTensor.
-
class
mixturelib.regularizers.
Regularizers
[source]¶ Base class for all regulizers.
-
E_step
(X, Y, Z, HyperParameters)[source]¶ Make some regularization on the E-step.
Parameters: - X (FloatTensor) – The tensor of shape num_elements \(\times\) num_feature.
- Y (FloatTensor) – The tensor of shape num_elements \(\times\) num_answers.
- Z (FloatTensor) – The tensor of shape num_elements \(\times\) 1.
- HyperParameters (dict) – The dictionary of all hyper parametrs. Where key is string and value is FloatTensor.
-
M_step
(X, Y, Z, HyperParameters)[source]¶ Make some regularization on the M-step.
Parameters: - X (FloatTensor) – The tensor of shape num_elements \(\times\) num_feature.
- Y (FloatTensor) – The tensor of shape num_elements \(\times\) num_answers.
- Z (FloatTensor) – The tensor of shape num_elements \(\times\) 1.
- HyperParameters (dict) – The dictionary of all hyper parametrs. Where key is string and value is FloatTensor.
-