Goto Books. Papers The following preprints are provided here to allow for a deeper view of our research work, as well as to promote the rapid dissemination of research results.
Please consider, on the other hand, that these preprints can differ from their published version in ways that may not be entirely negligible, and for this reason we recommend you to refer to the published version whenever they have to be used or cited. Gradient-based bilevel optimization for multi-penalty ridge regression through matrix differential calculus.
European Journal of Control 81 , Tweet Gradient-based bilevel optimization for multi-penalty ridge regression through matrix differential calculus Authors: Maroni, G. Abstract: Common regularization algorithms for linear regression, such as LASSO and Ridge regression, rely on a regularization hyperparameter that balances the trade-off between minimizing the fitting error and the norm of the learned model coefficients.
As this hyperparameter is scalar, it can be easily selected via random or grid search optimizing a cross-validation criterion. We optimize these hyperparameters using a gradient-based approach, wherein the gradient of a cross-validation criterion with respect to the regularization hyperparameters is computed analytically through matrix differential calculus. Additionally, we introduce two strategies tailored for sparse model learning problems aiming at reducing the risk of overfitting to the validation data.
Numerical examples demonstrate that the proposed multi-hyperparameter regularization approach outperforms LASSO, Ridge, and Elastic Net regression in terms of R2 score both in a static regression and in a system identification problem. Moreover, the analytical computation of the gradient proves to be more efficient in terms of computational time compared to automatic differentiation, especially when handling a large number of input variables, with an improvement of more than an order of magnitude.
