Comparison Some Robust Regularization Methods in Linear Regression via Simulation Study

Authors

  • Sherzad M. Ajeel Department of Mathematics, College of Sciences, University of Duhok, Kurdistan Region, Iraq
  • Hussein A. Hashem Department of Mathematics, College of Sciences, University of Duhok, Kurdistan Region, Iraq

DOI:

https://doi.org/10.25007/ajnu.v9n2a818

Abstract

In this paper, we reviewed some variable selection methods in linear regression model. Conventional methodologies such as the Ordinary Least Squares (OLS) technique is one of the most commonly used method in estimating the parameters in linear regression. But the OLS estimates performs poorly when the dataset suffer from outliers or when the assumption of normality is violated such as in the case of heavy-tailed errors. To address this problem, robust regularized regression methods like Huber Lasso (Rosset and Zhu, 2007) and quantile regression (Koenker and Bassett ,1978] were proposed. This paper focuses on comparing the performance of the seven methods, the quantile regression estimates, the Huber Lasso estimates, the adaptive Huber Lasso estimates, the adaptive LAD Lasso, the Gamma-divergence estimates, the Maximum Tangent Likelihood Lasso (MTE) estimates and Semismooth Newton Coordinate Descent Algorithm (SNCD ) Huber loss estimates.

Downloads

Download data is not yet available.

References

1. Bradic, J., Fan, J. and Wang, W. (2011). Penalized composite quasi-likelihood for ultrahigh-dimensional variable selection. J. R. Stat. Soc. Ser. B, 73, 325-349.
2. Fan, J. and Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties. J. Amer. Statist. Assoc., 96 1348-1360.
3. Fujisawa, H. and Eguchi, S. (2008). Robust parameter estimation with a small bias against heavy contamination, J. Multivar. Anal., 99(9), 2053-2081.
4. Hoerl, A.E. and Kennard, R.W. (1970). Ridge Regression: Biased Estimation for Nonorthogonal Problems. Technometrics , 12 (1), 55–67.
5. Kawashima, K., Fujisawa, H. (2017). Robust and Sparse Regression via γ Divergence. J. of entropy and infor. studies 19(11).
6. Koenker, R. (2004). Quantile regression for longitudinal data. J. Multivar. Anal. 91,74–89.
7. Koenker, R. (2013). quantreg: Quantile Regression. R package version 5.05. R Foundation for Statistical Computing: Vienna) Available at: http://CRAN. R-project. org/package= quantreg.
8. Koenker, R. and G. W. Bassett (1978). Regression quantiles. Economerrica 46, 33–50
9. Lambert-Lacroix, S. and Zwald, L. (2011). Robust regression through the Huber’s criterion and adaptive Lasso penalty. Electronic J. of Statist. 5,
10. Ogutu, J.O., T. Schulz-Streeck, and H.P. Piepho. (2012). Genomic selection using regularized linear regression models: Ridge regression, Lasso, elastic net and their extensions. BMC Proceedings 6:S10.
11. Qin, Y., Li, S. and Yu, Y. (2017). Penalized Maximum Tangent Likelihood Estimation and Robust Variable Selection.https://arxiv.org/pdf/1708.05439.pdf.
12. Rosset, S. and Zhu, J. ( 2007). Piecewise linear regularized solution paths. The Annals of Statist. 35 (3), 1012–1030.
13. Tibshirani, R. (1996). Regression shrinkage and selection via the Lasso. J. R. Stat. Soc. Ser. B, 58, 267–288.
14. Wang, H., Li, G., and Jiang, G. (2007). Robust regression shrinkage and consistent variable selection through the LAD-Lasso. J. of Business & Economic Statistics 25, 347 - 355.
15. Yi, C. Huang, J. (2016). Semismooth Newton Coordinate Descent Algorithmfor Elastic-Net Penalized Huber Loss Regression and Quantile Regression. J. of Com. and Graph. Statist. 3. 547–557
16. Xu, J. and Ying, Z. (2010). Simultaneous estimation and variable selection in median regression using Lasso-type penalty. Ann. of the Inst. of Statist. Math. 62, 487–514.
17. Zou, H. (2006). The adaptive Lasso and its oracle properties. J. Amer. Statist. Assoc., 101, 1418-1429.
18. Zou, H. and Hastie, t. (2005). Regularization and variable selection via the elastic net. J. R. Stat. Soc. Ser. B 67 301-320.

Published

2020-08-20

How to Cite

Ajeel, S. M., & Hashem, H. A. (2020). Comparison Some Robust Regularization Methods in Linear Regression via Simulation Study. Academic Journal of Nawroz University, 9(2), 244–252. https://doi.org/10.25007/ajnu.v9n2a818

Issue

Section

Articles