1. Introduction
Here the usual linear regression mode is considered in the paper given by:
(1)
where
are the observations,
is a vector of unknown parameters to be estimated,
is an
matrix of
predictor vectors of
observations and
is a random error vector with
which often is assumed that it is subject to normal distribution independently. Ordinary least squares (OLS) estimates are very common which can be obtained by minimizing the sum of square residuals. In general, the criteria for evaluating the quality of a model from the following two aspects. One is prediction accuracy on test data and the other is to tend to select a simple model. In other words, less variables would be selected in the condition of same prediction effects. Variable selection is necessary especially when the number of predictors is large. There are many applications using variable selection to solve problems like knowledge discovery with high-dimensional data source [1] and it could greatly enhance the pre- diction performance of the fitted model. Traditional model selection method is best-subset selection and its step-wise variants. However, best-subset selection is computationally prohibitive when the number of predictors is large. As analyzed by Breiman (1996) [2] , subset selection is unstable; thus, the resulting model has poor prediction accuracy. To overcome the drawbacks of subset selection, many variable selection methods are appeared and the most popular recently is regulation method.
In recent years, regularization method has attracted a great attention. It is used in applications such as machine learning, denoising, inpainting, deblurring, compressed sensing, source separation and more. Generally, the square loss function of penalized least squares estimates:
(2)
where
is a penalty parameter.
is considered in this paper.
. When
, it becomes Lasso procedure. The procedure of
minimizing the objective function is called ridge regression when
. As a continuous shrinkage method, ridge regression achieves its better prediction performance through a bias-variance trade off. However, ridge regression can not produce parsimonious model, which means all the predictors are kept in the model. Lasso is proposed by [3] in 1996. It imposes the
norm regularization to the loss function and becomes a widely popular regularization method. Further, Knignt and Fu (2000) [4] studied the asymptotic properties of lasso. Lasso can shrink some coefficients to zero to achieve the effects of variable selection. Due to this reason, lasso has gained popularity in high-dimensional data.
Although lasso is a popular method for variable selection, it still has several drawbacks. The first is the lack of oracle property. The oracle property means the probability of selecting the right set of nonzero coefficients converges to one, and the estimators of the nonzero coefficients have asymptotically normal distribution with the same means and covariances. Fan and Li (2001) [5] first pointed the parameters estimators of lasso are biased and don’t have the oracle property. Then adaptive lasso proposed by Zou (2006) [6] overcomes this limitation of lasso. In adaptive lasso, weights are used for regularizing different coefficients in the
norm regularization, which means different coefficients have different shrinkage factors. Secondly, lasso have very poor performance when there are highly correlated variables in high-dimensional data. What's more, lasso only selects
variables at most when the number of variables
is more than the number of sample observation size
, which is a big limitation in application areas. At last, there are grouping variables among genes for the microarray gene data. These genes mostly have a high pairwise correlation with each other while lasso can not select a group of correlated genes. Based on these limitations, Zou and Hastie [7] proposed the elastic net regularization which is a combination of
norm and
norm. Similarly, elastic net (ENET) lacks the oracle property even though it outperforms Lasso. Further, Zou and Zhang [8] proposed adaptive elastic net to achieve the oracle property and good estimation accuracy. In the base of that, Ghosh [9] further studied grouping effects in the adaptive elastic net by using the ordinary least squares as the initial weight in low dimension data for simplicity. However, the estimators of ordinary least squares is very bad in the case of large numbers predictors or high correlated variables. Additionally, elastic net and adaptive elastic net don’t take into account the information about the correlations of variables.
In the same spirits, there existed other penalty based on methods for handling grouping effects. Penalizing least squares via combining
and
named OSCAR is presented by Bondell and Reich (2008) [10] . The Oscar forces some coefficients to be identically equal, encouraging correlated variables that have similar effects on the response to form clusters represented by the same coefficients. However, the computation of Oscar estimation is slow for large
which is based on a sequential quadratic programming. Then, considering the infor- mation of correlations of variables, Tutz and Ulbricht (2009) researched the property of correlation-based penalty (CP) and pointed the grouping effect of CP term. In the article of Tutz and Ulbricht, blockwise boosting procedure is applied in the simulations, which updates at each step the coefficient of more than one variable. However, in practical implementation, the step length factor and the stopping number of iterations have to be determined. Sometimes, this may be difficult and affects the sparsity of the solution as well as the speed of convergence of the algorithm. Therefore, El Anbari and Mkhadri (2014) [11] proposed an alternative regularization procedure called L1CP by combing
norm with CP term. The method performs automatic variable selection and has the ability of grouping effects.
In this paper, motivated by the sparsity and grouping effect especially the case of the pairwise correlations are very high, a new regulation procedure called LqCP is proposed in linear regression setting. It combines the
norm and CP penalty. Similar to the L1CP method, LqCP also performs automatic variables selection and allows to select or to remove highly correlated variables together. Section 2 first introduces the property of elastic net and adaptive elastic net and then demonstrates LqCP and its algorithm as well as the method of choosing regularization parameters. In Section 3, simulation studies give the estimation mean squared errors for different circumstances to show the parameters estimation effects among LqCP, elastic net (ENET), adaptive elastic net (AENET) and L1CP. Section 4 is mainly about the application examples to show the prediction accuracy of models. The conclusion of this paper is given in Section 5.
2. Methods
2.1. The Elastic Net and Adaptive Elastic Net
Here the form of elastic net described in the above firstly is showed in the following. The naive elastic net estimator
is the minimizer of equation:
(3)
This method is called the naive elastic net which overcomes the limitations of Lasso in the case of
. The penalty of combining
norm and
norm which has been proved that it has grouping effect in Zou and Hastie (2005) [7] . The estimator of elastic net is a two-stage procedure: for each fixed
, they first find the ridge coefficients, and then do the Lasso-type shrinkage along the lasso coefficient solution paths. At last, through the criterion of model selection, the best
and
are obtained. It appears to incur a double amount of shrinkage. Double shrinkage does not help to reduce variances much and introduces unnecessary extra bias, compared with pure lasso or ridge shrinkage. Hence, the elastic net estimators are rescaled by
. Such a scaling transformation preserves the variable selection property of the naive elastic and improves the prediction performance by correcting this double shrinkage.
In a similar way to Lasso, the elastic net does not enjoy the oracle property. Combining the property of the adaptive Lasso, Zou and Zhang [8] proposed the adaptive elastic net which combines the adaptive Lasso and
norm. Ad- ditionally, they established the oracle property of the adaptive elastic net in the condition of weak regularity. The form of the adaptive elastic net is as follows
(4)
where
, and
is a positive constant. Choosing the
initial weight is crucial in the adaptive elastic net. Zou and Zhang [8] proposed using the elastic net as an initial weight either in low-dimensional data or high-dimensional data. In the paper of Ghosh [9] , the weights is donated by
. Ghosh [9] researched the ability of variable selection and grouping effect problem, proved that the adaptive elastic net also can automatically select the group variables and also had a better performance for prediction accuracy than elastic net. But the estimator of ordinary least squares is very bad in the case of high dimension or high correlations of variables. Therefore, based on the paper of Zou [8] , the elastic net estimators is regarded as the initial weight in simulations in this paper. To avoid the invalid values when the estimators
, the initial weights are defined by
, where
is the sample size and
. For adaptive methods, while as
it becomes a biased procedure like lasso. Similarly for
adaptive methods are un- biased. A good choice of
is a compromise between shrinkage and bias and can be obtained via cross-validation. As described in Ghosh,
should not be chosen too big a quantity and typically first should be given a range of values. Then for each fixed
, the cross-validation is applied to select other tuning parameters. But the elastic net and adaptive elastic net does not take into account the correlations structure of variables. Therefore, according to the CP term proposed by Tutz and Ulbricht (2009) [12] , proposed new method is as follows.
2.2. The Proposed Method
2.2.1. Introduction of Proposed Model
In the context of linear regression problems, the following penalty function based on residual sum squares is considered. The
penalty on
is defined as
(5)
When
, the corresponding penalty is discontinuous at the origin and consequently is not easy to compute. Thus in this paper
is designed. The least squares subject to the
penalty with
is first studied by Frank and Friedman (1993) [13] which is known as bridge regression. Fu (1998) [14] and Knight and Fu (2000) [4] studied asymptotic properties and the com- putation of bridge estimators. When
, the solution
never becomes zero unless
and it is biased. For
, the bridge estimator tends to shrink small absolute coefficients to exact zeros and hence selects important variables. As pointed out by Theorem 2 in Knight and Fu (2000) [14] , when
the amount of shrinkage towards zero increases with the magnitude of the regression coefficients being estimated. It suggests that if
, estimate nonzero regression parameters at the usual rate without asymptotic bias while shrinking the estimates of zero regression parameters to 0 with positive pro- bability. In practice, in order to avoid unacceptable large bias for large para- meters, the value of
is often chosen not too large. When
, the
penalty may achieve better sparsity than
penalty because larger penalty is imposed on small coefficients than
penalty.
So according to better sparsity property of
and the case of high correlations of variables, the model is defined by
(6)
where
(7)
,
,
are positive constants.
denotes the (empirical) corre- lation between the ith and jth predictors. Here the penalty
is introduced by Tutz and Ulbricht (2009) [12] and is defined assuming that
for
. When
, the model changes to the combination of
and
, which is the model called L1CP proposed by El Anbari M. and Mkhadri A. (2014) [11] .
As introduced in the above,
penalty in the loss of residual square sum have a better sparsity, while the correlation-based penalty
will en- courage grouping effect for highly correlated variables. In fact, it’s easy to see that for strong positive correlation (
) in (7), the first term becomes the dominant having the effect that estimates for
and
are similar (
). For strong negative correlation (
), the second term becomes dominant and
will be close to
. The effect is grouping, highly correlated effects show comparable values of estimates (
) with the sign being determined by positive or negative correlation.
Moreover, assuming that
for
, the penalty (7) can be written in a sample quadratic form
, where
, is a positive definite matrix with general term
(8)
Putting
,the optimization problem (6) is equivalent to
(9)
Proof of (8): In fact, the penalty
can be written as follows:
(10)
where
. So, the problem (6) is equivalent to finding
(11)
which is equivalent to the optimization problem (6).
2.2.2. The Algorithm Procedure
The estimators of
can be computed via the Cyclic Descent Algorithm for
sparsity penalized linear regression problem [15] . The main idea is to transform the LqCP problem into an equivalent problem on augmented data.
Indeed, the optimization problem (6) can be written as
(12)
where
is defined by (8), is a real symmetric positive-define square matrix,
assuming that
with Choleski decomposition
and
,
which always exists. Now, let
(13)
The LqCP estimator is defined as
(14)
Let
(15)
Note that the sample size in the augmented problem is
and
has rank
. As described in the paper of Marjanovic and Solo [15] in 2014,
sparsity penalizing linear regression can make a better sparsity result and a quick rate of convergence than
norm. Whats more, they proposed a new algorithm called Cyclic Descent and have proved that minimizing the problem of sparsity penalized
in linear regression has global minimizers. In the base of that, the algorithm is used to solve the Equation (15) and the procedure is as follows:
(16)
Here, the related values are defined that
,
,
. Then,
(17)
Through the whole procedure, calculating the model (12) has been trans- formed to (15). Then, the algorithm procedure is stated below:
Initialize with
and calculate the initial residual
. Denote the iteration counter by
and the coefficient index of an iterate by
. Set
, and:
(a) Calculate the adjusted gradient
by:
(18)
(b) Use (a) computing the map:
(19)
where
,
and
satisfies
. There are two solutions to this equation and
is the larger one. It can be computed from the iteration,
:
(20)
(c) Update
by:
(21)
where
has a 1 in the k-th position and 0's in the rest.
(d) Update the residual
with:
(22)
(e) Update the iteration counter
by
.
(f) Update the coefficient index
by:
(23)
(g) Go to (a)
2.2.3. Selection of Tuning Parameters
In practice, it is important to select appropriate tuning parameters in order to obtain a good prediction precision or estimation precision. There are three parameters
which need to be chosen. As mentioned in the previous section, how to choose a proper
is important, which depends on the nature of data. If sparsity of model is the point of focus, smaller
tends to be proper. The aim of this paper is not only to research the sparsity of variables but also to study grouping effect about strongly correlated variables. Therefore, the best
should be chosen by experiment data and cross-validation.
Firstly,
is given a grid of values to be compared. The choice of
will be different for different
. For each fixed
, cross-validation is used to choose the best
. Typically, given a grid of values for
, 5-fold cross-validation is applied to getting the best
in terms of minimized mean squared error of models or prediction error for each fixed
. In simulation studies, the mean square error are defined by
(24)
where
is approximated by the sample covariance matrix of
of out-of-sample data. The formula of MSE is mentioned by Tibshirani [3] can measure the estimation error of model for simulation studies. In real data, because of unknown true parameters, test error defined by
(25)
can measure prediction error of model. At last, the chosen value
is the best to compare with other models.
3. Simulation Studies
In this section, simulation studies are presented the finite sample performance of LqCP. The results analysis are considered from variable selection ability, the estimation errors, grouping effect. Data is generated from the true model:
(26)
is the response variable and
is an
matrix with
predictor vectors and
observations.
is a random error vector with
.
is
dimension parameters and
expresses the volatility of
. Three methods in the simulations study: the elastic net (ENET), the adaptive elastic net (AENET) and the L1CP are listed to be compared. Because these methods have the ability of grouped variable selection. Data is divided into two data sets: training data and testing data. Training data is used to do model fitting and cross-validation. Testing data is used to evaluate the error of models. For each estimator
, its estimation accuracy is measured by the mean squared error (MSE) in the testing data. The variables selection performance is gauged by C and IC, where C is the number of zero coefficients that are correctly estimated by zero and IC is the number of non-zero coefficients that are incorrectly estimated by zero. In addition, the algorithm’s stopping criterion is
. In these simulation studies,
is given five different values
which are
. The results in the following tables are chosen
by comparing MSE among different
. ENET and AENET are computed by “glmnet” package and L1CP is computed by “lars” package in R language .
Example 1. 100 data sets are generated with sample size 100 observations from the linear regression model
, The true regression coefficients vector
is
. Different dimension levels are considered as below. Smaller value
is more proper and can make all results stable through the simulation experiment. The correlation matrix is given by
. The observation sample size of training data is 70 and testing data is 30. Furthermore, considering the problem of sparsity, noise variables increase by increasing the number of zero in
to generate four different dimensional data. The dimensions of data are respectively 10 (30%), 30 (10%), 60 (5%), 100 (3%). The number of noise variables are respectively 7, 27, 57, 97. That is to say, this part concerns that how the results change with the increase of noise variables. The simulation results are in Table 1 and Table 2.
Example 2. This example is about the case of
. Similar to the example of Wu, Shen and Geyer (2009), corresponding to high dimensional simulation scenarios with correlated groups, large
and small
, the
are simulated from
, where the jk-th element of
is
and
.
So there are 20 grouped relevant predictors and 180 noise predictors and
. Also, 100 data sets are generated and split data 70/30 into two parts for training data and testing data.
Example 3. About the grouping effect, 100 sample size described by 40 pre- dictors is considered. The true parameters are
Table 1. Median mean squared errors of four methods based on 100 data replications with the standard errors estimated by using the bootstrap with B = 500 resamplings on the 100 mean squared errors for different numbers of noise variables.
Table 2. Median number of C and median number of IC based on 100 data replications for different numbers of noise variables. The number of true noise variables are 7, 27, 57, 97.
and
which is selected to see the performance of models in bigger volatility. The predictors are generated as:
and
and
and
are independently identically distributed
for
, where
are independently distributed
for
. In this data, three equally important groups have pairwise correlation
, and there are 25 pure noise features. Also, the data is split as 70 observations for training data and 30 observations for testing data.
From Table 1, It is anticipated that MSE of all methods increase with the increase of noise variables. In 30% and 10% levels, AENET is better than other methods. But in in 5% and 3% levels, LqCP has the minimized MSE, which indicates LqCP performs better in low sparsity levels. In addition, LqCP is much better than ENET and L1CP for all levels. Table 2 illustrates that LqCP can estimate all true zero coefficients and the number of incorrect selection for true non-zero coefficients is always 0 with the increase of noise variables. Especially relative to ENET and L1CP, both of them can not select all true zero coefficients in four circumstances.
In the case of
showed in Table 3, LqCP performs best and are more stable. As expressed in the example, the number of true noise variables is 180 and the result from LqCP is 179, which achieves a better selection of zero coefficients than other methods. From Table 4, LqCP performs with the minimized MSE and the minimized standard error which also has a better variable selection ability for true non-zero coefficients and zero coefficients in case of high correlations
. It states better estimation effect of para- meters of LqCP for grouping effect than other methods.
4. Real Data Sets Experiment
This part is about the performances of LqCP for two real world data sets: the US Crime and Gasoline described by
and
explanatory predictors respectively. The dimension
of US Crime data set is smaller than the sample
Table 3. Median mean squared errors of four methods based on 100 data replications with the standard errors estimated by using the bootstrap with B = 500 resamplings on the 100 mean squared errors. The dimension is 200 and the number of true noise variables is 180.
Table 4. Median mean squared errors of four methods based on 100 data replications with the standard errors estimated by using the bootstrap with B = 500 resamplings on the 100 mean squared errors. The number of true noise variables is 25.
size (
), while the number of variables of Gaoline exceeds largely the sample size
. Because the true parameters in application is unknown and the concern is prediction accuracy of response variable. Test Error is mentioned in the above as the criterion comparing among models. The selection of
is
also based on the minimized test error for
.
4.1. US Crime Data
This data set is taken from R package “MASS” which contains 47 observations and 15 variables as well as one response variable. Criminologists are interested in the effect of punishment regimes on crime rates which has been studied using aggregate data on 47 states of the USA for 1960. This data set contained the following 16 columns: percentage of males aged 14 - 24 (M), indicator variable for a Southern state (So), mean years of schooling (Ed), police expenditure in 1960 (Pol), police expenditure in 1959 (Po2), labor force participation rate (LF), numbers of males per 1000 females (M.F), state population (Pop), number of non-whites per 1000 people (NW), unemployment rate of urban males 14 - 24 (U1), unemployment rate of urban males 35 - 39 (U2), gross domestic product per head (GDP), income inequality (Ineq), probability of imprisonment (Prob), average time served in state prisons (Time) and the outcome is the rate of crimes in a particular category per head of population (y). The data is split 100 times with a training set of 24 observations and a test set of 23 observations. The results are listed in Table 5.
Clearly, Table 5 shows that LqCP selects 7 variables and has the minimized test error, which performs better than other methods followed by AENET on Crime data. Although the standard error of Test Error is not the lowest, it is very close to the standard error of AENET. The selected variables are percentage of males aged 14 - 24 (M), mean years of schooling (Ed), police expenditure in 1960 (Pol), numbers of males per 1000 females (M.F), number of non-whites per 1000 people (NW), income inequality (Ineq), probability of imprisonment (Prob). These variables can also be selected by other methods. Seeing from prediction accuracy and sparsity effect of models, LqCP is the best.
4.2. Gasoline Data
This data set “Gasoline” comes from R package “pls”. It is about infrared spectrum, which contains 69 observations. Recently, infrared spectrum is based on the function of diffuse reflecting degree measured by interval 2 nm from 900 nm to 1700 nm. Gasoline data have 401 prediction variables and the correlations of variables are very high that are almost 0.99. Similarly, the data set is split 100 times into a training set of 40 observations and a test set of 29. The prediction results are reported in Table 6.
In the circumstance of high dimension (
) and small sample obser- vations size(
), LqCP is the winner in term of test error, which also gets the least number of variables. Significantly, in this application, the correlation of variables is very high and approaches 1. The result shows that AENET, L1CP and LqCP have similar variable selection effect but ENET is the worst. Therefore, this application proved the efficiency of LqCP from the aspect of
and highly correlated variables.
5. Conclusion
In this paper, motivated by variable selection and grouped selection property in linear regression problems, a new method called LqCP is proposed, which is a regularization procedure based on the penalized least squares with a mix of
norm and correlation-based penalty. Firstly, this paper discusses the current models that have grouping effect including elastic net, adaptive elastic net and L1CP. Similar to them, LqCP can also encourage grouping effect, where strongly correlation among predictors tend to be in or out of the model together. Through the simulation studies in the above, LqCP has better per- formances in terms of variable selection ability with large numbers of noise
Table 5. US Crime Data-Median test errors of four methods based on 100 random splits with the standard errors estimated by using the bootstrap with B = 500 resamplings on the 100 test errors. The median number of selected variables by each method is also reported.
Table 6. Gasoline Data-Median test errors of four methods based on 100 random splits with the standard errors estimated by using the bootstrap with B = 500 resamplings on the 100 test errors. The median number of selected variables by each method is also reported.
variables, high dimension
and grouped variable selection automatically for high correlations of variables. Additionally, two real data proved LqCP’s efficiency from the aspects of
and
through comparing pre- diction error with other models. Moreover, the oracle property is important in statistics area. One of future works is to pay attention to prove the oracle property of LqCP and also LqCP can be expended to general regression like logistic regression, quantile regression to solve some regression problems or multi-calss classification problems especially for data with highly correlated variables.