Model-Free Feature Screening Based on Gini Impurity for Ultrahigh-Dimensional Multiclass Classification

Zhongzheng Wang¹, Guangming Deng^1,2*

¹College of Science, Guilin University of Technology, Guilin, China.
²Applied Statistics Institute, Guilin University of Technology, Guilin, China.

DOI: 10.4236/ojs.2022.125042                

Abstract

It is quite common that both categorical and continuous covariates appear in the data. But, most feature screening methods for ultrahigh-dimensional classification assume the covariates are continuous. And applicable feature screening method is very limited; to handle this non-trivial situation, we propose a model-free feature screening for ultrahigh-dimensional multi-classification with both categorical and continuous covariates. The proposed feature screening method will be based on Gini impurity to evaluate the prediction power of covariates. Under certain regularity conditions, it is proved that the proposed screening procedure possesses the sure screening property and ranking consistency properties. We demonstrate the finite sample performance of the proposed procedure by simulation studies and illustrate using real data analysis.

Keywords

Ultrahigh-Dimensional, Feature Screening, Model-Free, Gini Impurity, Multiclass Classification

1. Introduction

Ultrahigh-dimensional data are commonly available in a wide range of scientific research and applications. Feature screening plays an essential role in the ultrahigh-dimensional data, where Fan and Lv [1] first proposed the sure independence screening (SIS) in the seminal paper. For linear regressions, they showed that the approach based on Pearson correlation learning possesses a sure screening property. That is, even if the number of predictors p can grow much faster than the number of observations n with $\log p=O\left(n^{\alpha }\right)$ for some

$\alpha \in \left(\mathrm{0,}\frac{1}{2}\right)$, all relevant predictors can be selected with probability tending to one [2].

Lots of feature screening is the Model-based and Model-free approaches have been developed in recent years, see, for example, Wang [3] proposed forward regression for ultrahigh-dimensional data. Fan and Song [4] applied the maximum marginal likelihood estimates or the maximum marginal likelihood to ultrahigh-dimensional screening in generalized linear model. Fan et al. [5] further extend the correlation learning to marginal nonparametric learning. Zhu et al. [6] proposed a model-free feature screening approach for ultrahigh-dimensional data. Li et al. [7] proposed a robust rank correlation screening method to deal with ultrahigh-dimensional data based on the Kendall $\tau$ correlation coefficient. Li et al. [8] applied the distance correlation to sure independence screening procedure. He et al. [9] proposed a quantile-adaptive framework for nonlinear variable screening with high-dimensional heterogeneous data. Fan et al. [10] proposed nonparametric independence screening selects variables by ranking a measure of the nonparametric marginal contributions of each covariate given the exposure variable. Liu et al. [11] proposed a feature screening procedure for varying coefficient model based on conditional correlation coefficient. Nandy et al. [12] proposed a covariate information number sure independence screening, which used a marginal utility connected to the notion of the traditional Fisher information. Pouyap et al. [13] proposed a merge of the features selection methods in order to define the most relevant features in the texture of the vibration signal images.

To address the ultrahigh-dimensional feature screening in classification problem, Fan and Fan [14] proposed the t-test statistic for two-sample mean problem as a marginal utility for feature screening and establish its theoretical properties. Mai and Zou [15] applied the Kolmogorov filter to ultrahigh-dimensional binary classification. Cui et al. [16] proposed a screening procedure via used empirical conditional distribution functions. Lai et al. [17] proposed a feature screening procedure based on the expected conditional Kolmogorov filter for binary classification problem. However, the above-proposed screening methods assume that the types of data are continuous. For categorical covariates, Huang et al. [18] constructed a model-free discrete feature screening method based on the Pearson Chi-square statistics and showed its sure screening property fulfilling (Fan et al. [2]). When all the covariates are binary, Ni and Fang [19] proposed a model-free feature screening procedure based on information entropy theory for multi-class classification. Ni et al. [20] further proposed a feature screening procedure based on weighting Adjusted Pearson Chi-square for multi-class classification. Sheng and Wang [21] proposed a new model-free feature screening method based on classification accuracy of marginal classifiers for ultrahigh-dimensional classification. Anzarmou et al. [22] proposed a new model-free interaction screening method, termed Kendall Interaction Filter (KIF), for the classification in high-dimensional settings.

Based on the above study of classification models, in this paper, we propose a model-free feature screening for ultrahigh-dimensional multi-classification with both categorical and continuous covariates. The proposed feature screening method will be based on Gini impurity to evaluate the prediction power of covariates. Gini impurity is a non-purity attribute splitting index, which was proposed by Breiman et al. [23] and has been widely used in decision tree algorithms such as CART and SPRINT. With regard to categorical covariate screening, we can apply the index of purity gain, which is the same as information gain [19]. Similar to Ni and Fang [19], continuous covariates can be sliced via standard normal quantile. The proposed feature screening procedure is based on purity gain, which is referred to Purity Gain sure independence screening (PG-SIS). Theoretically, the PG-SIS is rigorously proven to enjoy. Fan and Lv [1] proposed sure screening property that ensures all important features can be obtained. Practically, as shown by the simulation results, compared with the existing feature screening method, PG-SIS satisfies the sure screening property.

This paper is organized as follows. Section 2 describes the proposed PG-SIS method in detail. Section 3 establishes its sure screening property. In Section 4, numerical simulations and an example for real data analysis are given to assess sure screening property of our method. Some concluding remarks are given in Section 5 and all the proofs are given in the Appendix.

2. Feature Screening Procedure

We first introduce Gini impurity and purity gain, and then propose the screening procedure based on purity gain.

2.1. Gini Index and Purity Gain

Suppose that Y is a categorical response with R classes $\left\{\mathrm{1,}\cdots \mathrm{,}R\right\}$, and covariate $X=\left(X_1\mathrm{,}{X}_2\mathrm{,}\cdots \mathrm{,}{X}_p\right)$ is a vector of p dimension, where each of these components $X_k$ with $J_k$ categories. Where $J_k=\left\{\mathrm{1,}\cdots \mathrm{,}J\right\}$. To introduce the Gini impurity and purity gain, assuming that $Y\in \left\{\mathrm{1,}\cdots \mathrm{,}R\right\}$ and $X_k\in \left\{\mathrm{1,2,}\cdots \mathrm{,}{J}_k\right\}$. Define $p_r=P\left(Y=r\right)$ represents the probability function of a response variable, $w_{k,j}=P\left(X_k=j\right)$ represents the probability function of covariates, $p_{k,jr}=P\left(Y=r|X_k=j\right)$ represents the probability function of response variables under the condition of covariates, where $r=1,2,\cdots ,R;j=1,2,\cdots ,J_k$ and $k=1,2,\cdots ,p$. Let $0\times \log 0=0$. Marginal Gini impurity of Y and X respectively is defined as

$Gini\left(Y\right)=1-{\displaystyle \underset{r=1}{\overset{R}{\sum }}}{p}_r^2$ (1)

$Gini\left(X_k\right)=1-{\displaystyle \underset{j=1}{\overset{J_k}{\sum }}}{w}_{k,j}^2$ (2)

Conditional Gini impurity is defined as

$Gini\left(Y|X_k\right)={\displaystyle \underset{j=1}{\overset{J_k}{\sum }}}{w}_{k,j}\left(1-{\displaystyle \underset{r=1}{\overset{R}{\sum }}}{p}_{k,jr}^2\right)$ (3)

Similar to the information gain, the purity gain is defined as

$\begin{array}{c}PG\left(Y|X_k\right)=Gini\left(Y\right)-Gini\left(Y|X_k\right)\\ =1-{\displaystyle \underset{r=1}{\overset{R}{\sum }}}{p}_r^2-{\displaystyle \underset{j=1}{\overset{J_k}{\sum }}}{w}_{k,j}\left(1-{\displaystyle \underset{r=1}{\overset{R}{\sum }}}{p}_{k,jr}^2\right)\end{array}$ (4)

In the Equation (1), $Gini\left(Y\right)$ is non-negative and acquires its maximum $1-\frac{1}{R}$ if and only if $p_1=\cdots =p_R=\frac{1}{R}$ by Jensen’s inequality [24]. And the $Gini\left(Y|X_k\right)$ in Equation (2) is the conditional Gini impurity of Y given $X_k=j$. Further support can be given by the following proposition.

Proposition 2.1. When $X_k$ is a categorical covariable, we can get $PG\left(Y\mathrm{<mo>\; |\; </mo>}{X}_k\right)\ge 0$, and $X_k$ and Y are independent if and only if $PG\left(Y\mathrm{<mo>\; |\; </mo>}{X}_k\right)=0$ .

For continuous $X_k$, the conditional Gini impurity can’t directly calculate, and purity gain by slicing X into several categories. For a fixed integer $J\ge 2$, let $q_{\left(j\right)}$ be the j/J^th percentile of X, $j=1,\cdots ,J-1$, $q_{\left(0\right)}=-\infty$ and $q_{\left(J\right)}=+\infty$. Replacing $w_{k\mathrm{,}j}$ and $p_{k\mathrm{,}jr}$ in Equation (3) respectively by

$w_{k,j}=P\left(X_k\in \left(q_{\left(j-1\right)},q_{\left(j\right)}\right]\right)$ and $p_{k,jr}=P\left(Y=r|X_k\in \left(q_{\left(j-1\right)},q_{\left(j\right)}\right]\right)$, we define conditional Gini impurity based on continuous covariates:

$\begin{array}{c}Gin{i}_J\left(Y|X_k\right)={\displaystyle \underset{j=1}{\overset{J_k}{\sum }}}{w}_{k,j}\left(1-{\displaystyle \underset{r=1}{\overset{R}{\sum }}}{p}_{k,jr}^2\right)\\ ={\displaystyle \underset{j=1}{\overset{J_k}{\sum }}}P\left(X_k\in \left(q_{\left(j-1\right)},q_{\left(j\right)}\right]\right)\left(1-{\displaystyle \underset{r=1}{\overset{R}{\sum }}}P{\left(Y=r|X_k\in \left(q_{\left(j-1\right)},q_{\left(j\right)}\right]\right)}^2\right)\end{array}$ (5)

$\begin{array}{c}PG\left(Y|X_k\right)=Gini\left(Y\right)-Gini\left(Y|X_k\right)\\ =1-{\displaystyle \underset{r=1}{\overset{R}{\sum }}}{p}_r^2-{\displaystyle \underset{j=1}{\overset{J_k}{\sum }}}{w}_{k,j}\left(1-{\displaystyle \underset{r=1}{\overset{R}{\sum }}}{p}_{k,jr}^2\right)\\ =1-{\displaystyle \underset{r=1}{\overset{R}{\sum }}}{p}_r^2-{\displaystyle \underset{j=1}{\overset{J_k}{\sum }}}P\left(X_k\in \left(q_{\left(j-1\right)},q_{\left(j\right)}\right]\right)\\ \times \left(1-{\displaystyle \underset{r=1}{\overset{R}{\sum }}}P{\left(Y=r|X_k\in \left(q_{\left(j-1\right)},q_{\left(j\right)}\right]\right)}^2\right)\end{array}$ (6)

Proposition 2.2. When $X_k$ is a continuous covariable, we can get $P{G}_J\left(Y\mathrm{<mo>\; |\; </mo>}{X}_k\right)\ge 0$, and $X_k$ and Y are independent if and only if $P{G}_J\left(Y\mathrm{<mo>\; |\; </mo>}{X}_k\right)=0$.

2.2. Feature Screening Procedure Based on Purity Gain

First, we select a medium scale of simplified model which can almost fully contain D, where $D=\{k\mathrm{<mo>\; :\; </mo>}F\left(Y\mathrm{<mo>\; |\; </mo>}x\right)$ functionally depends on $X_k$ for some $Y=r\}$, we use an adjusted purity gain index for each pair $\left(Y\mathrm{,}{X}_k\right)$ is as follows:

$e_k=\frac{1-{\displaystyle {\sum }_{r=1}^R{p}_r^2}-{\displaystyle {\sum }_{j=1}^{J_k}{w}_{k,j}}\left(1-{\displaystyle {\sum }_{r=1}^R{p}_{k,jr}^2}\right)}{\log J_k}$ (7)

where $p_r=P\left(Y=r\right)$, $w_{k,j}=P\left(X_k=j\right)$ and $p_{k,j_r}=P\left(Y=r|X_k=j\right)$ when $X_k$ is categorical, $J_k$ represents the number of categories of $X_k$. When $X_k$ is defined as a continuous covariates, $J_k$ represents the number of slices applied to $X_k$, $w_{k,j}=1/J_k$, $p_{k,jr}=P\left(Y=r|X_k\in \left(q_{k,\left(j-1\right)},q_{k,\left(j\right)}\right]\right)$ and $q_{k,\left(j\right)}$ represents j/J^th percentile of $X_k$.

There may be more categories of covariates associated with larger purity gain in the original definition of Equation (4), regardless of whether the covariates are important, especially when the number of categories involved in each covariate is different. So Ni and Fang [19] used $\log J_k$ to construct the information gain ratio to solve this problem, where each category of $X_k$ is the same. Similarly, when each category of $X_k$ is the same, for Equation (7), we apply the $\log J_k$ to build an adjusted purity gain index to address the problem, which is also applied to continuous $X_k$. However, each category of $X_k$ is different, $1-{\displaystyle {\sum }_{j=1}^{J_k}}{w}_{k,j}^2$ is defined as an adjustment factor, which is motivated by the split $X_k$ into several categories via the Decision Tree algorithm.

For sample data $\left\{x_{i1}\mathrm{,}\cdots \mathrm{,}{x}_{ip}\mathrm{,}{y}_i\right\}$, $i=1,\cdots ,n$, $e_k$ can be easily estimated by

${\hat{e}}_k=\frac{\left(1-{\displaystyle {\sum }_{r=1}^R{\hat{p}}_r^2}\right)-{\displaystyle {\sum }_{j=1}^{J_k}{\hat{w}}_{k,j}}\left(1-{\displaystyle {\sum }_{r=1}^R{\hat{p}}_{k,jr}^2}\right)}{\log J_k}$ (8)

When $X_k$ is categorical, ${\hat{w}}_{k,j}=\frac{1}{n}{\displaystyle {\sum }_{i=1}^n}I\left\{x_{ik}=j\right\}$ and ${\hat{p}}_{k,jr}=\frac{{\displaystyle {\sum }_{i=1}^n}I\left\{y_i=r,x_{ik}=j\right\}}{{\displaystyle {\sum }_{i=1}^n}I\left\{x_{ik}=j\right\}}$.

When $X_k$ is continuous,

${\hat{w}}_{k,j}=\frac{1}{n}{\displaystyle {\sum }_{i=1}^n}I\left\{x_{ik}\in \left({\hat{q}}_{k,\left(j-1\right)},{\hat{q}}_{k,\left(j\right)}\right]\right\}$

and

${\hat{p}}_{k,j_r}=\frac{{\displaystyle {\sum }_{i=1}^n}I\left\{y_i=r,x_{ik}\in \left({\hat{q}}_{k,\left(j-1\right)},{\hat{q}}_{k,\left(j\right)}\right]\right\}}{{\displaystyle {\sum }_{i=1}^n}I\left\{x_{ik}\in \left({\hat{q}}_{k,\left(j-1\right)},{\hat{q}}_{k,\left(j\right)}\right]\right\}}$

where ${\hat{q}}_{k,\left(j\right)}$ is the j/J^th sample normal percentile of $\left\{x_{ik}\mathrm{,}\cdots \mathrm{,}{x}_{nk}\right\}$. In either case, ${\hat{p}}_r=\frac{1}{n}{\displaystyle {\sum }_{i=1}^n}I\left\{y_i=r\right\}$.

We suggest selecting a sub-model: $\hat{D}=\left\{k:{\hat{e}}_k\ge c{n}^{-\tau },1\le k\le p\right\}$. Where both c and $\tau$ are predetermined thresholds established via Condition (C2) in Section 3. In practice, we can choose a model:

$\hat{D}=\left\{k\mathrm{<mo>\; :\; </mo>}{\hat{e}}_k\text{isamongthetopof}d\text{largestofall}\right\}$

where $d=\left[n/\log n\right]$.

3. Feature Screening Property

In this section, we establish the sure screening property of PG-SIS. Based on Ni and Fang [19] proposed sure independence screening theories, the following conditions are assumed.

Condition 1 (C1). There exist two positive constants $c_1$ and $c_2$ such that, $c_1/R\le p_r\le c_2/R$, $c_1+c_2\le R$, $c_1/R\le p_{k\mathrm{,}jr}\le c_2/R$ and $c_1/J_k\le w_{k\mathrm{,}j}\le c_2/J_k$ for every $1\le j\le J_k$, $1\le r\le R$ and $1\le k\le p$.

Condition 2 (C2). There exist a positive constant $c>0$ and $0\le \tau <1/2$ such that ${\min }_{k\in D}{e}_k\ge 2c{n}^{-\tau }$.

Condition 3 (C3). $R=O\left(n^{\epsilon }\right)$, $J={\max }_{1\le k\le P}{J}_k=O\left(n^k\right)$, where $\epsilon \ge 0$, $\kappa \ge 0$ and $2\tau +2\epsilon +2\kappa <1$.

Condition 4 (C4). There exist a positive constant $c_3$, such that $0<f_k\left(x|Y=r\right)<c_3$ for any $1\le r\le R$, and x in the domain of $X_k$, where $f_k\left(x|Y=r\right)$ is the Lebesgue density function of $X_k$ conditional on $Y=r$.

Condition 5 (C5). There exist a positive constant $c_4$ and $0\le \rho <1/2$ such that $f_k\left(x\right)\ge c_4{n}^{-\rho }$ for any $1\le k\le p$ and x in the domain of $X_k$, where $f_k\left(x\right)$ is the Lebesgue density function of $X_k$. Furthermore, $f_k\left(x\right)$ is continuous in the domain of $X_k$.

Condition 6 (C6). $R=O\left(n^{\epsilon }\right)$, $J={\max }_{1\le k\le P}{J}_k=O\left(n^{\kappa }\right)$, where $2\tau +2\epsilon +2\kappa +2\rho <1$ and $\epsilon \ge \mathrm{0,}\kappa \ge 0$.

Condition 7 (C7). $\lim {\inf }_{p\to \infty }\left\{{\min }_{k\in D}{e}_k-{\max }_{k\in I}{e}_k\right\}\ge \delta$, where $\delta >0$ is a constant.

Condition (C1) guarantees that the proportion of each class of variables cannot be either extremely small or extremely large. Similar assumption is also made in condition (C1) in Huang et al. [18] and Cui et al. [16]. According to Fan and Lv [1] and Cui et al. [16], Condition (C2) allows the minimum true signal to disappear to zero in the order of $n^{-\tau }$ as the sample size goes to infinity. According to [19] Condition (C3) provides the covariates to diverge with a certain order and the number of classes for the response, and Condition (C6) modifies Condition (C3) a little bit. To ensures the sample percentiles are close to the true percentiles, Condition (C4) rules out the extreme case that some $X_k$ put heavy mass in a small range. Condition (C5) asks for the $n^{-\rho }$ as lower bound to the density. According to [16] and Zhu et al. [6] proposed ranking consistency property, we need to assume the inactive covariate subset $I=\left\{\mathrm{1,}\cdots \mathrm{,}p\right\}\backslash D$, then Condition (C7) is established.

Theorem 3.1. (Sure screening property) Under conditions (C1) to (C3), if all the covariates are categorical, we get:

$P\left(D\subseteq \hat{D}\right)\ge 1-O\left(p\exp \left\{-b{n}^{1-\left(2\tau +2\epsilon +2\kappa \right)}+\left(\epsilon +\kappa \right)\log n\right\}\right)$

Theorem 3.2. (Sure screening property) Under conditions (C4) to (C6), when the covariates are composed of continuous and categorical variables, we get:

$P\left(D\subseteq \hat{D}\right)\ge 1-O\left(p\exp \left\{-b{n}^{1-\left(2\tau +2\epsilon +2\kappa +2\rho \right)}+\left(\epsilon +\kappa \right)\log n\right\}\right)$

where b is a positive constant. If $\log p=O\left(n^{\alpha }\right)$ and $\alpha <1-\left(2\tau +2\epsilon +2\kappa +2\rho \right)$, PG-SIS has a sure screening property.

Theorem 3.3. (Ranking consistency property) Under conditions (C1), (C4), (C5) and (C7), if $\log \frac{RJ}{\log n}=O\left(1\right)$ and $\frac{\max \left\{\log P\mathrm{,}\log n\right\}{R}^4{J}^4}{n^{1-2\rho }}=o\left(1\right)$, then $\lim {\inf }_{n\to \infty }\left\{{\min }_{k\in D}{\hat{e}}_k-{\max }_{k\in I}{\hat{e}}_k\right\}>0$, a.s.

Theorem 3.3 testifies that the proposed screening index can separate active and inactive covariates well in the sample level.

4. Numerical Studies

4.1. Simulation Results

In this subsection, we carry out three simulation studies to demonstrate the finite sample performance of our group screen methods described in Section 2. We compare PG-SIS with IG-SIS [19] and APC-SIS in performance via the below evaluation criteria: MMS, minimal model size, consists of all active covariates, the results generally existing 5%, 25%, 50%, 75%, 95% of MMS; CP1, CP2 and CP3 respectively represent the probability that the given model size $\left[n/\log n\right]$, $2\left[n/\log n\right]$ and $3\left[n/\log n\right]$ cover all active covariates, while CPa indicates whether the indicators of the selected model cover all active covariates.

Model 1: categorical covariates and binary response

We first consider the response variables of different categories. According to [19], we assume a model which response $y_i$ is binary in which $R=2$, and all the covariates are categorical. We think about two distributions for $y_i$ :

1) Balanced, $P\left(y_i=r\right)=1/2$ ;

2) Unbalanced, $p_r=2\left[1+\frac{R-r}{R-1}\right]/3R$ with ${\max }_{1\le r\le R}{p}_r=2{\min }_{1\le r\le R}{p}_r$.

The true model is defined at $D=\left\{\mathrm{1,2,}\cdots \mathrm{,20}\right\}$ with $d_0=\left|D\right|=20$. Condition on $y_i$, latent variable is generated as $z_i=\left(z_{i\mathrm{,1}}\mathrm{,}\cdots \mathrm{,}{z}_{i\mathrm{,}p}\right)$, where $z_{i\mathrm{,}k}\mathrm{<mtext>\; \hspace{0.17em}\; </mtext>}N\left({\mu }_{rk}\mathrm{,1}\right)$, $1\le k\le p$. Then, we construct active covariates:

1) If $k>d_0$, then ${\mu }_{rk}=0$ ;

2) If $k\le d_0$ and $r=1$, then ${\mu }_{rk}=-0.5$ ;

3) If $k\le d_0$ and $r=2$, then ${\mu }_{rk}=0.5$.

Next, we apply the quantile of the standard normal distribution to generate covariates. The specific approach is as follows:

1) When k as odd number, that is $x_{i,k}=I\left(z_{i,k}>z_{\left(\frac{j}{2}\right)}\right)+1$ ;

2) When k as even number, that is $x_{i,k}=I\left(z_{i,k}>z_{\left(\frac{j}{5}\right)}\right)+1$ ;

Where αth percentile of the standard normal distribution is $z_{\left(\alpha \right)}$.

Thus, amongst all p covariates, the covariates of two categories and five categories account for half, respectively. Similar to [20], we consider $p=1000,5000$ and $n=200,400$ in this model.

Table 1 reports the evaluation criteria over 100 simulations for Model 1. We

Table 1. Simulation results for example 1.

can see the following: The results argue that the proposed PG-SIS works quite well. When the sample size n increases, PG-SIS is close to the true model size $d_0=20$ in MMS, and both increase to 1 in coverage probability. MMS in unbalanced response is better than in balanced response in performance via comparing the response of different structures. The performances of PG-SIS and IG-SIS are quite close, and PG-SIS is slightly better than APC-SIS in higher coverage probabilities.

Model 2: categorical covariates and multi-class response

We consider more covariate classification, and response $y_i$ is multi-class which $R=10$. We think about $y_i$ of two distributions:

1) Balanced, $P\left(y_i=r\right)=1/R$ ;

2) Unbalanced, $p_r=2\left[1+\frac{R-r}{R-1}\right]/3R$ with ${\max }_{1\le r\le R}{p}_r=2{\min }_{1\le r\le R}{p}_r$.

Among the $p=2000$ covariates, the minimum set of active covariate set is $X^D=\left\{X_{200}\mathrm{,}{X}_{400}\mathrm{,}{X}_{600}\mathrm{,}{X}_{800}\mathrm{,}{X}_{1000}\mathrm{,}\cdots \mathrm{,}{X}_{2000}\right\}$ with the number of active covariates $d_0=10$. Condition on $y_i$, latent variable is generated as $z_i=\left(z_{i\mathrm{,1}}\mathrm{,}\cdots \mathrm{,}{z}_{i\mathrm{,}p}\right)$, for covariates $X_k$, $x_{i,k}=f_k\left({\epsilon }_{i,k}+{\mu }_{i,k}\right)$, where ${\epsilon }_{i\mathrm{,}k}~N\left(\mathrm{0,1}\right)$ and $f_k(\cdot )$ represents a quantile function of standard normal distribution. Then, we construct active covariates via defining ${\mu }_{i\mathrm{,}k}$ :

1) If $X\in X^D$ and $y_i=r$, then ${\mu }_{i,k}=1.5\times {\left(-0.9\right)}^r$ ;

2) If $X\notin X^D$, then ${\mu }_{i,k}=0$ ;

Next, we apply the $f_k(\cdot )$ to generate covariates, and take $p=2000$, $n=300,400,500$ in this model. The specific approach is as follows:

1) For $1\le k\le 400$, then $f_k\left({\epsilon }_{i,k}+{\mu }_{i.k}\right)=I\left(z_{i,k}>z_{\left(\frac{j}{2}\right)}\right)+1$ ;

2) For $400<k\le 800$, then $f_k\left({\epsilon }_{i,k}+{\mu }_{i.k}\right)=I\left(z_{i,k}>z_{\left(\frac{j}{4}\right)}\right)+1$ ;

3) For $800<k\le 1200$, then $f_k\left({\epsilon }_{i,k}+{\mu }_{i.k}\right)=I\left(z_{i,k}>z_{\left(\frac{j}{6}\right)}\right)+1$ ;

4) For $1200<k\le 1600$, then $f_k\left({\epsilon }_{i,k}+{\mu }_{i.k}\right)=I\left(z_{i,k}>z_{\left(\frac{j}{8}\right)}\right)+1$ ;

5) For $1600<k\le 2000$, then $f_k\left({\epsilon }_{i,k}+{\mu }_{i,k}\right)=I\left(z_{i,k}>z_{\left(\frac{j}{10}\right)}\right)+1$ ;

Thus, amongst all the p covariates, the covariates of two categories, four categories, six categories, eight categories and ten categories account for one-fifth each.

Table 2 reports the evaluation criteria over 100 simulations for Model 2. We can see the following: Two methods in performance under Model 1 is worse than Model 2. When the model is more intricate, PG-SIS in performance is close to IG-SIS. Particularly, PG-SIS and IG-SIS have a slightly small MMS under a small sample size n. When the sample size n increases, PG-SIS is close to

Table 2. Simulation results for example 2.

$d_0=10$ in MMS, and both increase to 1 in coverage probability. Four indexes of coverage probability of APC-SIS are worse than that of PG-SIS when the sample $n=200$. MMS in unbalanced response is better than in balanced response in performance via comparing the response of different structures. Furthermore, PG-SIS and IG-SIS are more robust in performance because the fluctuation range in MMS is small.

Model 3: continuous and categorical covariates

At last, among the covariates that are both continuous and categorical, we assume a more complex example, where response $y_i$ is multi-class which $R=4$. We think about $y_i$ of two distributions:

1) Balanced, $p_r=P\left(y_i=r\right)=1/R$ ;

2) Unbalanced, $p_r=2\left[1+\frac{R-r}{R-1}\right]/3R$ with ${\max }_{1\le r\le R}{p}_r=2{\min }_{1\le r\le R}{p}_r$.

In this model, we take $p=5000$, $n=400,600,800$. The true model is defined at $X^D=\left\{X_k:k=\left[\frac{k^{\prime }p}{20}\right],k^{\prime }=1,\cdots ,20\right\}$ with $d_0=20$. Condition on $y_i$,

latent variable is generated as $z_i=\left(z_{i\mathrm{,1}}\mathrm{,}\cdots \mathrm{,}{z}_{i\mathrm{,}p}\right)$. For covariates $X_k$, $z_{i\mathrm{,}k}\mathrm{<mtext>\; \hspace{0.17em}\; </mtext>}N\left({\mu }_{i\mathrm{,}k}\mathrm{,1}\right)\mathrm{,1}\le k\le p$, where $u_{ik}={\left(-1\right)}^r{\theta }_{rk}$ when $y_i=r$ and $k\in D$. According to Ni and Fang [19], ${\theta }_{rk}$ is given in Table 3. ${\mu }_{i,k}=0$ when $k\notin D$. To generate $X_k$ :

For $k\le \left[\frac{5p}{20}\right]$, then $x_{i,k}=j$, if $z_{i,k}\in \left(z_{\left(j-1\right)/4},z_{j/4}\right],j=1,2,3,4$ ;

For $\left[\frac{5p}{20}\right]<k\le \left[\frac{10p}{20}\right]$, then $x_{i,k}=j$, if $z_{ik}\in \left(z_{\left(j-1\right)/10}\mathrm{,}{z}_{j/10}\right]\mathrm{,}j=\mathrm{1,}\cdots \mathrm{,10}$ ;

For $\left[\frac{10p}{20}\right]<k\le p$, then $x_{i,k}=z_{i,k}$.

Thus, amongst all the p covariates, the covariates of four categories and ten categories account for one-fifth, respectively, the other covariates are continuous. Similarly, there respectively are 5 in four categories and ten categories in the active covariates, and the rest of active covariates are continuous accounting for half. For continuous covariates, we applied different slices $J_k=4,8,10$. The corresponding approaches are defined as PG-SIS-4, IG-SIS-4, APC-SIS = 4, PG-SIS-8, IG-SIS-8, APC-SIS-8, PG-SIS-10, IG-SIS-10 and APC-SIS-10. Table 4 and Table 5 show the simulation results with over 100 simulations for balanced and unbalanced case, respectively. We can see the following: When the sample size n increases, PG-SIS is close to $d_0=20$ in MMS, and both increase to 1 in coverage probability. And coverage probability of PG-SIS is close to IG-SIS in five indexes. Therefore, it is proved that the PG-SIS has the characteristics of feature screening. MMS in unbalanced response is better than in balanced response in performance via comparing the response of different structures. Furthermore, PG-SIS and IG-SIS are robust in performance because the fluctuation range in MMS is small for two types of responses. When different slices are

Table 3. Parameter specification of Model 3.

Table 4. Simulation results for example 3: balanced Y.

Table 5. Simulation results for example 3: unbalanced Y.

Table 6. Analysis results for real data example.

applied in continuous covariates, PG-SIS and IG-SIS are better in five indexes of coverage probability and MMS in performance via comparing the response of different structures. Therefore, three methods are independent of the number of slices in performance.

4.2. Real Data

In this subsection, we analyse a real data set from the feature selection database of Arizona State University (http://featureselection.asu.edu/). The GLIOMA biological data includes 50 samples and 4434 features, which is unbalanced due to the response variable. Every class is 14, 7, 14, 15, and the covariates are not only continuous, but also multiclass. We randomly divided the data into two parts where 90 percent of the data represent training data and 10 percent of the data represents test data. The sample size of training data and test data respectively are $n=45$ and $n=5$. The dimension of both training data and test data are $p=4434$.

We apply a ten-fold cross-validation to eliminate different training data that cause the model accuracy problems. To PG-SIS, IG-SIS and APC-SIS, we use three classification approaches, which are Support Vector Machine [25], Random Forest (RF) and Decision Tree (DT) [26] to them via the chose active covariates.

In training data, we use the G-mean and F-measure [27] evaluation, the same is true for test data. PG-SIS in performance for unbalanced data is reported in Table 6. In all classification methods, PG-SIS is the best in performance, where G-mean of PG-SIS is more closed to 1 than the other two methods. In a word, the proposed PG-SIS performs better.

5. Conclusions

In the data, there are continuous and categorical covariates, and the response is categorical, which is very common in practice, but the applicable screening methods are very limited. We propose a PG-SIS procedure based on Gini impurity to effectively screen covariates. PG-SIS has a sure screening property and ranking consistency property theoretically and is model free. When the numbers of categories of all the covariates are the same and different, PG-SIS is quite similar to IG-SIS in performance, which can be shown in simulation.

The features screening reports some difficulties based on missing data. In the future, based on the classification model, we intend to propose a new feature screening to either the missing variable or the response variable can be selected.

Acknowledgements

The work was supported by National Natural Science Foundation of China [grant number 71963008].

Appendix

Proof of Proposition 2.1. To prove Proposition 2.1, we need to define $f\left(x\right)=x^2$, proved to be close Ni and Fang [19]. By Jensen’s inequality [24],

$\begin{array}{c}{\displaystyle \underset{j=1}{\overset{J_k}{\sum }}}{w}_{k,j}{\displaystyle \underset{r=1}{\overset{R}{\sum }}}{p}_{k,jr}^2={\displaystyle \underset{r=1}{\overset{R}{\sum }}}\left[{\displaystyle \underset{j=1}{\overset{J_k}{\sum }}}{w}_{k,j}f\left(p_{k,jr}\right)\right]\\ \ge {\displaystyle \underset{r=1}{\overset{R}{\sum }}}f\left({\displaystyle \underset{j=1}{\overset{J_k}{\sum }}}{w}_{k,j}{p}_{k,jr}\right)\\ ={\displaystyle \underset{r=1}{\overset{R}{\sum }}}f\left({\displaystyle \underset{j=1}{\overset{J_k}{\sum }}}P\left(X_k=j\right)P\left(Y=r|X_k=j\right)\right)\\ ={\displaystyle \underset{r=1}{\overset{R}{\sum }}}{p}_r^2\end{array}$

then

$\begin{array}{c}PG=1-{\displaystyle \underset{r=1}{\overset{R}{\sum }}}{p}_r^2-{\displaystyle \underset{j=1}{\overset{J_k}{\sum }}}{w}_{k,j}\left(1-{\displaystyle \underset{r=1}{\overset{R}{\sum }}}{p}_{k,jr}^2\right)\\ =1-{\displaystyle \underset{r=1}{\overset{R}{\sum }}}{p}_r^2-{\displaystyle \underset{j=1}{\overset{J_k}{\sum }}}{w}_{k,j}+{\displaystyle \underset{j=1}{\overset{J_k}{\sum }}}{w}_{k,j}{\displaystyle \underset{r=1}{\overset{R}{\sum }}}{p}_{k,jr}^2\\ ={\displaystyle \underset{j=1}{\overset{J_k}{\sum }}}{w}_{k,j}{\displaystyle \underset{r=1}{\overset{R}{\sum }}}{p}_{k,jr}^2-{\displaystyle \underset{r=1}{\overset{R}{\sum }}}{p}_r^2\ge 0\end{array}$

The above equation holds if and only if $p_{k,jr}=p_{k,j^{\prime }r}$, for any $1\le r\le R$ and $1\le j\le j^{\prime }\le J$. That is, $X_k$ and Y are independent.

Proof of Proposition 2.2. From the same proof as Proposition 2.1, we can get $P{G}_J\left(Y\mathrm{<mo>\; |\; </mo>}{X}_k\right)\ge 0$ holds if and only if $p_{k,j^{\prime }r}=p_{k,j^{\prime }r}$ for any $1\le r\le R$ and $1\le j\le j^{\prime }\le J$, that is $P\left(Y=r|X\in \left(q_{\left(j-1\right)},q_{\left(j\right)}\right]\right)=P\left(Y=r|X\in \left(q_{\left(j^{\prime }-1\right)},q_{\left(j^{\prime }\right)}\right]\right)$. So when $X_k$ and Y are independent, $P{G}_J\left(Y|X_k\right)=0$. On the other hand, when $P{G}_J\left(Y|X_k\right)=0$ for any J, we need to show that $P\left(X\le x|Y=r\right)$ does not depend on r for any x in the domain of X. Proposition 2.2 has been proven by [19], so the proof is omitted here.

Lemma 1 (Bernstein inequality). If $Z_1\mathrm{,}\cdots \mathrm{,}{Z}_n$ is an independent random variable with a mean value of 0 and bounded supporter is $\left[-M\mathrm{,}M\right]$, then the inequality:

$P\left(\left|{\displaystyle {\sum }_{i=1}^n}{Z}_i\right|>t\right)\le 2\exp \left\{-\frac{t^2}{2\left(v+\frac{Mt}{3}\right)}\right\}$

where $v\ge Var\left({\displaystyle {\sum }_{i=1}^n}{Z}_i\right)$

Lemma 2. For discrete covariates $X_k$ and discrete response Y, we have the following three inequalities:

1) $P\left(\left|{\hat{p}}_r-p_r\right|>t\right)\le 2\exp \left\{-\frac{6n{t}^2}{3+4t}\right\}$

2) $P\left(\left|{\hat{w}}_{k,j}-w_{k,j}\right|>t\right)\le 2\exp \left\{-\frac{6n{t}^2}{3+4t}\right\}$

3) $P\left(\left|{\hat{p}}_{k,jr}-p_{k,jr}\right|>t\right)\le 2\exp \left\{-\frac{6n{t}^2}{3+4t}\right\}$

Proof of Lemma 2. Three inequalities are similar in the proofs, where inequality (1) and inequality (2) have been given at [27]. The following is the proof of inequality (3).

$\begin{array}{l}{\hat{p}}_{k,jr}=\frac{{\displaystyle {\sum }_{i=1}^{n}I}\left\{y_i=r,X_{i,k}=j\right\}}{{\displaystyle {\sum }_{i=1}^{n}I}\left\{X_{i,k}=j\right\}}\hfill \end{array}$

The expectation of ${\hat{p}}_{k\mathrm{,}jr}$ is

$E\left({\hat{p}}_{k,jr}\right)=E\left(\frac{{\displaystyle {\sum }_{i=1}^{n}I}\left\{y_i=r,X_{i,k}=j\right\}}{{\displaystyle {\sum }_{i=1}^{n}I}\left\{X_{i,k}=j\right\}}\right)=E\left(\frac{{\displaystyle {\sum }_{i=1}^n{y}_i=r,X_{i,k}=j}}{{\displaystyle {\sum }_{i=1}^n{X}_{i,k}=j}}\right)=p_{k,jr}$

Let $Z_i=I\left\{y_i=r|X_{i,k}=j\right\}-p_{k,jr}$, $Var\left({\displaystyle {\sum }_{i=1}^n}{Z}_i\right)=nVar\left(Z_i\right)=n{p}_{k,jr}\left(1-p_{k,jr}\right)\le \frac{n}{4}$ is known, then

$\begin{array}{c}P\left(\left|{\hat{p}}_{k,jr}-p_{k,jr}\right|>t\right)=P\left(\left|n^{-1}{\displaystyle \underset{i=1}{\overset{n}{\sum }}}{Z}_i\right|>t\right)=P\left(\left|{\displaystyle \underset{i=1}{\overset{n}{\sum }}}{Z}_i\right|>nt\right)\\ \le 2\exp \left\{-\frac{n^2{t}^2}{2\left(\frac{n}{4}+\frac{nt}{3}\right)}\right\}\le 2\exp \left\{-\frac{6n{t}^2}{3+4t}\right\}\end{array}$

According to Bernstein inequality, the formula is held.

Lemma 3. With regard to discrete covariates $X_k$ and discrete response Y, for any $0<\epsilon <1$, under condition (C1), we have

$P\left(\left|{\hat{e}}_k-e_k\right|>2\epsilon \right)\le O\left(R{J}^3\right)\exp \left\{-c_5\frac{n{\epsilon }^2}{R^2{J}^6}\right\}$,

where $c_5$ represents a positive constant.

Proof of Lemma 3. By $e_k$ and ${\hat{e}}_k$ in Section 2.2, we have

$\begin{array}{l}\log J_K\left({\hat{e}}_k-e_k\right)\\ =\left[\left(1-{\displaystyle \underset{r=1}{\overset{R}{\sum }}}{\hat{p}}_r^2\right)-{\displaystyle \underset{j=1}{\overset{J_k}{\sum }}}{\hat{w}}_{k,j}\left(1-{\displaystyle \underset{r=1}{\overset{R}{\sum }}}{\hat{p}}_{k,jr}^2\right)\right]-\left[1-{\displaystyle \underset{r=1}{\overset{R}{\sum }}}{p}_r^2-{\displaystyle \underset{j=1}{\overset{J_k}{\sum }}}{w}_{k,j}\left(1-{\displaystyle \underset{r=1}{\overset{R}{\sum }}}{p}_{k,jr}^2\right)\right]\\ =\left({\displaystyle \underset{r=1}{\overset{R}{\sum }}}{p}_r^2-{\displaystyle \underset{r=1}{\overset{R}{\sum }}}{\hat{p}}_r^2\right)+\left({\displaystyle \underset{j=1}{\overset{J_k}{\sum }}}{w}_{k,j}-{\displaystyle \underset{j=1}{\overset{J_k}{\sum }}}{\hat{w}}_{k,j}\right)+\left({\displaystyle \underset{j=1}{\overset{J_k}{\sum }}}{\hat{w}}_{k,j}{\displaystyle \underset{r=1}{\overset{R}{\sum }}}{\hat{p}}_{k,jr}^2-{\displaystyle \underset{j=1}{\overset{J_k}{\sum }}}{w}_{k,j}{\displaystyle \underset{r=1}{\overset{R}{\sum }}}{p}_{k,jr}^2\right)\\ ={\displaystyle \underset{r=1}{\overset{R}{\sum }}}\left(p_r^2-{\hat{p}}_r^2\right)+{\displaystyle \underset{j=1}{\overset{J_k}{\sum }}}\left(w_{k,j}-{\hat{w}}_{k,j}\right)+{\displaystyle \underset{j=1}{\overset{J_k}{\sum }}}{\displaystyle \underset{r=1}{\overset{R}{\sum }}}\left({\hat{w}}_{k,j}{\hat{p}}_{k,jr}^2-w_{k,j}{p}_{k,jr}^2\right)\end{array}$

$\begin{array}{l}={\displaystyle \underset{r=1}{\overset{R}{\sum }}}\left(p_r-{\hat{p}}_r\right)\left(p_r+{\hat{p}}_r\right)+{\displaystyle \underset{j=1}{\overset{J_k}{\sum }}}\left(w_{k,j}-{\hat{w}}_{k,j}\right)\\ +{\displaystyle \underset{j=1}{\overset{J_k}{\sum }}}{\displaystyle \underset{r=1}{\overset{R}{\sum }}}\left[\left({\hat{w}}_{k,j}{\hat{p}}_{k,jr}+w_{k,j}{p}_{k,jr}\right)\left({\hat{p}}_{k,jr}-p_{k,jr}\right)+{\hat{p}}_{k,jr}{p}_{k,jr}\left(w_{k,j}-{\hat{w}}_{k,j}\right)\right]\\ =I_1+I_2+I_3\end{array}$

Since $\log J\ge \log 2\ge 0.5$, we have

$P\left(\left|{\hat{e}}_k-e_k\right|>\epsilon \right)\le P\left(\left|I_1\right|>\frac{\epsilon }{3}\right)+P\left(\left|I_2\right|>\frac{\epsilon }{3}\right)+P\left(\left|I_3\right|>\frac{\epsilon }{3}\right)$

For $I_1$, we have

$\begin{array}{c}P\left(\left|I_1\right|>\frac{\epsilon }{3}\right)\le {\displaystyle \underset{r=1}{\overset{R}{\sum }}}P\left(\left|\left(p_r-{\hat{p}}_r\right)\left(p_r+{\hat{p}}_r\right)\right|>\frac{\epsilon }{3}\right)\\ \le {\displaystyle \underset{r=1}{\overset{R}{\sum }}}P\left(\left|p_r-{\hat{p}}_r\right|>\frac{c_1\epsilon }{3R{J}^3}\right)\\ \le R{J}^{3}2\exp \left\{-\frac{6n{\left(\frac{c_1\epsilon }{3R{J}^3}\right)}^2}{3+4\left(\frac{c_1\epsilon }{3R{J}^3}\right)}\right\}\end{array}$

For $I_2$, we have

$P\left(\left|I_2\right|>\frac{\epsilon }{3}\right)\le {\displaystyle \underset{j}{\overset{J_k}{\sum }}}P\left(\left|{\hat{w}}_{k,j}-w_{k,j}\right|>\frac{c_1\epsilon }{3J^3}\right)\le J^{3}2\exp \left\{-\frac{6n{\left(\frac{c_1\epsilon }{3J^3}\right)}^2}{3+4\left(\frac{c_1\epsilon }{3J^3}\right)}\right\}$

For $I_3$, we have

$\begin{array}{c}{I}_3={\displaystyle \underset{j=1}{\overset{J_k}{\sum }}}{\displaystyle \underset{r=1}{\overset{R}{\sum }}}\left[\left({\hat{w}}_{k,j}{\hat{p}}_{k,jr}+w_{k,j}{p}_{k,jr}\right)\left({\hat{p}}_{k,jr}-p_{k,jr}\right)+{\hat{p}}_{k,jr}{p}_{k,jr}\left(w_{k,j}-{\hat{w}}_{k,j}\right)\right]\\ ={\displaystyle \underset{j}{\overset{J_k}{\sum }}}{\displaystyle \underset{r=1}{\overset{R}{\sum }}}\left[\left({\hat{w}}_{k,j}{\hat{p}}_{k,jr}+w_{k,j}{p}_{k,jr}\right)\left({\hat{p}}_{k,jr}-p_{k,jr}\right)\right]+{\displaystyle \underset{j}{\overset{J_k}{\sum }}}{\displaystyle \underset{r=1}{\overset{R}{\sum }}}{\hat{p}}_{k,jr}{p}_{k,jr}\left({\hat{w}}_{k,j}-w_{k,j}\right)\\ :=I_{31}+I_{32}\end{array}$

For $I_{31}$ and $I_{32}$, we have

$P\left(\left|I_3\right|>\frac{\epsilon }{3}\right)\le P\left(\left|I_{31}\right|>\frac{\epsilon }{6}\right)+P\left(\left|I_{32}\right|>\frac{\epsilon }{6}\right)$

$\begin{array}{c}P\left(\left|I_{31}\right|>\frac{\epsilon }{6}\right)\le {\displaystyle \underset{j}{\overset{J_k}{\sum }}}{\displaystyle \underset{r=1}{\overset{R}{\sum }}}P\left(\left|\left({\hat{w}}_{k,j}{\hat{p}}_{k,jr}+w_{k,j}{p}_{k,jr}\right)\left({\hat{p}}_{k,jr}-p_{k,jr}\right)\right|>\frac{\epsilon }{6}\right)\\ \le {\displaystyle \underset{j}{\overset{J_k}{\sum }}}{\displaystyle \underset{r=1}{\overset{R}{\sum }}}p\left(\left|{\hat{p}}_{k,jr}-p_{k,jr}\right|>\frac{c_1\epsilon }{6R{J}^3}\right)\\ \le R{J}^{3}2\exp \left\{-\frac{6n{\left(\frac{c_1\epsilon }{6R{J}^3}\right)}^2}{3+4\left(\frac{c_1\epsilon }{6R{J}^3}\right)}\right\}\end{array}$

$\begin{array}{c}P\left(\left|I_{32}\right|>\frac{\epsilon }{6}\right)\le {\displaystyle \underset{j}{\overset{J_k}{\sum }}}{\displaystyle \underset{r=1}{\overset{R}{\sum }}}P\left(\left|{\hat{p}}_{k,jr}{p}_{k,jr}\left({\hat{w}}_{k,j}-w_{k,j}\right)\right|>\frac{\epsilon }{6}\right)\\ \le {\displaystyle \underset{j}{\overset{J_k}{\sum }}}{\displaystyle \underset{r=1}{\overset{R}{\sum }}}P\left(\left|{\hat{w}}_{k,j}-w_{k,j}\right|\frac{c_1\epsilon }{6R{J}^3}\right)\\ \le R{J}^{3}2\exp \left\{-\frac{6n{\left(\frac{c_1\epsilon }{6R{J}^3}\right)}^2}{3+4\left(\frac{c_1\epsilon }{6R{J}^3}\right)}\right\}\end{array}$

In a word, we have the inequality,

$P\left(\left|{\hat{e}}_k-e_k\right|>2\epsilon \right)\le O\left(R{J}^3\right)\exp \left\{-c_5\frac{n{\epsilon }^2}{R^2{J}^6}\right\}$

where $c_5$ represents a positive constant.

Proof of Theorem 3.1. By Conditions (C1) to (C3) and Lemma 3, we can get

$\begin{array}{c}P\left(D\subseteq \hat{D}\right)\ge P\left(\left|{\hat{e}}_k-e_k\right|\le c{n}^{-\tau },\forall k\in D\right)\\ \ge P\left(\underset{1\le k\le P}{\max }\left|{\hat{e}}_k-e_k\right|\le c{n}^{-\tau }\right)\\ \ge 1-{\displaystyle \underset{k=1}{\overset{P}{\sum }}}P\left(\underset{1\le k\le P}{\max }\left|{\hat{e}}_k-e_k\right|>c{n}^{-\tau }\right)\\ \ge 1-O\left(R{J}^3\right)p\exp \left\{-c_5\frac{c^2{n}^{1-2\tau }}{R^2{J}^6}\right\}\\ \ge 1-O\left(p\exp \left\{-b{n}^{1-2\tau -2\epsilon -2\kappa }+\left(\epsilon +\kappa \right)\log n\right\}\right)\end{array}$

Where b is a positive constant.

Lemma 4 (Lemma A.2 [19]). For any continuous covariate $X_k$ satisfying conditions (C4) and (C5), let $F_k\left(y\mathrm{,}x\right)$ be the cumulative distribution function of $\left(Y\mathrm{,}{X}_k\right)$ and ${\hat{F}}_k\left(y\mathrm{,}x\right)$ be the empirical cumulative distribution function, we have $P\left(\left|{\hat{F}}_k\left(r,{\hat{q}}_{k,\left(j\right)}\right)-F_k\left(r,q_{k,\left(j\right)}\right)\right|>\epsilon \right)\le c_6\exp \left\{-c_7{n}^{1-2\rho }{\epsilon }^2\right\}$ for any $\epsilon >0$, $1\le r\le R$ and $1\le j\le J_k$, where $c_6$ and $c_7$ are two positive constants.

Lemma 5 (Lemma A.3 [19]). Under (C1), (C4) and (C5), for any $0<\epsilon <1$, so for continuous $X_k$, we have $P\left(\left|{\hat{e}}_k-e_k\right|>2\epsilon \right)\le O\left(RJ\right)\exp \left\{-c_9\frac{n^{1-2\rho }{\epsilon }^2}{R^4{J}^4}\right\}$, there exists a positive constant $c_9$ .

Proof of Theorem 3.2. According to Lemma 4 and Lemma 5, the proof of Theorem 3.2 is the same as Theorem 3.1 and hence is omitted.

Proof of Theorem 3.3. According to Lemma 3 and 5 and under Conditions (C1), (C4), (C5) and (C7). The proof of Theorem 3.2 is proved by [19] and hence is omitted.

Conflicts of Interest

The authors declare no conflicts of interest.

References

[1]	Fan, J.Q. and Lv, J.C. (2008) Sure Independence Screening for Ultrahigh Dimensional Feature Space. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 70, 849-911. https://doi.org/10.1111/j.1467-9868.2008.00674.x
[2]	Fan, J.Q., Samworth, R. and Wu, Y.C. (2009) Ultrahigh Dimensional Feature Selection: Beyond the Linear Model. Journal of Machine Learning Research, 10, 2013-2038. http://arxiv.org/abs/0812.3201
[3]	Wang, H.S. (2009) Forward Regression for Ultra-High Dimensional Variable Screening. Journal of the American Statistical Association, 104, 1512-1524. https://doi.org/10.1198/jasa.2008.tm08516
[4]	Fan, J.Q. and Song, R. (2010) Sure Independence Screening in Generalized Linear Models with NP-Dimensionality. Annals of Statistics, 38, 3567-3604. https://doi.org/10.1214/10-AOS798
[5]	Fan, J.Q., Feng, Y. and Song, R. (2011) Nonparametric Independence Screening in Sparse Ultra-High-Dimensional Additive Models. Journal of the American Statistical Association, 106, 544-557. https://doi.org/10.1198/jasa.2011.tm09779
[6]	Zhu, L.P., Li, L.X., Li, R.Z. and Zhu, L.X. (2011) Model-Free Feature Screening for Ultrahigh-Dimensional Data. Journal of the American Statistical Association, 106, 1464-1475. https://doi.org/10.1198/jasa.2011.tm10563
[7]	Li, G.R., Peng, H., Zhang, J. and Zhu, L.X. (2012) Robust Rank Correlation Based Screening. Annals of Statistics, 40, 1846-1877. https://doi.org/10.1214/12-AOS1024
[8]	Li, R.Z., Zhong, W. and Zhu, L.P. (2012) Feature Screening via Distance Correlation Learning. Journal of the American Statistical Association, 107, 1129-1139. https://doi.org/10.1080/01621459.2012.695654
[9]	He, X.M., Wang, L. and Hong, H.G. (2013) Quantile-Adaptive Model-Free Variable Screening for High-Dimensional Heterogeneous Data. Annals of Statistics, 41, 342-369. https://doi.org/10.1214/13-AOS1087
[10]	Fan, J.Q., Ma, Y.B. and Dai, W. (2014) Nonparametric Independence Screening in Sparse Ultra-High-Dimensional Varying Coefficient Models. Journal of the American Statistical Association, 109, 1270-1284. https://doi.org/10.1080/01621459.2013.879828
[11]	Liu, J.Y., Li, R.Z. and Wu, R.L. (2014) Feature Selection for Varying Coefficient Models with Ultrahigh-Dimensional Covariates. Statistics & Probability Letters, 109, 266-274. https://doi.org/10.1080/01621459.2013.850086
[12]	Nandy, D., Chiaromonte, F. and Li, R.Z. (2021) Covariate Information Number for Feature Screening in Ultrahigh-Dimensional Supervised Problems. Journal of the American Statistical Association, 117, 1516-1529. https://doi.org/10.1080/01621459.2020.1864380
[13]	Pouyap, M., Bit joka, L., Mfoumou, E. and Toko, D. (2021) Improved Bearing Fault Diagnosis by Feature Extraction Based on GLCM, Fusion of Selection Methods, and Multiclass-Naive Bayes Classification. Journal of Signal and Information Processing, 12, 71-85. https://doi.org/10.4236/jsip.2021.124004
[14]	Fan, J.Q. and Fan, Y.Y. (2008) High-Dimensional Classification Using Features Annealed Independence Rules. Annals of Statistics, 36, 2605-2637. https://doi.org/10.1214/07-AOS504
[15]	Mai, Q. and Zou, H. (2013) The Kolmogorov Filter for Variable Screening in High-Dimensional Binary Classification. Biometrika, 100, 229-234. https://doi.org/10.1093/biomet/ass062
[16]	Cui, H.J., Li, R.Z. and Zhong, W. (2015) Model-Free Feature Screening for Ultrahigh Dimensional Discriminant Analysis. Journal of the American Statistical Association, 110, 630-641. https://doi.org/10.1080/01621459.2014.920256
[17]	Lai, P., Song, F.L., Chen, K.W. and Liu, Z. (2017) Model Free Feature Screening with Dependent Variable in Ultrahigh Dimensional Binary Classification. Statistics & Probability Letters, 125, 141-148. https://doi.org/10.1016/j.spl.2017.02.011
[18]	Huang, D.Y., Li, R.Z. and Wang, H.S. (2014) Feature Screening for Ultrahigh Dimensional Categorical Data with Applications. Journal of Business & Economic Statistics, 32, 237-244. https://doi.org/10.1080/07350015.2013.863158
[19]	Ni, L. and Fang, F. (2016) Entropy-Based Model-Free Feature Screening for Ultrahigh-Dimensional Multiclass Classification. Journal of Nonparametric Statistics, 28, 515-530. https://doi.org/10.1080/10485252.2016.1167206
[20]	Ni, L., Fang, F. and Wan, F.J. (2017) Adjusted Pearson Chi-Square Feature Screening for Multi-Classification with Ultrahigh Dimensional Data. Metrika, 80, 805-828. https://doi.org/10.1007/s00184-017-0629-9
[21]	Sheng, Y. and Wang, Q.H. (2020) Model-Free Feature Screening for Ultrahigh Dimensional Classification. Journal of Multivariate Analysis, 178, 1-12. https://doi.org/10.1016/j.jmva.2020.104618
[22]	Anzarmou, Y., Mkhadri, A. and Oualkacha, K. (2022) The Kendall Interaction Filter for Variable Interaction Screening in High Dimensional Classification Problems. Journal of Applied Statistics, 1-19. https://doi.org/10.1080/02664763.2022.2031125
[23]	Breiman, L., Friedman, J.H., Stone, C.J. and Olshen, R.A. (1984) Classification and Regression Trees. Wadsworth International Group, Belmont.
[24]	Marco, T. (2012) Lectures on Probability Theory and Mathematical Statistics. CreateSpace Independent Publishing Platform, Scotts Valley.
[25]	Suykens, J.A.K. and Vandewalle, J. (1999) Least Squares Support Vector Machine Classifiers. Neural Processing Letters, 9, 293-300. https://doi.org/10.1023/A:1018628609742
[26]	Lantz, B. (2015) Machine Learning with R. Mathematical & Statistical Software. Packt Publishing, Birmingham.
[27]	He, H.J. and Deng, G.M. (2022) Grouped Feature Screening for Ultra-High Dimensional Data for the Classification Model. Journal of Statistical Computation and Simulation, 1-24.