A Geometric View on Inner Transformation between the Variables of a Linear Regression Model

Zhaoyang Li¹, Bostjan Antoncic^2*

¹Shanghai University of International Business and Economics, Shanghai, China.
²School of Economics and Business, University of Ljubljana, Ljubljana, Slovenia.

DOI: 10.4236/am.2021.1210061                

Abstract

In the teaching and researching of linear regression analysis, it is interesting and enlightening to explore how the dependent variable vector can be inner-transformed into regression coefficient estimator vector from a visible geometrical view. As an example, the roadmap of such inner transformation is presented based on a simple multiple linear regression model in this work. By applying the matrix algorithms like singular value decomposition (SVD) and Moore-Penrose generalized matrix inverse, the dependent variable vector lands into the right space of the independent variable matrix and is metamorphosed into regression coefficient estimator vector through the three-step of inner transformation. This work explores the geometrical relationship between the dependent variable vector and regression coefficient estimator vector as well as presents a new approach for vector rotating.

Keywords

Matrix Singular Value Decomposition, Moore-Penrose Generalized Inverse, Matrix Inner Transformation, Regression Analysis

1. Introduction

A matrix can be factorized into the product of several matrices with special properties. Particularly, by singular value decomposition (SVD) which is widely used in regression analysis [1] [2] [3], a matrix can be factorized into the product of three matrices with orthogonal or diagonal properties respectively. SVD can be formularized like $X=U\theta V^{\text{T}}$ [4] [5] [6] [7].

Moore-Penrose generalized inverse is a special case of generalized matrix inverse [8] [9] [10] [11] [12] and can be applied in regression analysis [13] [14] and least square analysis [15]. For example, in the study of Tian and Zhang [13], the linear unbiased estimator of partial coefficients is derived through Moore-Penrose generalized inverse algorithm. Herein, multiplied by Moore-Penrose generalized inverse of the independent variable matrix, the dependent variable vector is transformed into the new coordinate systems, or left space and then right space of the independent variable matrix [16]. In fact, this process can also be regarded as a vector rotating algorithm. In addition to Moore-Penrose’s generalized inverse algorithm, singular value decomposition (SVD) facilitates such transformation by dividing the transformation process into three-step to present a visible geometrical view.

2. y Can Be Transformed into $\hat{\beta }$

In order to avoid the irrelevant calculation minutia when adopting an independent matrix with a large rank and only to highlight inner transformation, herein,

a simple multiple linear regression model $y=X\hat{\beta }+\hat{\epsilon }$ is adopted with the independent matrix $X=\left[\begin{array}{cc}1& 1\\ 1& 2\\ 1& 3\end{array}\right]$ and the dependent vector $y=\left[\begin{array}{c}1\\ 0\\ 2\end{array}\right]$, $\hat{\beta }$ symbolizes the regression coefficient estimator vector.

In this multiple linear regression model, the matrix X’s singular value decomposition (SVD) can be demonstrated as below:

$X=U\theta V^{\text{T}}=\left[\begin{array}{ccc}0.324& 0.854& 0.420\\ 0.549& 0.186& -0.820\\ 0.774& -0.490& 0.410\end{array}\right]\left[\begin{array}{cc}4.07& 0\\ 0& 0.6\\ 0& 0\end{array}\right]{\left[\begin{array}{cc}0.403& 0.915\\ 0.915& -0.403\end{array}\right]}^{\text{T}}$ (E.1)

U and V are respectively the left and right singular vector matrix of X, and $\theta$ is eigen value matrix of X [6] [7] [17].

Because X is a matrix with full rank column, $\hat{\beta }$ can be demonstrated as $\hat{\beta }=X^{+}y$ [13] [16], $X^{+}$ signifies the Moore-Penrose generalized inverse of matrix X. And $\hat{\beta }$ can also be demonstrated as below when X is substituted by its SVD form presented in equation (E.1),

$\hat{\beta }=X^{+}y={\left(U\theta V^{\text{T}}\right)}^{+}y={\left(V^{\text{T}}\right)}^{-1}{\theta }^{+}{U}^{-1}y$ (E.2)

The Equation (E.2) can be regarded as a process that the vector y is transformed 3 times from right to left multiplied by ${\left(V^{\text{T}}\right)}^{-1}{\theta }^{+}{U}^{-1}$.

Transformation 1

$y_1=U^{-1}y={\left[\begin{array}{ccc}0.324& 0.854& 0.420\\ 0.549& 0.186& -0.820\\ 0.774& -0.490& 0.410\end{array}\right]}^{-1}\left[\begin{array}{c}1\\ 0\\ 2\end{array}\right]=\left[\begin{array}{c}1.86\\ -0.12\\ 1.24\end{array}\right]$ (E.3)

U’s columns are demonstrated as $u_1=\left[\begin{array}{c}0.324\\ 0.549\\ 0.774\end{array}\right]$, $u_2=\left[\begin{array}{c}0.854\\ 0.186\\ -0.490\end{array}\right]$ and $u_3=\left[\begin{array}{c}0.420\\ -0.820\\ 0.410\end{array}\right]$.

Let the coordinate axes of a 3-dimension original coordinate system be symbolized by $a_1=\left[\begin{array}{c}1\\ 0\\ 0\end{array}\right]$, $a_2=\left[\begin{array}{c}0\\ 1\\ 0\end{array}\right]$ and $a_3=\left[\begin{array}{c}0\\ 0\\ 1\end{array}\right]$ . The quintessence of the equation (E.3) is that multiplied by the matrix $U^{-1}$, vector $y=\left[\begin{array}{c}1\\ 0\\ 2\end{array}\right]$ in the coordinate system constructed by $a_1,a_2$ and $a_3$, can be transformed into $y_1=\left[\begin{array}{c}1.86\\ -0.12\\ 1.24\end{array}\right]$

which is located in a coordinate system constructed by $u_1,u_2$ and $u_3$, or left space of X. In fact, y and $y_1$ are at the same spatial location. However, y is in the coordinate system constructed by $a_1,a_2$ and $a_3$, meanwhile $y_1$ is presented by the coordinate system constructed by $u_1,u_2$ and $u_3$, in another word, $y_1$ is in the left space of X. These locations can be demonstrated as below in Figure 1.

Transformation 2

$y_2={\theta }^{+}{U}^{-1}y={\theta }^{+}{y}_1={\left[\begin{array}{cc}4.07& 0\\ 0& 0.6\\ 0& 0\end{array}\right]}^{+}\left[\begin{array}{c}1.86\\ -0.12\\ 1.24\end{array}\right]=\left[\begin{array}{c}0.46\\ -0.20\\ 0\end{array}\right]$ (E.4)

During this transformation, multiplied by $\theta$ ’s Moore–Penrose inverse ${\theta }^{+}$, the vector $y_1$ in $u_1,u_2$ and $u_3$ coordinate system is transformed into the vector $y_2$ which is in the same coordinate system as $y_1$. Such a transformation can be called “Vector Stretching”. During this transformation, the coordinate value of $y_1$ in $u_1$ axis diminishes, meanwhile, the coordinate value of $y_1$ in $u_2$ axis gets enlarged. The coordinate value of $y_1$ in $u_3$ axis vanishes. The “Vector Stretching” of $y_1$ into $y_2$ can be demonstrated as below in Figure 2.

The vanishing coordinate value of $y_1$ in $u_3$ axis can signify the degree of freedom of X from a geometric view.

Transformation 3

$\hat{\beta }={\left(V^{\text{T}}\right)}^{-1}{\theta }^{+}{U}^{-1}y={\left(V^{\text{T}}\right)}^{-1}{y}_2={\left({\left[\begin{array}{cc}0.403& 0.915\\ 0.915& -0.403\end{array}\right]}^{\text{T}}\right)}^{-1}\left[\begin{array}{c}0.46\\ -0.20\\ 0\end{array}\right]=\left[\begin{array}{c}0\\ 0.5\end{array}\right]$ (E.5)

X’s right singular vectors can be denoted as $v_1^{\text{T}}=\left[\begin{array}{c}0.403\\ 0.915\end{array}\right]$ and $v_2^{\text{T}}=\left[\begin{array}{c}0.915\\ -0.403\end{array}\right]$. $v_1^{\text{T}}$ and $v_2^{\text{T}}$ vectors are located in a coordinate system constructed by $u_1$ and $u_2$ vectors [16].

Figure 1. The dependent variable vector y transformed into y₁ by landing on to the left space of matrix X.

Figure 2. Vector y₁ transformed intoy₂ through “Vector Stretching”.

The quintessence of Equation (E.5) is that the vector $y_2$ in the coordinate system is constructed by $u_1$ and $u_2$ can be embodied by the vector $\hat{\beta }$ in the coordinate system $v_1^{\text{T}}$ and $v_2^{\text{T}}$. This transformation is demonstrated as below in Figure 3.

Though the vector $\hat{\beta }$ has the same spatial position as $y_2$ in Figure 3, these two vectors are situated in two different coordinate systems constructed respectively by $v_1^{\text{T}},v_2^{\text{T}}$ (the right space of X) and $u_1,u_2$ (the left space of X). In another word, the distinction between the vector $\hat{\beta }$ and $y_2$ is that $\hat{\beta }$ is demonstrated by the coordinate system of $v_1^{\text{T}}$ and $v_2^{\text{T}}$, meanwhile, $y_2$ is demonstrated by $u_1$ and $u_2$. As demonstrated by the aforementioned 3 steps of transformation, the dependent variable y vector is inner-transformed into parameter $\hat{\beta }$ vector.

Figure 3. Vector y₂ transformed into $\hat{\beta }$ by landing on to the right space of matrix X.

3. y Can Be Transformed into $\hat{y}$

The independent variable X’s left singular vector matrix U [6] [7] [17] ) can be blocked into two sub-matrics $U_1$ and $U_2$ as below.

$U=\left[\begin{array}{ccc}0.324& 0.854& 0.420\\ 0.549& 0.186& -0.820\\ 0.774& -0.490& 0.410\end{array}\right]=\left[\begin{array}{cc}{U}_1& U_2\end{array}\right]=\left[\begin{array}{ccc}0.324& 0.854& 0.420\\ 0.549& 0.186& -0.820\\ 0.774& -0.490& 0.410\end{array}\right]$

Let $\hat{y}$ symbolize the estimator of the dependent variable y. For that $U_1$ is the left space of X, $\hat{y}$ can be demonstrated as $\hat{y}=X\hat{\beta }=U_1{U}_1^{\text{T}}y$ [16], based on which the following result can be derived.

$\hat{y}=U_1{U}_1^{\text{T}}y=\left[\begin{array}{cc}0.324& 0.854\\ 0.549& 0.186\\ 0.774& -0.490\end{array}\right]{\left[\begin{array}{cc}0.324& 0.854\\ 0.549& 0.186\\ 0.774& -0.490\end{array}\right]}^{\text{T}}\left[\begin{array}{c}1\\ 0\\ 2\end{array}\right]=\left[\begin{array}{c}0.5\\ 1\\ 1.5\end{array}\right]$

Because $U_1$ is an orthonormal matrix, $U_1$ ’s Moore-Penrose inverse $U_1^{+}$ is equal with $U_1^{\text{T}}$ [12]. So, $\hat{y}$ can be demonstrated as below:

$\hat{y}=U_1{U}_1^{+}y=\left[\begin{array}{cc}0.324& 0.854\\ 0.549& 0.186\\ 0.774& -0.490\end{array}\right]{\left[\begin{array}{cc}0.324& 0.854\\ 0.549& 0.186\\ 0.774& -0.490\end{array}\right]}^{+}\left[\begin{array}{c}1\\ 0\\ 2\end{array}\right]=\left[\begin{array}{c}0.5\\ 1\\ 1.5\end{array}\right]$ (E.6)

Based on E.6, y can be transformed into $\hat{y}$ by two steps as below.

Transformation 4

$y^{\ast }=U_1^{+}y={\left[\begin{array}{cc}0.324& 0.854\\ 0.549& 0.186\\ 0.774& -0.490\end{array}\right]}^{+}\left[\begin{array}{c}1\\ 0\\ 2\end{array}\right]=\left[\begin{array}{c}1.86\\ -0.12\end{array}\right]$ (E.7)

In Transformation 4, y transforms into $y^{\ast }$ by being projected into $U_1$, the left space of X. Another word, $y^{\ast }$ is the projection of y in X.

Transformation 5

$\hat{y}=U_1{U}_1^{+}y=U_1{y}^{\ast }=\left[\begin{array}{cc}0.324& 0.854\\ 0.549& 0.186\\ 0.774& -0.490\end{array}\right]\left[\begin{array}{c}1.86\\ -0.12\end{array}\right]=\left[\begin{array}{c}0.5\\ 1\\ 1.5\end{array}\right]$ (E.8)

In Transformation 5, multiplied by $U_1$, $y^{\ast }$ transforms into $\hat{y}$ and is presented in the coordinate system constructed by $a_1,a_2$ and $a_3$. Their spatial locations are demonstrated in Figure 4.

In Figure 4, it can be found that the error estimator $\hat{\epsilon }$ is located in the same direction with $u_3$ which is the left null space of matrix X, and can be expressed as below:

$\hat{\epsilon }=\left[\begin{array}{c}0.5\\ -1\\ 0.5\end{array}\right]\cong 1.22\left[\begin{array}{c}0.42\\ -0.82\\ 0.41\end{array}\right]=1.22u_3$.

$\hat{y}$ is the projection of y into the left space of X, but presented in the original coordinate system constructed by $a_1,a_2$ and $a_3$. $\hat{\epsilon }$ is located in the left null space of matrix X and perpendicular with $\hat{y}$. This perpendicular result can be geometrically demonstrated in Figure 4 as well as in the following multiplication.

${\hat{y}}^{\text{T}}\hat{\epsilon }=\left[\begin{array}{ccc}0.5& 1& 1.5\end{array}\right]\left[\begin{array}{c}0.5\\ -1\\ 0.5\end{array}\right]=0$.

4. Finding and Conclusion

By applying SVD and Moore-Penrose generalized inverse of the independent variable X in a multiple linear regression model, the dependent variable y can be transformed into the regression coefficient estimator vector $\hat{\beta }$ and its own estimator $\hat{y}$. This process presents a new geometric perspective to study the relationship between X, y, $\hat{\beta }$ and $\hat{y}$, through the inner-transformation algorithm.

As demonstrated from Figures 1-3, y transforms into $\hat{\beta }$ by transforming from the original coordinate system of $a_1,a_2$ and $a_3$ into the right space of X constructed by $v_1^{\text{T}}$ and $v_2^{\text{T}}$. Through this process, y transforms into $y_1$ by transferring into the coordinate system of $u_1,u_2$ and $u_3$, that is presented by Transformation 1 (E.3) and demonstrated in Figure 1.

Multiplied by ${\theta }^{+}$, the Moore-Penrose inverse of $\theta$, $y_1$ stretches into $y_2$, that is presented by Rotation 2 (E.4) and demonstrated in Figure 2. In this transformation, the coordinate value of $y_1$ in $u_3$ axis vanished, which also signifies the degree of freedom of X from a geometric view.

Multiplied by ${\left(V^{\text{T}}\right)}^{-1}$, the inverse of the right singular value matrix of X, $y_2$ transforms into $\hat{\beta }$, that is presented by Rotation 3 (E.5) and demonstrated in Figure 3.

Figure 4. The dependent variable estimator $\hat{y}$ landing on to the left space of X, meanwhile, the error estimator $\hat{\epsilon }$ landing on to the left null space of X.

As demonstrated in Figure 4, y transforms into $y^{\ast }$ by projecting into $U_1$, the left space of X. And then, multiplied by $U_1$, $y^{\ast }$ transforms into $\hat{y}$ and returns back into the original coordinate system constructed by $a_1,a_2$ and $a_3$. $\hat{y}$ is the projection of y into the left space of X, and perpendicular with error estimator $\hat{\epsilon }$ which is located in the left null space of X.

With the aid of the algorithms like matrix decomposition and Moore-Penrose generalized matrix inverse, the dependent variable y of a multiple linear regression model can be inner-transformed into the regression coefficient estimator $\hat{\beta }$ and its own estimator $\hat{y}$. This process is a new approach to illustrate the inner-transformation between variables from a geometric view as well as presenting the spatial locations of the variables. To date, there is no study to explore the relationship between the variables of the multiple linear regression model from the view of geometric transformation. This study fills such a gap and provides a new perspective for studying multiple linear regression.

The limitation of this work is that a simple example of the multiple linear model is adopted to present such intricate inner transformation. But, it can demonstrate more inference art of inner transformation if more complex examples of multiple linear regression model are adopted in future studies.

Conflicts of Interest

The authors declare no conflicts of interest regarding the publication of this paper.

References

[1]	Mandel, J. (1982) Use of the Singular Value Decomposition in Regression Analysis. The American Statistician, 36, 15-24. https://doi.org/10.1080/00031305.1982.10482771
[2]	Mishra, A., Dey, D.K. and Chen, K. (2017) Sequential Co-Sparse Factor Regression. Journal of Computational and Graphical Statistics, 26, 814-826. https://doi.org/10.1080/10618600.2017.1340891
[3]	Nelder, J.A. (1985) An Alternative Interpretation of Singular Value Decomposition in Regression. The American Statistician, 39, 63-65. https://doi.org/10.2307/2683911
[4]	De Schutter, B. and De Moor, B. (1998) The QR Decomposition and the Singular Value Decomposition in the Symmetrized Max-Plus Algebra. SIAM Journal on Matrix Analysis and Applications, 19, 378-406. https://doi.org/10.1137/S0895479896304782
[5]	Kalman, D. (1996) A Singularly Valuable Decomposition: SVD of a Matrix. The College Mathematics Journal, 27, 2-23. https://doi.org/10.1080/07468342.1996.11973744
[6]	Stewart G.W. (1993) On the Early History of the Singular Value Decomposition. SIAM Review, 35, 551-566. https://doi.org/10.1137/1035134
[7]	Stewart, G.W. (1998) Matrix Algorithms: Volume 1: Basic Decompositions. Society for Industrial and Applied Mathematics, Philadelphia. https://doi.org/10.1137/1.9781611971408
[8]	Ben-Isreal, A. and Charnes, A. (1963) Contributions to the Theory of Generalized Inverses. Journal of the Society for Industrial and Applied Mathematics, 11, 667-699. https://doi.org/10.1137/0111051
[9]	Campbell, S.L. and Meyer, C.D. (1991) Generalized Inverse of Linear Transformations. Dover Publications, New York.
[10]	Fill, J.A. and Fishkind, D.E. (2000) The Moore—Penrose Generalized Inverse for Sums of Matrices. SIAM Journal on Matrix Analysis and Applications, 21, 629-635. https://doi.org/10.1137/S0895479897329692
[11]	Penrose, R. (1955) A Generalized Inverse for Matrices. Mathematical Proceedings of the Cambridge Philosophical Society, 51, 406-413. https://doi.org/10.1017/S0305004100030401
[12]	Wang, G.R., Wei, Y.M. and Qiao, S. (2003) Generalized Inverse: Theory and Computations. Science Press, Beijing.
[13]	Tian, Y. and Zhang, J. (2011) Some Equalities for Estimations of Partial Coefficients under a General Linear Regression Model. Statistical Papers, 52, 911-920. https://doi.org/10.1007/s00362-009-0298-5
[14]	Xu, W. and Tang, X. (2009) Ridge Estimate on Moore-Penrose Inverse Matrix of Generalized Linear Regression Model. Journal of Northeast Forestry University, 37, 108-119.
[15]	Rao, C.R. and Mitra, S.K. (1972) Generalized Inverse of a Matrix and Its Applications. In: Le Cam, L.M., Neyman, J. and Scott, E.L., Eds., Proceedings of the Sixth Berkeley Symposium on Statistics and Probability, Berkeley and Los Angeles, Vol. 1: Theory of Statistics, University of California Press, 355-372.
[16]	Li, Z.Y. (2006) Matris Ayrışımı. Yüksek Lisans Tezi. Master Degree Thesis, İstanbul üniversitesi, İstanbul.
[17]	Piziak, R. and Odell, P.L. (1999) Full Rank Factorization of Matrix. Mathematics Magazine, 72, 193-201. https://doi.org/10.1080/0025570X.1999.11996730