Mathematical Reinforcement to the Minibatch of Deep Learning

Kazuyuki Fujii^*

International College of Arts and Sciences, Yokohama City University, Yokohama, Japan.

DOI: 10.4236/apm.2018.83016                

Abstract

We elucidate a practical method in Deep Learning called the minibatch which is very useful to avoid local minima. The mathematical structure of this method is, however, a bit obscure. We emphasize that a certain condition, which is not explicitly stated in ordinary expositions, is essential for the minibatch method. We present a comprehensive description Deep Learning for non-experts with the mathematical reinforcement.

Keywords

Deep Learning, Minibatch, Error (Loss) Function, Local Minima

1. Introduction

Deep Learning is one of Machine Learning methods and it has attracted much attention recently. Leaning in general is divided into the supervised learning and unsupervised learning. In this paper we discuss only supervised learning. For general introduction to Deep Learning, see for example [1] . The monographs [2] [3] [4] are standard textbooks.

*Dedicated to the memory of Professor Ichiro Yokota.

Deep Learning is based on big data, so the supervised learning will give a heavy load to the computer. In order to alleviate the burden we usually use a practical method called the minibatch (a collection of randomly selected small data from big data, see Figure 5). Although the method is commonly used it is not widely understood why it is so effective from the mathematical viewpoint. We point out in Theorem II that a certain condition, which is practically satisfied in ordinary applications, is essential for the effectiveness of the minibatch method.

In this paper we present a short and concise review of Deep Learning for non-experts and provide a mathematical reinforcement to the minibatch from the viewpoint of Linear Algebra. Theorem I and II in the text are our main results and some related problems are presented.

After reading this paper, readers are encouraged to tackle a remarkable paper [5] which is definitely a monumental achievement.

2. Simple Neural Network

As a general introduction to Neuron Dynamics see for example [6] [7] .

For non-experts let us draw a neuron model based on the step function $S\left(x\right)$ . In Figure 1 the set $\left\{x_1\mathrm{,}{x}_2\mathrm{,}\cdots \mathrm{,}{x}_n\right\}$ is the input data, $z=S\left(y\right)$ is the output data and the set $\left\{w_1\mathrm{,}{w}_2\mathrm{,}\cdots \mathrm{,}{w}_n\right\}$ is the weights of synaptic connections. The parameter q is the threshold of the neuron.

Here, $z=S\left(x\right)$ is a simple step function defined by

$S\left(x\right)=\{\begin{array}{l}0x<0\hfill \\ 1x\ge 0\hfill \end{array}$

Since an algorithm called the backpropagation in Neural Network System uses derivatives of some activation functions, the simple step function $S\left(x\right)$ is not suitable. Therefore, we usually use a function called the sigmoid function instead of the step function (see Figure 2). This function is used for a nonlinear compression of data.

Figure 1. Simple neuron model.

Figure 2. Sigmoid function.

Definition The sigmoid function is defined by

$\sigma \left(x\right)=\frac{1}{1+{\text{e}}^{-x}}\left(x\in R\right)\mathrm{.}$ (1)

It is easy to see that the function satisfies

$\sigma \left(-\infty \right)=0,\sigma \left(0\right)=\frac{1}{2},\sigma \left(\infty \right)=1$

and its derivatives are expressed as polynomials of the original function

$\begin{array}{l}{\sigma }^{\prime }\left(x\right)=\sigma \left(x\right)\left(1-\sigma \left(x\right)\right)\mathrm{,}\\ {\sigma }^{″}\left(x\right)=\sigma \left(x\right)\left(1-\sigma \left(x\right)\right)\left(1-2\sigma \left(x\right)\right)\mathrm{.}\end{array}$ (2)

The graph is as follows.

Comment In Mathematics this is a famous function called the standard logistic function, while in Deep Learning this is called the sigmoid function.

A general logistic function $f\left(x\right)$ has three parameters $\left\{L\mathrm{,}\lambda \mathrm{,}{x}_0\right\}$ , the overall height L, the parameter l specifying the inverse width of the transient region and the location of the center $x_0$ :

$f\left(x\right)=\frac{L}{1+{\text{e}}^{-\lambda \left(x-x_0\right)}}\mathrm{.}$

The standard logistic function is characterized by $L=1,\lambda =1,x_0=0$ . See a lecture note [8] as to why the function is so important.

Let us draw a neuron model based on the sigmoid function, See Figure 3.

The most important thing is to improve the set of weights of the synoptic connections $\left\{w_1\mathrm{,}{w}_2\mathrm{,}\cdots \mathrm{,}{w}_n\right\}$ by hard learning. For the purpose we prepare m input data and teacher signals. In the following we assume $m<n$ and $\theta =0$ for simplicity. This causes no problem for the present explanation. There is a suitable value of q for each specific application.

For $j=1,2,\cdots ,m$ we set

$\text{Input}\text{data}\mathrm{<mo>\; :\; </mo>}{x}^{\left(j\right)}={\left(x_1^{\left(j\right)}\mathrm{,}{x}_2^{\left(j\right)}\mathrm{,}\cdots \mathrm{,}{x}_n^{\left(j\right)}\right)}^{\text{T}}\mathrm{,}$

$\text{Teacher}\text{signal}\mathrm{<mo>\; :\; </mo>}{d}^{(j)}$

where T denotes the transpose of a vector (or a matrix).

Figure 3. Simple neuron model modified.

Our method is to determine the weights of synaptic connections which match the output data with given teacher signals as much as possible. For the purpose we consider a square error like

$E=E\left(w\right)=\frac{1}{2}{\displaystyle \underset{j=1}{\overset{m}{\sum }}}{\left(d^{\left(j\right)}-z^{\left(j\right)}\right)}^2\mathrm{,}$ (3)

$z^{\left(j\right)}=\sigma \left(y^{\left(j\right)}\right)=\sigma \left({\displaystyle \underset{k=1}{\overset{n}{\sum }}}{w}_k{x}_k^{\left(j\right)}\right)$ (4)

and determine $w={\left(w_1\mathrm{,}{w}_2\mathrm{,}\cdots \mathrm{,}{w}_n\right)}^{\text{T}}$ so that $E\left(w\right)$ is minimized.

We want to realize this purpose by repeated learning. For that we consider a discrete dynamical system. Namely, we set

$E=E\left(w\left(t\right)\right)=\frac{1}{2}{\displaystyle \underset{j=1}{\overset{m}{\sum }}}{\left(d^{\left(j\right)}-z^{\left(j\right)}\left(t\right)\right)}^2;z^{\left(j\right)}\left(t\right)=\sigma \left(y^{\left(j\right)}\left(t\right)\right)=\sigma \left({\displaystyle \underset{k=1}{\overset{n}{\sum }}}{w}_k\left(t\right)x_k^{\left(j\right)}\right)$ (5)

for $t=\mathrm{0,1,2,}\cdots \mathrm{,}T\left(\in N\right)$ .

The initial value $w\left(0\right)$ is given appropriately (not so important).

2.1. Gradient Descent

For a general introduction to Gradient Descent see for example [9] .

The gradient descent is performed by changing $w_k\left(t\right)$ to

$w_k\left(t+1\right)=w_k\left(t\right)-ϵ\frac{\partial E\left(w\left(t\right)\right)}{\partial w_k\left(t\right)}\left(0<ϵ<1\right)$ (6)

and we evaluate the error with the renewed weights

$E=E\left(w\left(t+1\right)\right)\mathrm{.}$

This renewal of the weights is the learning process. This is very popular in Mathematics¹. We repeat this procedure T steps which is large enough like

$E\left(w\left(0\right)\right)\Rightarrow E\left(w\left(1\right)\right)\Rightarrow \cdots \Rightarrow E\left(w\left(T\right)\right)\equiv E\left(w\right)\mathrm{.}$

Let us proceed. We set $E=E\left(w\right)$ for simplicity (t omitted). From (5) simple calculation gives

$\begin{array}{c}\frac{\partial E}{\partial w_k}={\displaystyle \underset{j=1}{\overset{m}{\sum }}}\left(d^{\left(j\right)}-z^{\left(j\right)}\right)\left(-\frac{\partial z^{\left(j\right)}}{\partial w_k}\right)\\ =-{\displaystyle \underset{j=1}{\overset{m}{\sum }}}\left(d^{\left(j\right)}-z^{\left(j\right)}\right)\sigma \left({\displaystyle \underset{k=1}{\overset{n}{\sum }}}{w}_k{x}_k^{\left(j\right)}\right)\left\{1-\sigma \left({\displaystyle \underset{k=1}{\overset{n}{\sum }}}{w}_k{x}_k^{\left(j\right)}\right)\right\}{x}_k^{\left(j\right)}\\ =-{\displaystyle \underset{j=1}{\overset{m}{\sum }}}\left(d^{\left(j\right)}-z^{\left(j\right)}\right)x_k^{\left(j\right)}\sigma \left({\displaystyle \underset{k=1}{\overset{n}{\sum }}}{w}_k{x}_k^{\left(j\right)}\right)\left\{1-\sigma \left({\displaystyle \underset{k=1}{\overset{n}{\sum }}}{w}_k{x}_k^{\left(j\right)}\right)\right\}\end{array}$

and from (6) we obtain

$\begin{array}{c}{w}_k\left(t+1\right)=w_k\left(t\right)-ϵ\frac{\partial E\left(w\left(t\right)\right)}{\partial w_k\left(t\right)}\\ =w_k\left(t\right)+ϵ{\displaystyle \underset{j=1}{\overset{m}{\sum }}}\left(d^{\left(j\right)}-z^{\left(j\right)}\left(t\right)\right)x_k^{\left(j\right)}\sigma \left({\displaystyle \underset{k=1}{\overset{n}{\sum }}}{w}_k\left(t\right)x_k^{\left(j\right)}\right)\left\{1-\sigma \left({\displaystyle \underset{k=1}{\overset{n}{\sum }}}{w}_k\left(t\right)x_k^{\left(j\right)}\right)\right\}.\end{array}$

By substituting this into (5) we obtain

$z^{\left(j\right)}\left(t+1\right)=\sigma ({\displaystyle \underset{k=1}{\overset{n}{\sum }}}{w}_k\left(t+1\right)x_k^{(j)})$

and the square error becomes

$E=E\left(w\left(t+1\right)\right)=\frac{1}{2}{\displaystyle \underset{j=1}{\overset{m}{\sum }}}{\left(d^{\left(j\right)}-z^{\left(j\right)}\left(t+1\right)\right)}^2\mathrm{,}$

which is complicated enough. The renewal of the weights terminates when all $\partial E\left(w\right)/\partial w_k=0$ , which is a critical point as shown in the next subsection.

2.2. Number of Critical Points

Here we evaluate the number of critical points to be encountered in the process. First of all, let us show the definition of a critical point with a simple example.

Definition Let $f=f\left(x\mathrm{,}y\mathrm{,}z\right)$ be a smooth function. Then a critical (or stationary) point $\left(x_0\mathrm{,}{y}_0\mathrm{,}{z}_0\right)$ is defined by

$\{\begin{array}{l}{f}_x\left(x_0,y_0,z_0\right)=0,\hfill \\ f_y\left(x_0,y_0,z_0\right)=0,\hfill \\ f_z\left(x_0,y_0,z_0\right)=0.\hfill \end{array}$ (7)

A critical point is not necessarily a point giving local minimum or local maximum.

We make an important assumption with respect to the input data.

Assumption (good data) For $m<n$

$\left\{x^{\left(1\right)}\mathrm{,}{x}^{\left(2\right)}\mathrm{,}\cdots \mathrm{,}{x}^{\left(m\right)}\right\}$ (8)

are linearly independent.

The assumption tells that there is a set of integers $k_1<k_2<\cdots <k_m$ satisfying

$\left|\begin{array}{cccc}{x}_{k_1}^{\left(1\right)}& x_{k_1}^{\left(2\right)}& \cdots & x_{k_1}^{\left(m\right)}\\ x_{k_2}^{\left(1\right)}& x_{k_2}^{\left(2\right)}& \cdots & x_{k_2}^{\left(m\right)}\\ ⋮& ⋮& \ddots & ⋮\\ x_{k_m}^{\left(1\right)}& x_{k_m}^{\left(2\right)}& \cdots & x_{k_m}^{\left(m\right)}\end{array}\right|\ne 0.$ (9)

Let us recall

$E=\frac{1}{2}{\displaystyle \underset{j=1}{\overset{m}{\sum }}}{\left(d^{\left(j\right)}-z^{\left(j\right)}\right)}^2=\frac{1}{2}{\displaystyle \underset{j=1}{\overset{m}{\sum }}}{\left(d^{\left(j\right)}-\sigma \left(y^{\left(j\right)}\right)\right)}^2,\sigma \left(y^{\left(j\right)}\right)=\sigma \left({\displaystyle \underset{k=1}{\overset{n}{\sum }}}{w}_k{x}_k^{(j)}\right)$

and try to find if E has a critical point. Suppose E has a critical point at the weights $w={\left\{w_1\mathrm{,}\cdots \mathrm{,}{w}_n\right\}}^{\text{T}}$ ,

$0=\frac{\partial E}{\partial w_k}=-{\displaystyle \underset{j=1}{\overset{m}{\sum }}}\left(d^{\left(j\right)}-z^{\left(j\right)}\right){\sigma }^{\prime }\left(y^{\left(j\right)}\right)x_k^{\left(j\right)},k=1,\cdots ,n.$

Let us set

$A^{\left(j\right)}=\left(d^{\left(j\right)}-z^{\left(j\right)}\right){\sigma }^{\prime }\left(y^{(j)}\right)$

for simplicity. The equation can be written as

${\displaystyle \underset{j=1}{\overset{m}{\sum }}}{A}^{\left(j\right)}{x}_k^{\left(j\right)}=0\text{for}k=1,\cdots ,n$

and it implies

${\displaystyle \underset{j=1}{\overset{m}{\sum }}}{x}_k^{\left(j\right)}{A}^{\left(j\right)}=0\text{for}k=k_1,k_2,\cdots ,k_m$

or in matrix form

$\left(\begin{array}{cccc}{x}_{k_1}^{\left(1\right)}& x_{k_1}^{\left(2\right)}& \cdots & x_{k_1}^{\left(m\right)}\\ x_{k_2}^{\left(1\right)}& x_{k_2}^{\left(2\right)}& \cdots & x_{k_2}^{\left(m\right)}\\ ⋮& ⋮& \ddots & ⋮\\ x_{k_m}^{\left(1\right)}& x_{k_m}^{\left(2\right)}& \cdots & x_{k_m}^{\left(m\right)}\end{array}\right)\left(\begin{array}{c}{A}^{\left(1\right)}\\ A^{\left(2\right)}\\ ⋮\\ A^{\left(m\right)}\end{array}\right)=\left(\begin{array}{c}0\\ 0\\ ⋮\\ 0\end{array}\right).$

Since the determinant of coefficient matrix is non-zero from (9) we have

$\left(\begin{array}{c}{A}^{\left(1\right)}\\ A^{\left(2\right)}\\ ⋮\\ A^{\left(m\right)}\end{array}\right)=\left(\begin{array}{c}\left(d^{\left(1\right)}-z^{\left(1\right)}\right){\sigma }^{\prime }\left(y^{\left(1\right)}\right)\\ \left(d^{\left(2\right)}-z^{\left(2\right)}\right){\sigma }^{\prime }\left(y^{\left(2\right)}\right)\\ ⋮\\ \left(d^{\left(m\right)}-z^{\left(m\right)}\right){\sigma }^{\prime }\left(y^{\left(m\right)}\right)\end{array}\right)=\left(\begin{array}{c}0\\ 0\\ ⋮\\ 0\end{array}\right)$

and

$d^{\left(j\right)}-z^{\left(j\right)}=d^{\left(j\right)}-\sigma \left(y^{\left(j\right)}\right)=0\text{for}j=1,2,\cdots ,m$

because ${\sigma }^{\prime }\left(y\right)\ne 0$ . This means

$E=\frac{1}{2}{\displaystyle \underset{j=1}{\overset{m}{\sum }}}{\left(d^{\left(j\right)}-z^{\left(j\right)}\right)}^2=0.$

The result shows that there is no critical point in this process except for $E=0$ . Therefore, when modifying $w_k\left(t\right)$ by the gradient descent (6) successively there is no danger of being trapped by points taking local minima. Let us summarize the result :

Theorem I Under the assumption (8) there is only one global minimum $E=0$ .

Next, let us explain how to choose linearly independent data. We assume that n is huge and m is sufficiently smaller than n.

For m input data

$\left\{x^{\left(1\right)}\mathrm{,}{x}^{\left(2\right)}\mathrm{,}\cdots \mathrm{,}{x}^{\left(m\right)}\right\}$

we consider the $m\times m$ Gram’s deterninant

$G\left(x^{\left(1\right)}\mathrm{,}{x}^{\left(2\right)}\mathrm{,}\cdots \mathrm{,}{x}^{\left(m\right)}\right)=\left|\begin{array}{cccc}\langle x^{\left(1\right)}|x^{\left(1\right)}\rangle & \langle x^{\left(1\right)}|x^{\left(2\right)}\rangle & \cdots & \langle x^{\left(1\right)}|x^{\left(m\right)}\rangle \\ \langle x^{\left(2\right)}|x^{\left(1\right)}\rangle & \langle x^{\left(2\right)}|x^{\left(2\right)}\rangle & \cdots & \langle x^{\left(2\right)}|x^{\left(m\right)}\rangle \\ ⋮& ⋮& \ddots & ⋮\\ \langle x^{\left(m\right)}|x^{\left(1\right)}\rangle & \langle x^{\left(m\right)}|x^{\left(2\right)}\rangle & \cdots & \langle x^{\left(m\right)}|x^{\left(m\right)}\rangle \end{array}\right|\mathrm{.}$ (10)

If m is not large, evaluation of the Gram’s determinant is not so difficult. It is well-known that the input data are linearly independent if the Gram’s determinant is non-zero (see for example [10] ).

Let us give a short explanation for non-experts. For simplicity we consider the case of $m=3$ and set

$x^{\left(1\right)}=a,x^{\left(2\right)}=b,x^{\left(3\right)}=c.$

Then

$G\left(a,b,c\right)=\left|\begin{array}{ccc}\langle a|a\rangle & \langle a|b\rangle & \langle a|c\rangle \\ \langle b|a\rangle & \langle b|b\rangle & \langle b|c\rangle \\ \langle c|a\rangle & \langle c|b\rangle & \langle c|c\rangle \end{array}\right|$

and we assume that $\left\{a,b,c\right\}$ are linearly dependent. For example, we can set

$c=\alpha a+\beta b\left(\alpha ,\beta \text{is}\text{real}\right).$

Short calculation after substituting this into the Gram's determinant above gives

$G\left(a,b,c\right)=\left|\begin{array}{ccc}\langle a|a\rangle & \langle a|b\rangle & \langle a|c\rangle \\ \langle b|a\rangle & \langle b|b\rangle & \langle b|c\rangle \\ \alpha \langle a|a\rangle +\beta \langle b|a\rangle & \alpha \langle a|b\rangle +\beta \langle b|b\rangle & \alpha \langle a|c\rangle +\beta \langle b|c\rangle \end{array}\right|$

and some fundamental operations of determinant give

$G\left(a,b,c\right)=\left|\begin{array}{ccc}\langle a|a\rangle & \langle a|b\rangle & \langle a|c\rangle \\ \langle b|a\rangle & \langle b|b\rangle & \langle b|c\rangle \\ 0& 0& 0\end{array}\right|=0.$

Namely, we have shown

$\left\{a,b,c\right\}\text{is}\text{linearly}\text{dependent}\Rightarrow G\left(a,b,c\right)=0.$

Taking the contraposition gives

$G\left(a,b,c\right)\ne 0\Rightarrow \left\{a,b,c\right\}\text{is}\text{linearly}\text{independent}\mathrm{.}$

Comment For $n>m$ we denote by $M\left(n\mathrm{,}m\mathrm{<mo>\; ;\; </mo>}R\right)$ a set of all $n\times m$ matrices over R and denote

$\begin{array}{l}{M}_{n,m}\left(R\right)=\{\left(x^{\left(1\right)},x^{\left(2\right)},\cdots ,x^{\left(m\right)}\right)\in M\left(n,m;R\right)|\\ \left\{x^{\left(1\right)},x^{\left(2\right)},\cdots ,x^{\left(m\right)}\right\}\text{is}\text{linearly}\text{independent}\}.\end{array}$

It is well-known that the subset $M_{n,m}\left(R\right)$ is open and dense in $M\left(n\mathrm{,}m\mathrm{<mo>\; ;\; </mo>}R\right)$ . Therefore, if we choose m vectors from Rⁿ randomly they are almost all linearly independent.

3. Deep Neural Network

Deep Learning is the heart of Artificial Intelligence (AI) and we provide a concise review for non-experts. Deep Learning is a deep neural network consisting of the input layer (K components), the (multiplex) intermediate layers ( $M_1\mathrm{,}{M}_2\mathrm{,}\cdots \mathrm{,}N$ components and the output layer (N components). See Figure 4 (in which $M_1=L,M_2=M,M_3=N$ ).

3.1. Backpropagation

First, let us show a figure of deep neural network² (the figure is short in intermediate layers).

Let us explain some notations.

$x={\left(x_1\mathrm{,}{x}_2\mathrm{,}\cdots \mathrm{,}{x}_K\right)}^{\text{T}}$

is the input data. In this case, the weights of synaptic connections are represented by matrices $U\mathrm{,}V\mathrm{,}W$ given by

$\begin{array}{l}U=\left(u_{ij}\right)\in M\left(L\mathrm{,}K\mathrm{<mo>\; ;\; </mo>}R\right)\mathrm{,}\\ V=\left(v_{ij}\right)\in M\left(M\mathrm{,}L\mathrm{<mo>\; ;\; </mo>}R\right)\mathrm{,}\\ W=\left(w_{ij}\right)\in M\left(N\mathrm{,}M\mathrm{<mo>\; ;\; </mo>}R\right)\end{array}$ (11)

and our aim is educating these matrices.

The synaptic connections are expressed as

$r_i={\displaystyle \underset{j=1}{\overset{M}{\sum }}}{w}_{ij}{\beta }_j-\theta ,i=1,\cdots ,N$

$z_i=\sigma (ri)$

and

$q_i={\displaystyle \underset{j=1}{\overset{L}{\sum }}}{v}_{ij}{\alpha }_j-\theta ,i=1,\cdots ,M$

${\beta }_i=\sigma (qi)$

and

$p_i={\displaystyle \underset{j=1}{\overset{K}{\sum }}}{u}_{ij}{x}_j-\theta ,i=1,\cdots ,L$

${\alpha }_i=\sigma (pi)$

respectively. See Figure 4 once more.

A comment is in order. The flow of data is as follows:

$x=\left(\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_K\end{array}\right)\Rightarrow \alpha =\left(\begin{array}{c}{\alpha }_1\\ {\alpha }_2\\ ⋮\\ {\alpha }_L\end{array}\right)\Rightarrow \beta =\left(\begin{array}{c}{\beta }_1\\ {\beta }_2\\ ⋮\\ {\beta }_M\end{array}\right)\Rightarrow z=\left(\begin{array}{c}{z}_1\\ z_2\\ ⋮\\ z_N\end{array}\right).$ (12)

Now, if

$d={\left(d_1\mathrm{,}{d}_2\mathrm{,}\cdots \mathrm{,}{d}_N\right)}^{\text{T}}$

is a set of teacher signals, then the square error becomes

Figure 4. Deep neural network model.

$E=E\left(U,V,W\right)=\frac{1}{2}{\displaystyle \underset{k=1}{\overset{N}{\sum }}}{\left(d_k-z_k\right)}^2.$

Let us consider (discrete) time evolution of the model. For $t=0,1,2,\cdots$ , we must determine

$t\to t+1$

$U\left(t\right)\to U\left(t+1\right)\mathrm{,}$

$V\left(t\right)\to V\left(t+1\right)\mathrm{,}$

$W\left(t\right)\to W\left(t+1\right)\mathrm{,}$

$E\left(t\right)\to E\left(t+1\right)$

and we expect

$E\left(t\right)\approx 0$

if t is large enough (the end of Learning !).

Assume now that the system is in the t-th step

$E\left(t\right)=E\left(U\left(t\right),V\left(t\right),W\left(t\right)\right)=\frac{1}{2}{\displaystyle \underset{k=1}{\overset{N}{\sum }}}{\left(d_k-z_k\left(t\right)\right)}^2.$ (13)

Then the time evolution of the weights of synaptic connections is determined by the gradient descent:

$\begin{array}{l}{w}_{ij}\left(t+1\right)=w_{ij}\left(t\right)-ϵ\frac{\partial E\left(t\right)}{\partial w_{ij}\left(t\right)},\\ v_{ij}\left(t+1\right)=v_{ij}\left(t\right)-ϵ\frac{\partial E\left(t\right)}{\partial v_{ij}\left(t\right)},\\ u_{ij}\left(t+1\right)=u_{ij}\left(t\right)-ϵ\frac{\partial E\left(t\right)}{\partial u_{ij}\left(t\right)},\end{array}$ (14)

which is a natural generalization of that of Section 2.1.

Before the calculation let us make a comment on the derivatives of a composite function.

Comment Let us explain with a simple example. For a composite function

$x\to f\left(x\right)\to g\left(f\left(x\right)\right)\to E\left(g\left(f\left(x\right)\right)\right)$

the derivative is given by

$\frac{\text{d}E}{\text{d}x}=\frac{\text{d}E\left(g\left(f\left(x\right)\right)\right)}{\text{d}g\left(f\left(x\right)\right)}\frac{\text{d}g\left(f\left(x\right)\right)}{\text{d}f\left(x\right)}\frac{\text{d}f\left(x\right)}{\text{d}x}=E^{\prime }\left(g\left(f\left(x\right)\right)\right)g^{\prime }\left(f\left(x\right)\right)f^{\prime }\left(x\right).$

A composite function is in general complicated, while its derivative is not so complicated.

3.2. Calculation

Let us perform the calculation of (14).

1) Case of $W\left(t\right)=\left(w_{ij}\left(t\right)\right)$ : From

$\begin{array}{c}\frac{\partial E\left(t\right)}{\partial w_{ij}\left(t\right)}=\frac{\partial E\left(t\right)}{\partial z_i\left(t\right)}\frac{\partial z_i\left(t\right)}{\partial r_i\left(t\right)}\frac{\partial r_i\left(t\right)}{\partial w_{ij}\left(t\right)}\\ =-\left(d_i-z_i\left(t\right)\right){\sigma }^{\prime }\left(r_i\left(t\right)\right){\beta }_j\left(t\right)\\ =-\left(d_i-z_i\left(t\right)\right)\sigma \left(r_i\left(t\right)\right)\left(1-\sigma \left(r_i\left(t\right)\right)\right){\beta }_j(t)\end{array}$

we obtain

$w_{ij}\left(t+1\right)=w_{ij}\left(t\right)+ϵ\left(d_i-z_i\left(t\right)\right)\sigma \left(r_i\left(t\right)\right)\left(1-\sigma \left(r_i\left(t\right)\right)\right){\beta }_j\left(t\right).$

2) Case of $V\left(t\right)=\left(v_{ij}\left(t\right)\right)$ : From

$\begin{array}{c}\frac{\partial E\left(t\right)}{\partial v_{ij}\left(t\right)}={\displaystyle \underset{k=1}{\overset{N}{\sum }}}\frac{\partial E\left(t\right)}{\partial z_k\left(t\right)}\frac{\partial z_k\left(t\right)}{\partial r_k\left(t\right)}\frac{\partial r_k\left(t\right)}{\partial {\beta }_i\left(t\right)}\frac{\partial {\beta }_i\left(t\right)}{\partial q_i\left(t\right)}\frac{\partial q_i\left(t\right)}{\partial v_{ij}\left(t\right)}\\ =-{\displaystyle \underset{k=1}{\overset{N}{\sum }}}\left(d_k-z_k\left(t\right)\right){\sigma }^{\prime }\left(r_k\left(t\right)\right)w_{ki}\left(t\right){\sigma }^{\prime }\left(q_i\left(t\right)\right){\alpha }_j\left(t\right)\\ =-{\displaystyle \underset{k=1}{\overset{N}{\sum }}}\left(d_k-z_k\left(t\right)\right)\sigma \left(r_k\left(t\right)\right)\left(1-\sigma \left(r_k\left(t\right)\right)\right)\\ \times w_{ki}\left(t\right)\sigma \left(q_i\left(t\right)\right)\left(1-\sigma \left(q_i\left(t\right)\right)\right){\alpha }_j(t)\end{array}$

we have

$\begin{array}{c}{v}_{ij}\left(t+1\right)=v_{ij}\left(t\right)+ϵ{\displaystyle \underset{k=1}{\overset{N}{\sum }}}\left(d_k-z_k\left(t\right)\right)\sigma \left(r_k\left(t\right)\right)\left(1-\sigma \left(r_k\left(t\right)\right)\right)\\ \times w_{ki}\left(t\right)\sigma \left(q_i\left(t\right)\right)\left(1-\sigma \left(q_i\left(t\right)\right)\right){\alpha }_j\left(t\right).\end{array}$

3) Case of $U\left(t\right)=\left(u_{ij}\left(t\right)\right)$ : From

$\begin{array}{c}\frac{\partial E\left(t\right)}{\partial u_{ij}\left(t\right)}={\displaystyle \underset{l=1}{\overset{N}{\sum }}}{\displaystyle \underset{k=1}{\overset{M}{\sum }}}\frac{\partial E\left(t\right)}{\partial z_l\left(t\right)}\frac{\partial z_l\left(t\right)}{\partial r_l\left(t\right)}\frac{\partial r_l\left(t\right)}{\partial {\beta }_k\left(t\right)}\frac{\partial {\beta }_k\left(t\right)}{\partial q_k\left(t\right)}\frac{\partial q_k\left(t\right)}{\partial {\alpha }_i\left(t\right)}\frac{\partial {\alpha }_i\left(t\right)}{\partial p_i\left(t\right)}\frac{\partial p_i\left(t\right)}{\partial u_{ij}\left(t\right)}\\ =-{\displaystyle \underset{l=1}{\overset{N}{\sum }}}{\displaystyle \underset{k=1}{\overset{M}{\sum }}}\left(d_l-z_l\left(t\right)\right){\sigma }^{\prime }\left(r_l\left(t\right)\right)w_{lk}\left(t\right){\sigma }^{\prime }\left(q_k\left(t\right)\right)v_{ki}\left(t\right){\sigma }^{\prime }\left(p_i\left(t\right)\right)x_j\end{array}$

we have

$\begin{array}{l}{u}_{ij}\left(t+1\right)\\ =u_{ij}\left(t\right)+ϵ{\displaystyle \underset{l=1}{\overset{N}{\sum }}}{\displaystyle \underset{k=1}{\overset{M}{\sum }}}\left(d_l-z_l\left(t\right)\right)\sigma \left(r_l\left(t\right)\right)\left(1-\sigma \left(r_l\left(t\right)\right)\right)w_{lk}\left(t\right)\\ \times \sigma \left(q_k\left(t\right)\right)\left(1-\sigma \left(q_k\left(t\right)\right)\right)v_{ki}\left(t\right)\sigma \left(p_i\left(t\right)\right)\left(1-\sigma \left(p_i\left(t\right)\right)\right)x_j.\end{array}$

The calculation so far is based on one input data, for simplicity. However, we must treat a lot of data simultaneously. Let us denote by a the number of data to be considered.

For $l=1,2,\cdots ,a$ we set

$\text{Input}\text{Data}{x}^{\left(l\right)}={\left(x_1^{\left(l\right)}\mathrm{,}{x}_2^{\left(l\right)}\mathrm{,}\cdots \mathrm{,}{x}_K^{\left(l\right)}\right)}^{\text{T}}\mathrm{,}$

$\text{Teacher}\text{Signal}{d}^{\left(l\right)}={\left(d_1^{\left(l\right)},d_2^{\left(l\right)},\cdots ,d_N^{\left(l\right)}\right)}^{\text{T}},$

$\text{Output}\text{Data}{z}^{\left(l\right)}={\left(z_1^{\left(l\right)},z_2^{\left(l\right)},\cdots ,z_N^{\left(l\right)}\right)}^{\text{T}}.$

By taking all these into consideration the square error function in (13) changes to

$\hat{E}\left(t\right)={\displaystyle \underset{l=1}{\overset{a}{\sum }}}{E}^{\left(l\right)}\left(t\right)={\displaystyle \underset{l=1}{\overset{a}{\sum }}}\left\{\frac{1}{2}{\displaystyle \underset{k=1}{\overset{N}{\sum }}}{\left(d_k^{\left(l\right)}-z_k^{\left(l\right)}\left(t\right)\right)}^2\right\}$ (15)

at the t-th step.

In this case the calculation is very simple. For example, since

$w_{ij}\left(t+1\right)=w_{ij}\left(t\right)-ϵ\frac{\partial \hat{E}\left(t\right)}{\partial w_{ij}\left(t\right)}=w_{ij}\left(t\right)-ϵ{\displaystyle \underset{l=1}{\overset{a}{\sum }}}\frac{\partial E^{\left(l\right)}\left(t\right)}{\partial w_{ij}(t)}$

it is only changed to

$w_{ij}\left(t+1\right)=w_{ij}\left(t\right)+ϵ{\displaystyle \underset{l=1}{\overset{a}{\sum }}}\left(d_i^{\left(l\right)}-z_i^{\left(l\right)}\left(t\right)\right)\sigma \left(r_i^{\left(l\right)}\left(t\right)\right)\left(1-\sigma \left(r_i^{\left(l\right)}\left(t\right)\right)\right){\beta }_j^{\left(l\right)}\left(t\right).$

The calculation for both $v_{ij}\left(t\right)$ and $u_{ij}\left(t\right)$ is quite similar (omitted).

In the following we make a mysterious assumption.

Assumption We assume that the data in the final step of the intermediate layers are linearly independent:

$\begin{array}{l}1)a<M,\\ 2)\left\{{\beta }^{\left(1\right)},{\beta }^{\left(2\right)},\cdots ,{\beta }^{\left(a\right)}\right\}\text{are}\text{linearly}\text{independent}\mathrm{.}\end{array}$ (16)

See Figure 4 and the flow of data (12).

We can show that there is no critical point in our system except for $\hat{E}=0$ . Therefore, when modifying $u_{ij}\left(t\right)\mathrm{,}{v}_{ij}\left(t\right)\mathrm{,}{w}_{ij}\left(t\right)$ by the gradient descent (14) (replaced by $\hat{E}$ ) successively there is no danger of being trapped by points taking local minima.

Proof: The proof is essentially the same as that of Section 2.2. However, the proof is the heart of the paper, so we repeat it.

A critical point is defined by the equations

$\{\begin{array}{l}\frac{\partial \hat{E}\left(t\right)}{\partial w_{ij}\left(t\right)}=0,\hfill \\ \frac{\partial \hat{E}\left(t\right)}{\partial v_{ij}\left(t\right)}=0,\hfill \\ \frac{\partial \hat{E}\left(t\right)}{\partial u_{ij}\left(t\right)}=0.\hfill \end{array}$

From (15) we have and set

$\begin{array}{c}0=\frac{\partial \hat{E}\left(t\right)}{\partial w_{ij}\left(t\right)}=-{\displaystyle \underset{l=1}{\overset{a}{\sum }}}\left(d_i^{\left(l\right)}-z_i^{\left(l\right)}\left(t\right)\right){\sigma }^{\prime }\left(r_i^{\left(l\right)}\left(t\right)\right){\beta }_j^{\left(l\right)}\left(t\right)\\ ={\displaystyle \underset{l=1}{\overset{a}{\sum }}}{\beta }_j^{\left(l\right)}\left(t\right)\left(d_i^{\left(l\right)}-z_i^{\left(l\right)}\left(t\right)\right){\sigma }^{\prime }\left(r_i^{\left(l\right)}\left(t\right)\right)\left(-\text{is}\text{omitted}\right)\\ \equiv {\displaystyle \underset{l=1}{\overset{a}{\sum }}}{\beta }_j^{\left(l\right)}\left(t\right)A_i^{\left(l\right)}\left(j=1,2,\cdots ,M\right)\end{array}$

and it implies

$0={\displaystyle \underset{l=1}{\overset{a}{\sum }}}{\beta }_j^{\left(l\right)}\left(t\right)A_i^{\left(l\right)}\left(j=j_1<j_2<\cdots <j_a\right).$

Therefore, its matrix form becomes

$\left(\begin{array}{cccc}{\beta }_{j_1}^{\left(1\right)}& {\beta }_{j_1}^{\left(2\right)}& \cdots & {\beta }_{j_1}^{\left(a\right)}\\ {\beta }_{j_2}^{\left(1\right)}& {\beta }_{j_2}^{\left(2\right)}& \cdots & {\beta }_{j_2}^{\left(a\right)}\\ ⋮& ⋮& \ddots & ⋮\\ {\beta }_{j_a}^{\left(1\right)}& {\beta }_{j_a}^{\left(2\right)}& \cdots & {\beta }_{j_a}^{\left(a\right)}\end{array}\right)\left(\begin{array}{c}{A}_i^{\left(1\right)}\\ A_i^{\left(2\right)}\\ ⋮\\ A_i^{\left(a\right)}\end{array}\right)=\left(\begin{array}{c}0\\ 0\\ ⋮\\ 0\end{array}\right).$

Since the determinant of coefficient matrix is non-zero from (16) we have

$\left(\begin{array}{c}{A}_i^{\left(1\right)}\\ A_i^{\left(2\right)}\\ ⋮\\ A_i^{\left(a\right)}\end{array}\right)=\left(\begin{array}{c}0\\ 0\\ ⋮\\ 0\end{array}\right)$

and

$A_i^l\equiv \left(d_i^{\left(l\right)}-z_i^{\left(l\right)}\left(t\right)\right){\sigma }^{\prime }\left(r_i^{\left(l\right)}\left(t\right)\right)=0\left(l=1,2,\cdots ,a\right)$

and ${\sigma }^{\prime }\left(r_i^{\left(l\right)}\left(t\right)\right)\ne 0$ gives

$d_i^{\left(l\right)}-z_i^{\left(l\right)}\left(t\right)=0\left(l=1,2,\cdots ,a\right).$

Moreover, these equations are independent of $i=1,2,\cdots ,N$ , so that we finally obtain

$\hat{E}\left(t\right)={\displaystyle \underset{l=1}{\overset{a}{\sum }}}{E}^{\left(l\right)}\left(t\right)={\displaystyle \underset{l=1}{\overset{a}{\sum }}}\left\{\frac{1}{2}{\displaystyle \underset{k=1}{\overset{N}{\sum }}}{\left(d_k^{\left(l\right)}-z_k^{\left(l\right)}\left(t\right)\right)}^2\right\}=0.$

Let us summarize the result :

Theorem II Under the assumption (16) there is only one global minimum $\hat{E}=0$ .

Let us emphasize that the assumption (16) is “local” because it refers to the linear independence of the data of the layer one step before the last of the flow:

$x=\left(\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_K\end{array}\right)\Rightarrow \alpha =\left(\begin{array}{c}{\alpha }_1\\ {\alpha }_2\\ ⋮\\ {\alpha }_L\end{array}\right)\Rightarrow \beta =\left(\begin{array}{c}{\beta }_1\\ {\beta }_2\\ ⋮\\ {\beta }_M\end{array}\right)\Rightarrow z=\left(\begin{array}{c}{z}_1\\ z_2\\ ⋮\\ z_N\end{array}\right).$

A question arises naturally.

Problem I What is the meaning of the assumption in the total flow of data ?

Last in this section let us comment on the minibatch of Deep Learning. When input data A is huge the calculation of time evolution of the weights of synaptic connections will give a heavy load to the computer. In order to alleviate it the minibatch is practical and very useful, see Figure 5.

Figure 5. Minibatch.

Then a question also arises naturally.

Problem II Is there a relation between A and T?

One may conjecture

$T=\left[A/a\right]$

where [k] is the Gauss symbol, which is the greatest integer less than or equal to k. For example, [3.14] = 3. However, I do not believe this is correct.

4. Concluding Remarks

Nowadays, studying Deep Learning is standard for university students in the world. However, it is not so easy for them because Deep Learning requires a lot of knowledge.

In this paper, we treated the minibatch of Deep Learning and gave the mathematical reinforcement to it from the viewpoint of Linear Algebra and presented some related problems. We expect that young researchers will solve the problems in the near future.

As a related topic of Artificial Intelligence, there is Information Retrieval. Since there is no more space, we only refer to some relevant works [11] [12] [13] .

Acknowledgements

We wish to thank Ryu Sasaki for useful suggestions and comments.

NOTES

¹However, to choose ò in (6) correctly is a very hard problem.

²Figure 4 is of course not perfect because drawing a figure of a big size is not easy.

Conflicts of Interest

The authors declare no conflicts of interest.

References

[1]	Wikipedia. Deep Learning. https://en.wikipedia.org/wiki/Deep_learning
[2]	Patterson, J. and Gibson, A. (2017) Deep Learning: A Practitioner’s Approach. O’Reilly Media, Inc., Sebastopol, CA.
[3]	Okaya, T. (2015) Deep Learning. Kodansha Ltd., Tokyo. (In Japanese)
[4]	Mizoguchi, R. and Ishida, R. (2000) Artificial Intelligence. Ohmsha Ltd., Tokyo. (In Japanese)
[5]	Le, Q.V., et al. (2012) Building High-Level Features Using Large Scale Unsupervised Learning. Proceedings of the 29th International Conference on Machine Learning, Edinburgh, June 27th-July 3rd, 2012, 11.
[6]	Aihara, K. and Kanzaki, R. (2008) Theoretical and Engineering Approaches to Brainscience. University of Tokyo Press, Tokyo. (In Japanese)
[7]	Amari, S. (2016) Brain·Heart·Artificial Intelligence. Blue Backs B-1968. Kodansha Ltd., Tokyo. (In Japanese)
[8]	Fujii, K. (2014) Verhulst Model of Population Growth. Lecture Note. Yokohama City University, Yokohama. (In Japanese)
[9]	Wikipedia. Gradient Descent. https://en.wikipedia.org/wiki/Gradient_descent
[10]	Kasahara, K. (2000) Linear Algebra. Saiensu Ltd., Tokyo. (In Japanese)
[11]	Kita, K., Tsuda, K. and Shishibori, M. (2002) Information Retrieval Algorithms. Kyoritsu Shuppann, Ltd., Tokyo. (In Japanese)
[12]	Fujii, K. and Oike, H. (2015) A Superintroduction to Google Matrices for Undergraduates. Far East Journal of Mathematical Education, 14, 55-68. https://doi.org/10.17654/FJMEFeb2015_055_068
[13]	Fujii, K. and Oike, H. (2015) How to Understand Google from the Mathematical Point of View. Yokohama City University, Yokohama.