Solving Neumann Boundary Problem with Kernel-Regularized Learning Approach

Xuexue Ran, Baohuai Sheng

Department of Mathematics, Shaoxing University, Shaoxing, China.

DOI: 10.4236/jamp.2024.124069                

Abstract

We provide a kernel-regularized method to give theory solutions for Neumann boundary value problem on the unit ball. We define the reproducing kernel Hilbert space with the spherical harmonics associated with an inner product defined on both the unit ball and the unit sphere, construct the kernel-regularized learning algorithm from the view of semi-supervised learning and bound the upper bounds for the learning rates. The theory analysis shows that the learning algorithm has better uniform convergence according to the number of samples. The research can be regarded as an application of kernel-regularized semi-supervised learning.

Keywords

Neumann Boundary Value, Kernel-Regularized Approach, Reproducing Kernel Hilbert Space, The Unit Ball, The Unit Sphere

1. Introduction

It is known that approximation theory and skills have been used to give the analytic solution for PDE with boundary value problems and form the method of fundamental solutions (see e.g. [1] - [6] ; Appendices 1-3). Recently, the kernel-based collocation method for solving several PDEs with boundary problems has been developed (see e.g. [7] [8] [9] [10] [11] ) from the view of minimal norm interpolation of reproducing kernel Hilbert spaces (RKHSs), the existent theorem and the representation theorem for the numerical solutions are shown qualitatively. It is suggested by [12] that kernel-regularized gradient learning may be used to give numerical solution for PDE. Indeed, some kernel-regularized learning algorithms have been used to study the PDE with Dirichlet boundary value problem quantitatively (see e.g. [13] [14] [15] [16] ). For a given domain $D\subset R^d$ , the pairwise distinct collocation points chosen in the kernel-based collocation method are:

$X_D=\left\{x_1\mathrm{,}\cdots \mathrm{,}{x}_N\right\}\subset D\mathrm{,}$

$X_{\partial D}=\left\{x_{N+1}\mathrm{,}\cdots \mathrm{,}{x}_{N+M}\right\}\subset \partial D,$

and the sample values from the PDE (where P and B are given differential operators):

$\{\begin{array}{l}Pu=f^{\mathrm{<mo>\; *\; </mo>}}\mathrm{,}\text{in}D\subset R^d\mathrm{,}\\ Bu=g\mathrm{,}\text{on}\partial D\end{array}$ (1)

at the given collocation, points are (see [7] [11] ):

$y_j=f^{\mathrm{<mo>\; *\; </mo>}}\left(x_j\right)+{\eta }_{x_j}\mathrm{,}j=\mathrm{1,2,}\cdots \mathrm{,}N\mathrm{,}$

$y_k=g\left(x_{N+k}\right),k=1,2,\cdots ,M,$

where the random variable $\stackrel{\to }{\eta }\mathrm{<mo>\; :\; </mo>}={\left({\eta }_{x_1}\mathrm{,}\cdots \mathrm{,}{\eta }_{x_N}\right)}^{{\rm T}}~\mathcal{N}\left(\stackrel{\to }{0}\mathrm{,}{\stackrel{\to }{\Psi }}_{\partial D}\right)$ .

There are two typical PDE problems (see Chapter 1 of [17] ). When $P=\nabla ={\displaystyle \underset{j=1}{\overset{d}{\sum }}}\frac{{\partial }^2}{\partial x_j^2}$ is the Laplace operator and $B=I$ is the unit operator, we have the Dirichlet problem:

$\{\begin{array}{l}\Delta u=f^{\mathrm{<mo>\; *\; </mo>}}\mathrm{,}\text{in}D\subset R^d\mathrm{,}\\ u=g\mathrm{,}\text{on}\partial D\end{array}$ (2)

When $P=I$ is the unit operator and $B=\frac{\partial }{\partial \stackrel{\to }{n}}$ is the directional derivative along the outward normal vector $\stackrel{\to }{n}$ , i.e. $Bu=\frac{\partial u}{\partial \stackrel{\to }{n}}=\nabla u\cdot \stackrel{\to }{n}$ , problem (1) become the Neumann boundary problem:

$\{\begin{array}{l}u=f^{\mathrm{<mo>\; *\; </mo>}}\mathrm{,}\text{in}D\subset R^d\mathrm{,}\\ \frac{\partial u}{\partial \stackrel{\to }{n}}=g\mathrm{,}\text{on}\partial D\mathrm{.}\end{array}$ (3)

The observations ${\left\{\left(x_j\mathrm{,}{y}_j\right)\right\}}_{j=1}^N$ can be regarded as the supervised labeled observations in the setting of semi-supervised learning and $y_k\left(k=\mathrm{1,2,}\cdots \mathrm{,}M\right)$ maybe regarded as the unlabeled ones. This similarity encourages us to construct kernel-regularized learning algorithms to give numerical solution for problem (3) referring to the kernel semi-supervised learning frameworks (see e.g. [18] [19] [20] [21] ). Along this line, we constructed in [15] a kind of kernel-regularized learning algorithm for solving problem (2) and showed the convergence rate. In the present paper, we shall construct a kind of kernel-regularized regression algorithm to solve problem (2) in case that D is the unit ball and $\partial D$ is the unit sphere. To this aim, we restate problem (3) in the setting of Sobolev spaces.

Let $\Omega \subset R^d$ be a given bounded closed domain with a smoothness surface (i.e. the outward normal derivative is continuous) and ${\rho }_{\Omega }$ be a Borel measure on Ω. Let $C^{\left(s\right)}\left(\Omega \right)$ denote the set of functions such that ${\partial }_x^{\alpha }f\left(x\right)\in C\left(\Omega \right)$ and $\left|\alpha \right|\le s$ , where for $\alpha =\left({\alpha }_1\mathrm{,}\cdots \mathrm{,}{\alpha }_d\right)\in Z_{+}^d$ we define $\left|\alpha \right|={\displaystyle \underset{i=1}{\overset{d}{\sum }}}{\alpha }_i$ and

${\partial }^{\alpha }f\left(x\right)\mathrm{<mo>\; :\; </mo>}={\partial }_x^{\alpha }f\left(x\right)={\partial }_1^{{\alpha }_1}\cdots {\partial }_d^{{\alpha }_d}f\left(x\right)=\frac{{\partial }^{\left|\alpha \right|}f\left(x\right)}{\partial {\left(x^1\right)}^{{\alpha }_1}\cdots {\left(x^d\right)}^{{\alpha }_d}}\mathrm{.}$

For a $K\in C^{\left(s\right)}\left(\Omega \times \Omega \right)$ , we define:

${\partial }_x^{\alpha }{\partial }_y^{\beta }K\left(x\mathrm{,}y\right)=\frac{{\partial }^{\left|\alpha \right|+\left|\beta \right|}K\left(x\mathrm{,}y\right)}{\partial {\left(x^1\right)}^{{\alpha }_1}\cdots {\left(x^d\right)}^{{\alpha }_d}\partial {\left(y^1\right)}^{{\beta }_1}\cdots {\left(y^d\right)}^{{\beta }_d}}\mathrm{,}x=\left(x^1\mathrm{,}\cdots \mathrm{,}{x}^d\right)\mathrm{,}y=\left(y^1\mathrm{,}\cdots \mathrm{,}{y}^d\right)\mathrm{.}$

Denoted by $W^1\left({\rho }_{\Omega }\right)$ , the set of all functions whose 1-order partial derivatives are all in $L^2\left({\rho }_{\Omega }\right)$ , i.e.

$W^1\left({\rho }_{\Omega }\right)=\left\{f\mathrm{<mo>\; :\; </mo>}{\Vert f\Vert }_{W^1\left({\rho }_{\Omega }\right)}={\left({\displaystyle \underset{\left|\alpha \right|\le 1}{\sum }}{\Vert D^{\alpha }f\Vert }_{\mathrm{2,}{\rho }_{\Omega }}^2\right)}^{1/2}<+\infty \right\}\mathrm{,}$

and $L^2\left({\rho }_{\Omega }\right)=\left\{f\left(x\right)\mathrm{<mo>\; :\; </mo>}{\Vert f\Vert }_{\mathrm{2,}{\rho }_{\Omega }}={\left({\displaystyle {\int }_{\Omega }}{\left|f\left(x\right)\right|}^2\text{d}{\rho }_{\Omega }\right)}^{\frac{1}{2}}<+\infty \right\}$ . Defined by $\frac{\partial }{\partial \stackrel{\to }{n}}$ , the outward normal derivative operator, i.e. $\frac{\partial f\left(x\right)}{\partial \stackrel{\to }{n}}\mathrm{<mo>\; :\; </mo>}=\frac{{\partial }_{x}f\left(x\right)}{\partial \stackrel{\to }{n}}=\nabla f\left(x\right)\cdot \stackrel{\to }{n}$ , where $\nabla f\left(x\right)=\left(\frac{\partial f}{\partial x^1}\mathrm{,}\cdots \mathrm{,}\frac{\partial f}{\partial x^d}\right)$ and $\stackrel{\to }{n}$ is the outward normal vector at $x=\left\{x^1\mathrm{,}\cdots \mathrm{,}{x}^d\right\}\in \partial \Omega$ .

To borrow the setting of learning theory (see [22] ), we rewrite the problem (3). Let $f^{\mathrm{<mo>\; *\; </mo>}}$ be an unknown function and g be a given function. Then, the collocation points $z=\left\{x_1\mathrm{,}\cdots \mathrm{,}{x}_m\right\}\subset int\Omega$ and $\nu =\left\{x_{m+1}\mathrm{,}\cdots \mathrm{,}{x}_{m+u}\right\}\subset \partial \Omega$ for the Neumann boundary problem:

$\{\begin{array}{ll}f\left(x\right)=f^{\mathrm{<mo>\; *\; </mo>}}\left(x\right)\mathrm{,}\hfill & x\in \Omega \mathrm{,}\hfill \\ \frac{{\partial }_{x}f\left(x\right)}{\partial \stackrel{\to }{n}}=g\left(x\right)\mathrm{,}\hfill & x\in \partial \Omega \hfill \end{array}$ (4)

are scattered points with values $y_{x_i}=f^{\mathrm{<mo>\; *\; </mo>}}\left(x_i\right)+{\eta }_{x_i}$ , $i=\mathrm{1,2,}\cdots \mathrm{,}m$ , and $y_{x_i}=g\left(x_i\right)$ , $i=m+\mathrm{1,}m+\mathrm{2,}\cdots \mathrm{,}m+u$ , where for a given $x_i=\left(x_i^1\mathrm{,}\cdots \mathrm{,}{x}_i^d\right)\in \Omega$ , ${\xi }_{x_i}$ is a random variable subject to a condition distribution $\rho \left(y\mathrm{<mo>\; |\; </mo>}{x}_i\right)$ satisfying $\left|\rho \left(y\mathrm{<mo>\; |\; </mo>}{x}_i\right)\right|\le B$ , B is a given constant number, $E_{\rho \left(\cdot \mathrm{<mo>\; |\; </mo>}x\right)}\left({\eta }_x\right)={\displaystyle {\int }_{\left[-B\mathrm{,}B\right]}}{\eta }_x\left(y\right)\text{d}\rho \left(y\mathrm{<mo>\; |\; </mo>}x\right)=0$ and $\sigma ={\left({\displaystyle \underset{i=1}{\overset{m}{\sum }}}{\sigma }_{x_i}^2\right)}^{\frac{1}{2}}<+\infty$ and ${\sigma }_x^2=E_{\rho \left(\cdot \mathrm{<mo>\; |\; </mo>}x\right)}\left({\eta }_x^2\right)$ . The correspondence of (4) is:

$\{\begin{array}{ll}f\left(x_i\right)=y_{x_i}\mathrm{,}\hfill & i=\mathrm{1,2,}\cdots \mathrm{,}m\mathrm{,}\hfill \\ {\frac{{\partial }_{x}f\left(x\right)}{\partial \stackrel{\to }{n}}|}_{x=x_i}=g\left(x_i\right)\mathrm{,}\hfill & i=m+\mathrm{1,}\cdots \mathrm{,}m+u\hfill \end{array}$ (5)

We shall give an investigation on the convergence analysis of problem (5) when $\Omega$ is the unit ball $B^d=\left\{x\in R^d\mathrm{<mo>\; :\; </mo>}\Vert x\Vert \le 1\right\}$ and $\partial \Omega$ is the unit sphere $S^{d-1}=\left\{x\in R^d\mathrm{<mo>\; :\; </mo>}\Vert x\Vert =1\right\}$ . The paper is organized as follows. In Section 2.1, we shall provide some notions on the kernel-regularized regression learning, which contain the concept of reproducing kernel Hilbert spaces, the concept of kernel-regularized regression learning model and the kernel-regularized semi-supervised regression learning model. In Section 2.2, we shall provide some notions and results on spherical analysis. In particular, we shall define an RKHS with spherical harmonics, which have the reproducing property for the outward normal vector operator. The Neumann boundary value problem with the RKHS as the hypothesis space is defined. Based on these notions, we define in Section 2.3 a kind of kernel-regularized learning algorithm for solving the Neumann boundary value problem, show the representation theorem and give the error decomposition, with which we give the learning rates (i.e. the main Theorem 2.1) in Section 2.4. In Section 3, we shall give some lemmas, which will be used to show Theorem 2.1 in Section 4. Section 5 is the appendices containing some knowledge about the convex function, a kind of RKHS ${\mathcal{H}}_K^{\stackrel{\to }{n}}\left({\rho }_{\Omega }\right)$ defined in a Sobolev space $H^{\stackrel{\to }{n}}\left({\rho }_{\Omega }\right)$ associating a general domain $\Omega$ and its boundary $\partial \Omega$ , and a probability inequality defined on a general RKHS.

Throughout the paper, we denote by $A=O\left(B\right)$ the fact that there is a constant C independent of A and B such that $A\le CB$ . We say $a~B$ if both $A=O\left(B\right)$ and $B=O\left(A\right)$ .

2. Kernel-Regularized Regression and Error Analysis

We first provide some notions and results about the kernel-regularized regression learning problem.

2.1. Notions and Results on Kernel-Regularized Regression

Follow all the definitions and notions in Section 1. Let $K_x\left(y\right)=K\left(x\mathrm{,}y\right)\mathrm{<mo>\; :\; </mo>}\Omega \times \Omega \to R$ be a Mercer kernel (i.e. it is continuous, symmetry and positive semi-definite, i.e. for any given integer $l\ge 1$ and any given set $\left\{x_1\mathrm{,}{x}_2\mathrm{,}\cdots \mathrm{,}{x}_l\right\}\subset \Omega$ , the matrix ${\left(K\left(x_i\mathrm{,}{x}_j\right)\right)}_{i\mathrm{,}j=\mathrm{1,2,}\cdots \mathrm{,}l}$ is positive definite) satisfying ${\left({\displaystyle {\int }_{\Omega \times \Omega }}{\left|K\left(x\mathrm{,}y\right)\right|}^2\text{d}{\rho }_{\Omega }\left(x\right)\text{d}{\rho }_{\Omega }\left(y\right)\right)}^{\frac{1}{2}}<+\infty$ . The reproducing kernel Hilbert space $\left(H_K\mathrm{,}{\Vert \cdot \Vert }_K\right)$ associated with $K\left(x\mathrm{,}y\right)$ is a Hilbert space consisting of all the real functions defined on Ω such that:

$f\left(x\right)={\langle f\mathrm{,}{K}_x\rangle }_K\mathrm{,}\forall x\in \Omega \mathrm{,}\forall f\in H_K\mathrm{.}$

Define an operator ${\mathcal{L}}_K\left(f\mathrm{,}x\right)$ as:

${\mathcal{L}}_K\left(f\mathrm{,}x\right)={\displaystyle {\int }_{\Omega }}f\left(y\right)K_x\left(y\right)\text{d}{\rho }_{\Omega }\mathrm{,}x\in \Omega \mathrm{.}$

Then, $\mathcal{L}\mathrm{<mo>\; :\; </mo>}{L}^2\left({\rho }_{\Omega }\right)\to L^2\left({\rho }_{\Omega }\right)$ . Denoted by ${\lambda }_k$ the k-th eigenvalue associated with eigenfunction ${\phi }_k$ . Then, we have by the Mercer theorem that:

$K\left(x\mathrm{,}y\right)={\displaystyle \underset{l=0}{\overset{+\infty }{\sum }}}{\lambda }_l{\phi }_l\left(x\right){\phi }_l\left(y\right)\mathrm{,}x\mathrm{,}y\in \Omega \mathrm{,}$

where we assume the convergence on the right side is absolute (for every $x\mathrm{,}y\in \Omega$ ) and uniform on $x\mathrm{,}y\in \Omega$ . If ${\left\{{\phi }_k\right\}}_{k=0}^{+\infty }$ forms an orthonormal system in $L^2\left({\rho }_{\Omega }\right)$ , then:

$\begin{array}{c}{H}_K={\mathcal{L}}_K^{\frac{1}{2}}\left(L^2\left({\rho }_{\Omega }\right)\right)\\ =\left\{f\left(x\right)={\displaystyle \underset{l=0}{\overset{+\infty }{\sum }}}{a}_l\left(f\right){\phi }_l\left(x\right):{\Vert f\Vert }_K={\left({\displaystyle \underset{l=0}{\overset{+\infty }{\sum }}}\frac{{\left|a_l\left(f\right)\right|}^2}{{\lambda }_l}\right)}^{\frac{1}{2}}<+\infty \right\},\end{array}$

where ${\mathcal{L}}_K={\mathcal{L}}_K^{\frac{1}{2}}\circ {\mathcal{L}}_K^{\frac{1}{2}}$ .

Let $z={\left\{\left(x_k\mathrm{,}{y}_k\right)\right\}}_{k=1}^m$ be a set of observations about an unknown function $f^{\mathrm{<mo>\; *\; </mo>}}$ . Then, to obtain an good approximation of $f^{\mathrm{<mo>\; *\; </mo>}}$ , one usually borrow the kernel-regularized regression learning model:

$f_z=\arg \underset{f\in H_K}{\min }\left({\mathcal{E}}_z\left(f\right)+\lambda {\Vert f\Vert }_K^2\right),$ (6)

where ${\mathcal{E}}_z\left(f\right)=\frac{1}{m}{\displaystyle \underset{k=1}{\overset{m}{\sum }}}{\left(y_k-f\left(x_k\right)\right)}^2$ is the empirical variance. In learning theory, the convergence analysis is sum up to bound the convergence rate for the error ( [23] or [24] ):

${\Vert f_z-f^{\mathrm{<mo>\; *\; </mo>}}\Vert }_{L^2\left({\rho }_{\Omega }\right)}.$ (7)

When the observation set z has the form $z={\left\{\left(x_k,y_k\right)\right\}}_{k=1}^m\cup {\left\{x_{m+l}\right\}}_{l=1}^u$ , i.e. the nodes ${\left\{x_{m+l}\right\}}_{l=1}^u$ have no labeled observation values ${\left\{y_{m+l}\right\}}_{l=1}^u$ , we call this case the semi-supervised learning. In practical applications, most of the observations belong to semi-supervised samples since data is precious. Many mathematicians have paid their attentions to this field (see e.g. [18] [19] [21] [25] [26] [27] [28] ). The main ideas of dealing with this problem are add a term to make the use of unlabeled data, i.e. we need to modify (8) as the form of:

$f_{z,\lambda ,\gamma }=\arg \underset{f\in H_K}{\min }\left(E_z\left(f\right)+\gamma {\Omega }_{u,m}\left(f\right)+\lambda {\Vert f\Vert }_K^2\right),\gamma >0.$ (8)

For example, in [19] , one choose $\gamma {\Omega }_{u\mathrm{,}m}\left(f\right)=\frac{\gamma }{{\left(u+m\right)}^2}{\displaystyle \underset{i\mathrm{,}j=1}{\overset{l+u}{\sum }}}{\left(f\left(x_i\right)-f\left(x_j\right)\right)}^2{W}_{i\mathrm{,}j}$ , where $W_{i\mathrm{,}j}$ are edge weights in the data adjacency graph. Also, in [21] , one chooses:

$\begin{array}{l}\gamma {\Omega }_{u\mathrm{,}m}\left(f\right)=\frac{\gamma }{\left(m+u\right)\left(m+u-1\right)}{\displaystyle \underset{i\mathrm{,}j=\mathrm{1,}i\ne j}{\overset{m+u}{\sum }}}{w}_{ij}^{\left(\sigma \right)}\times {\left(f\left(x_i\right)-f\left(x_j\right)\right)}^2\mathrm{,}\\ w_{ij}^{\left(\sigma \right)}=\exp \left\{-\frac{d_K\left(x_i\mathrm{,}{x}_j\right)}{\sigma }\right\}\mathrm{,}\end{array}$

where $\sigma >0$ and

$d_K\left(x\mathrm{,}y\right)={\Vert K_x-K_y\Vert }_K=\sqrt{K\left(x\mathrm{,}x\right)+K\left(y\mathrm{,}y\right)-2K\left(x\mathrm{,}y\right)}\mathrm{.}$

These choices encourage us to choose suitable $\gamma {\Omega }_{u\mathrm{,}m}\left(f\right)$ to give good approximation solution for problem (5).

2.2. Some Results on Spherical Analysis

In this subsection, we shall define a kind of RKHS with spherical harmonics, with which define a kernel-regularized regression learning algorithm for solving problem (5) when Ω is the unit ball $B^d$ and show the learning rates.

Let $B^d=\left\{x\in R^d\mathrm{<mo>\; :\; </mo>}\Vert x\Vert \le 1\right\}$ denote the unit ball in d-dimensional Euclidean space $R^d$ with the usual inner product $\langle x\mathrm{,}y\rangle$ , and $\Vert x\Vert =\sqrt{\langle x\mathrm{,}x\rangle }$ is the usual Euclidean norm. For weight $W_{\mu }\left(x\right)={\left(1-{\Vert x\Vert }^2\right)}^{\mu }$ , $\mu >-1$ , we denote by $L_{p,\mu }\left(B^d\right)\equiv L_p\left(B^d,W_{\mu }\right)$ , $1\le p<+\infty$ , the space of measurable functions defined on $B^d$ with:

${\Vert f\Vert }_{L_{p\mathrm{,}\mu }\left(B^d\right)}={\left({\displaystyle {\int }_{B^d}}{\left|f\left(x\right)\right|}^p{W}_{\mu }\left(x\right)\text{d}x\right)}^{\frac{1}{2}}<+\infty \mathrm{,}1\le p<+\infty$

and for $p=+\infty$ , we assume that $L_{\infty \mathrm{,}\mu }$ denotes the space $C\left(B^d\right)$ of continuous functions on $B^d$ with the uniform norm.

We denote by ${\Pi }_n^d$ the space of all polynomials in d variables of degree at most n, and by ${\nu }_n^d\left(W_{\mu }\right)$ the space of all polynomials of degree n which are orthogonal to polynomials of low degree in $L_{\mathrm{2,}\mu }\left(B^d\right)$ . The ${\nu }_n^d\left(W_{\mu }\right)$ is mutually orthogonal in $L_{\mathrm{2,}\mu }\left(B^d\right)$ and (see [29] ):

$L_{\mathrm{2,}\mu }\left(B^d\right)=\underset{n=0}{\overset{\infty }{{\displaystyle \oplus }}}{\nu }_n^d\left(W_{\mu }\right)\mathrm{,}{\Pi }_n^d=\underset{k=0}{\overset{n}{{\displaystyle \oplus }}}{\nu }_k^d\left(W_{\mu }\right)\mathrm{.}$

Let $\text{d}\sigma$ denote the Lebesgue measure on $S^{d-1}=\left\{x\mathrm{<mo>\; :\; </mo>}\Vert x\Vert =1\right\}$ and denote the area of $S^{d-1}$ by ${\sigma }_d$ , ${\sigma }_d={\displaystyle {\int }_{S^{d-1}}}\text{d}\sigma =2{\pi }^{d/2}/\Gamma \left(d/2\right)$ . Let ${\mathcal{H}}_n^d$ denote the space of homogeneous harmonic polynomials of degree n, which are homogeneous polynomials of degree n satisfying equation $\Delta p={\displaystyle \underset{k=1}{\overset{d}{\sum }}}\frac{{\partial }^{2}p}{\partial {\left(x^k\right)}^2}=0$ . Also, we denote by ${\mathcal{P}}_n^d$ the set of homogeneous polynomials of degree n. It is well known that:

$a_n^d\mathrm{<mo>\; :\; </mo>}=\dim H_n^d=\left(\begin{array}{c}n+d-1\\ n-1\end{array}\right)-\left(\begin{array}{c}n+d-3\\ n-2\end{array}\right)\mathrm{.}$

Let $W^1{\left(B^d\right)}_{\mu }$ denote the set of functions whose 1-th derivatives are all in $L_{\mathrm{2,}\mu }\left(B^d\right)$ , i.e.

$W^1{\left(B^d\right)}_{\mu }=\left\{f\mathrm{<mo>\; :\; </mo>}{\Vert f\Vert }_{W^2{\left(B^d\right)}_{\mu }}={\left({\displaystyle \underset{\left|\alpha \right|\le 1}{\sum }}{\Vert {\partial }^{\alpha }f\Vert }_{L_{\mathrm{2,}\mu }\left(B^d\right)}^2\right)}^{1/2}<+\infty \right\}\mathrm{.}$

In this case, $S^{d-1}=\left\{x\mathrm{<mo>\; :\; </mo>}{\Vert x\Vert }^2={\displaystyle \underset{i=1}{\overset{d}{\sum }}}{\left(x^i\right)}^2=1\right\}$ and

$\frac{{\partial }_{x}f}{\partial \stackrel{\to }{n}}\left(x\right)={\displaystyle \underset{i=1}{\overset{d}{\sum }}}{x}^i\frac{\partial f}{\partial x^i}\left(x\right)\mathrm{,}x=\left(x^1\mathrm{,}\cdots \mathrm{,}{x}^d\right)\in S^{d-1}\mathrm{.}$ (9)

Define a subclass of $W^1{\left(B^d\right)}_{\mu }$ as:

${\mathcal{H}}_{\mu }^{\stackrel{\to }{n}}=\left\{f\in W^1{\left(B^d\right)}_{\mu }:{\Vert f\Vert }_{H_{\mu }^{\stackrel{\to }{n}}}={\left({\displaystyle {\int }_{B^d}}{\left|f\left(x\right)\right|}^2{W}_{\mu }\left(x\right)\text{d}x+{\displaystyle {\int }_{S^{d-1}}}{\left|\frac{{\partial }_{\xi }f}{\partial \stackrel{\to }{n}}\left(\xi \right)\right|}^2\text{d}\sigma \left(\xi \right)\right)}^{\frac{1}{2}}<+\infty \right\}.$

An inner product defined on ${\mathcal{H}}_{\mu }^{\stackrel{\to }{n}}$ is:

${\langle f\mathrm{,}g\rangle }_{{\mathcal{H}}_{\mu }^{\stackrel{\to }{n}}}={\displaystyle {\int }_{B^d}}f\left(x\right)g\left(x\right)W_{\mu }\left(x\right)\text{d}x+{\displaystyle {\int }_{S^{d-1}}}\frac{\partial f}{\partial \stackrel{\to }{n}}\left(\xi \right)\frac{\partial g}{\partial \stackrel{\to }{n}}\left(\xi \right)\text{d}\sigma \left(\xi \right)\mathrm{.}$

Denoted by ${\nu }_n^d\left(W_{\mu }\mathrm{,}S\right)$ the space of orthogonal polynomials with respect to ${\langle \cdot \mathrm{,}\cdot \rangle }_{{\mathcal{H}}_{\mu }^{\stackrel{\to }{n}}}$ . Then, by Theorem 3 of [30] , we know ${\nu }_n^d\left(W_{\mu }\mathrm{,}S\right)$ contains a mutually orthonormal basis $\left\{Q_{n\mathrm{,}k}\equiv Q_{n\mathrm{,}k}^d\mathrm{<mo>\; :\; </mo>}k=\mathrm{1,2,}\cdots \mathrm{,}{a}_n^d\right\}$ with respect to ${\langle \cdot \mathrm{,}\cdot \rangle }_{{\mathcal{H}}_{\mu }^{\stackrel{\to }{n}}}$ . Then, there holds the expansion:

$f\left(x\right)~{\displaystyle \underset{k=0}{\overset{\infty }{\sum }}}{\displaystyle \underset{l=1}{\overset{a_k^d}{\sum }}}{a}_{k\mathrm{,}l}\left(f\right)Q_{k\mathrm{,}l}\left(x\right)\mathrm{,}x\in B^d\mathrm{,}f\in {\mathcal{H}}_{\mu }^{\stackrel{\to }{n}}\mathrm{,}$

where $a_{k\mathrm{,}l}\left(f\right)={\langle f\mathrm{,}{Q}_{k\mathrm{,}l}\rangle }_{\mu }^S$ and by the Bessel inequality, we have:

${\left({\displaystyle \underset{k=0}{\overset{\infty }{\sum }}}{\displaystyle \underset{l=1}{\overset{d_k^d}{\sum }}}{\left|a_{k,l}\left(f\right)\right|}^2\right)}^{\frac{1}{2}}\le {\Vert f\Vert }_{{\mathcal{H}}_{\mu }^{\stackrel{\to }{n}}}<+\infty .$

Let $K^{\mu }\left(x\mathrm{,}y\right)\mathrm{<mo>\; :\; </mo>}{B}^d\times B^d\to R$ be a Mercer kernel with the form:

$K_x^{\mu }\left(y\right)=K^{\mu }\left(x,y\right):={\displaystyle \underset{k=0}{\overset{\infty }{\sum }}}{\lambda }_k{P}_k\left(x,y\right),x\in B^d,y\in B^d,$ (10)

where $P_k\left(x,y\right)={\displaystyle \underset{l=1}{\overset{a_k^d}{\sum }}}{Q}_{k,l}\left(x\right)Q_{k,l}\left(y\right)$ and ${\displaystyle \underset{k=0}{\overset{\infty }{\sum }}}{\lambda }_k{c}_k<+\infty$ with ${\lambda }_k>0$ being defined as $\underset{x\in B^d}{\sup }{P}_k\left(x,x\right)=c_k\left(k=0,1,2,\cdots \right)$ .

Define

$\begin{array}{c}{\mathcal{L}}_{K^{\mu }}\left(f\right)\left(x\right)={\mathcal{L}}_{K^{\mu }}\left(f,x\right):={\langle f,K_x(\cdot )\rangle }_{H_{\mu }^{\stackrel{\to }{n}}}\\ ={\displaystyle \underset{k=0}{\overset{\infty }{\sum }}}{\lambda }_k{\displaystyle \underset{l=1}{\overset{a_k^d}{\sum }}}{a}_{k,l}\left(f\right)Q_{k,l}\left(x\right),x\in B^d\end{array}$

and ${\mathcal{H}}_{K^{\mu }}^{\stackrel{\to }{n}}={\mathcal{L}}_{K^{\mu }}^{\frac{1}{2}}\left({\mathcal{H}}_{\mu }^{\stackrel{\to }{n}}\right)$ . Then,

${\mathcal{H}}_{K^{\mu }}^{\stackrel{\to }{n}}=\left\{{\displaystyle \underset{k=0}{\overset{\infty }{\sum }}}{\displaystyle \underset{l=1}{\overset{a_k^d}{\sum }}}{a}_{k,l}\left(f\right)Q_{k,l}\left(x\right):{\Vert f\Vert }_{K^{\mu }}={\left({\displaystyle \underset{k=0}{\overset{\infty }{\sum }}}{\displaystyle \underset{l=1}{\overset{a_k^d}{\sum }}}\frac{{\left|a_{k,l}\left(f\right)\right|}^2}{{\lambda }_k}\right)}^{\frac{1}{2}}<+\infty \right\}.$

For $f\left(x\right)={\displaystyle \underset{k=0}{\overset{\infty }{\sum }}}{\displaystyle \underset{l=1}{\overset{a_k^d}{\sum }}}{a}_{k,l}\left(f\right)Q_{k,l}\left(x\right)\in {\mathcal{H}}_{K^{\mu }}^{\stackrel{\to }{n}}$ and $g\left(x\right)={\displaystyle \underset{k=0}{\overset{\infty }{\sum }}}{\displaystyle \underset{l=1}{\overset{a_k^d}{\sum }}}{a}_{k,l}\left(g\right)Q_{k,l}\left(x\right)\in {\mathcal{H}}_{K^{\mu }}^{\stackrel{\to }{n}}$ , we define:

${\langle f,g\rangle }_{K^{\mu }}={\displaystyle \underset{k=0}{\overset{\infty }{\sum }}}{\displaystyle \underset{l=1}{\overset{a_k^d}{\sum }}}\frac{a_{k,l}\left(f\right)a_{k,l}\left(g\right)}{{\lambda }_k}.$

Then, we shall show in Proposition 2.1 that ${\mathcal{H}}_{K^{\mu }}^{\stackrel{\to }{n}}$ is an RKHS associated with kernel (10).

To give quantitatively description for the kernel $K^{\mu }$ , we give two assumptions.

Assumption A. Assume $K^{\mu }\in C^{\left(1\right)}\left(B^d\times B^d\right)$ .

Assumption B. Assume $\underset{x\in S^{d-1}}{\sup }\left|\frac{{\partial }_x{P}_k\left(x\mathrm{,}x\right)}{\partial \stackrel{\to }{n}}\right|={c^{\prime }}_k\left(k=\mathrm{0,1,2,}\cdots \right)$ and

${\displaystyle \underset{k=0}{\overset{\infty }{\sum }}}{\lambda }_k{c^{\prime }}_k<+\infty .$ (11)

Since $\left\{Q_{n\mathrm{,}k}\equiv Q_{n\mathrm{,}k}^d\mathrm{<mo>\; :\; </mo>}k=\mathrm{1,2,}\cdots \mathrm{,}{a}_n^d\right\}$ are algebraic polynomials, $c_k$ and ${c^{\prime }}_k$ must exist. The real numbers ${\lambda }_k$ satisfy (11) are also existent, for example, we can take ${\lambda }_k={\text{e}}^{-\left(c_k+{c^{\prime }}_k\right)}\left(k=\mathrm{0,1,2,}\cdots \right)$ .

If (11) holds, then by Theorem 2.4 in [31] , or Theorem 4.2 in [32] or Proposition 6.2 in [33] that:

${\partial }_x^{\alpha }\left({\partial }_y^{\beta }{K}_x^{\mu }\left(y\right)\right)=K^{\mu }\left(x,y\right):={\displaystyle \underset{k=0}{\overset{\infty }{\sum }}}{\lambda }_k{\partial }_x^{\alpha }\left({\partial }_y^{\beta }{P}_k\left(x,y\right)\right),x\in S^{d-1},y\in S^{d-1},$ (12)

and the convergence in the right-side of (12) is absolute and uniform on $S^{d-1}\times S^{d-1}$ .

Proposition 2.1. Assume above Assumptions A and B hold. Then,

1) There holds the reproducing property:

$f\left(x\right)={\langle f\mathrm{,}{K}_x^{\mu }(\cdot )\rangle }_{K^{\mu }}\mathrm{,}f\in {\mathcal{H}}_{K^{\mu }}^{\stackrel{\to }{n}}\mathrm{,}x\in B^d\mathrm{.}$ (13)

2) There holds the reproducing property for the outward normal vector operator, i.e.

$\frac{{\partial }_{x}f\left(x\right)}{\partial \stackrel{\to }{n}}={\langle f,\frac{{\partial }_x{K}_x^{\mu }(\cdot )}{\partial \stackrel{\to }{n}}\rangle }_{K^{\mu }},f\in {\mathcal{H}}_{K^{\mu }}^{\stackrel{\to }{n}},x\in S^{d-1}.$ (14)

3) Define

$k=\underset{x\in B^d}{\sup }{\Vert K_x^{\mu }(\cdot )\Vert }_{K^{\mu }}+\underset{x\in S^{d-1}}{\sup }{\Vert \frac{{\partial }_x}{\partial \stackrel{\to }{n}}{K}_x^{\mu }(\cdot )\Vert }_{K^{\mu }}\mathrm{.}$

Then, for all $x\in B^d$ and $y\in S^{d-1}$ , we have:

$\left|f\left(x\right)\right|\le k{\Vert f\Vert }_{K^{\mu }}\mathrm{,}\left|\frac{{\partial }_{x}f\left(y\right)}{\partial \stackrel{\to }{n}}\right|\le k{\Vert f\Vert }_{K^{\mu }}\mathrm{.}$ (15)

Proof. See the proofs in Section 4.

Let ${\left\{x_i\right\}}_{i=1}^{m+l}$ be observations drawn i.i.d. according to ${\rho }_{B^d}$ , $y_i=f^{*}\left(x_i\right)+{\eta }_{x_i}$ , $i=1,2,\cdots ,m$ and for a given $x_i\in B^d{\xi }_{x_i}$ is a random variable subject to a condition distribution ${\rho }_{B^d}\left(y\mathrm{<mo>\; |\; </mo>}{x}_i\right)$ satisfying $\left|\rho \left(y\mathrm{<mo>\; |\; </mo>}{x}_i\right)\right|\le B$ (B is a given constant number), $E_{\rho \left(\cdot \mathrm{<mo>\; |\; </mo>}x\right)}\left({\eta }_x\right)={\displaystyle {\int }_{\left[-B\mathrm{,}B\right]}}{\eta }_x\left(y\right)d{\rho }_{B^d}\left(y\mathrm{<mo>\; |\; </mo>}x\right)=0$ , $\sigma ={\left({\displaystyle {\int }_{B^d}}{\sigma }_x^{2}d{\rho }_{B^d}\right)}^{\frac{1}{2}}<+\infty$ and ${\sigma }_x^2=E_{{\rho }_{B^d}\left(\cdot \mathrm{<mo>\; |\; </mo>}x\right)}\left({\eta }_x^2\right)$ . Then, $z={\left\{\left(x_i,y_i\right)\right\}}_{i=1}^m$ can be regarded as observations drawn i.i.d. according to $\rho \left(x\mathrm{,}y\right)={\rho }_{B^d}\left(x\right){\rho }_{B^d}\left(y\mathrm{<mo>\; |\; </mo>}x\right)$ and ${\left\{x_i\right\}}_{i=m+1}^{m+l}$ be samples drawn i.i.d. according to ${\rho }_{S^{d-1}}$ . The correspondence of problem (5) then is:

$\{\begin{array}{ll}f\left(x_i\right)=y_{x_i},\hfill & i=1,2,\cdots ,m,\hfill \\ \frac{\partial f\left(x_i\right)}{\partial \stackrel{\to }{n}}={\displaystyle \underset{k=1}{\overset{d}{\sum }}}{x}_i^k\frac{\partial f\left(x_i\right)}{\partial x^k}=g\left(x_i\right),\hfill & i=m+1,\cdots ,m+l\hfill \end{array}$ (16)

We shall give an investigation on the numerical solutions of problem (16) with kernel-regularized approaches. A kernel learning algorithm with ${\mathcal{H}}_{K^{\mu }}^{\stackrel{\to }{n}}$ being the hypothesis space will be defined in Section 2.2. The representation theorem for it is provided and an error decomposition for its error analysis is given, from which a learning rate for Algorithm (16) is shown.

2.3. Kernel-Regularized Regression

With above notions in hand, we now give following kernel-regularized learning algorithms for giving solutions for problem (16):

$f_{z\mathrm{,}\lambda }=\arg \underset{f\in {\mathcal{H}}_{K^{\mu }}^{\stackrel{\to }{n}}}{\min }{\mathcal{E}}_z\left(f\right)+\frac{1}{l}{\displaystyle \underset{i=m}{\overset{m+l}{\sum }}}{\left(\frac{{\partial }_{x}f\left(x_i\right)}{\partial \stackrel{\to }{n}}-g\left(x_i\right)\right)}^2+\lambda {\Vert f\Vert }_{K^{\mu }}^2\mathrm{,}$ (17)

where $l~m$ , i.e. there exist $c_1>0,c_2>0$ such that $c_1\le \frac{l}{m}\le c_2$ and

${\mathcal{E}}_z\left(f\right)=\frac{1}{m}{\displaystyle \underset{i=1}{\overset{m}{\sum }}}{\left(f\left(x_i\right)-y_{x_i}\right)}^2\mathrm{.}$

Corresponding to (17), we define a model for the observations without the disturbances ${\xi }_{x_i}$ by:

$f_{\bar{X}\mathrm{,}\lambda }=\arg \underset{f\in {\mathcal{H}}_{K^{\mu }}^{\stackrel{\to }{n}}}{\min }{\mathcal{E}}_{\bar{X}}\left(f\right)+\frac{1}{l}{\displaystyle \underset{i=m}{\overset{m+l}{\sum }}}{\left(\frac{{\partial }_{x}f\left(x_i\right)}{\partial \stackrel{\to }{n}}-g\left(x_i\right)\right)}^2+\lambda {\Vert f\Vert }_{K^{\mu }}^2\mathrm{,}$ (18)

where

${\mathcal{E}}_{\bar{X}}\left(f\right)=\frac{1}{m}{\displaystyle \underset{i=1}{\overset{m}{\sum }}}{\left(f\left(x_i\right)-f^{\mathrm{<mo>\; *\; </mo>}}\left(x_i\right)\right)}^2\mathrm{.}$

The integral model for (18) is defined as:

$f_{\lambda }=\arg \underset{f\in {\mathcal{H}}_{K^{\mu }}^{\stackrel{\to }{n}}}{\min }\mathcal{E}\left(f\right)+{\displaystyle {\int }_{S^{d-1}}}{\left(\frac{{\partial }_{x}f\left(x\right)}{\partial \stackrel{\to }{n}}-g\left(x\right)\right)}^2\text{d}{\rho }_{S^{d-1}}+\lambda {\Vert f\Vert }_{K^{\mu }}^2\mathrm{,}$ (19)

where

$\mathcal{E}\left(f\right)={\displaystyle {\int }_{B^d}}{\left(f\left(x\right)-f^{\mathrm{<mo>\; *\; </mo>}}\left(x\right)\right)}^2\text{d}{\rho }_{B^d}\mathrm{.}$

Proposition 2.2. (Representation theorem). Assume above Assumptions A and B hold. Then,

1) Algorithm (17) has unique solution $f_{z\mathrm{,}\lambda }$ and there are coefficients ${\left\{a_i\right\}}_{i=1}^{m+l}$ depending upon $\lambda \mathrm{,}m\mathrm{,}l\mathrm{,}{f}_{z\mathrm{,}\lambda }\mathrm{,}z\mathrm{,}{f}^{\mathrm{<mo>\; *\; </mo>}}$ and g such that:

$f_{z,\lambda }(\cdot )={\displaystyle \underset{i=1}{\overset{m}{\sum }}}{a}_i{K}_{x_i}^{\mu }(\cdot )+{\displaystyle \underset{k=1}{\overset{l}{\sum }}}{a}_{m+k}\frac{{\partial }_x{K}_{x_{m+k}}^{\mu }(\cdot )}{\partial \stackrel{\to }{n}}.$ (20)

2) Algorithm (18) has unique solution $f_{\bar{X}\mathrm{,}\lambda }$ and there are coefficients ${\left\{b_i\right\}}_{i=1}^{m+l}$ depending upon $\lambda \mathrm{,}m\mathrm{,}l\mathrm{,}{f}_{\bar{X}\mathrm{,}\lambda }\mathrm{,}\bar{X}\mathrm{,}{f}^{\mathrm{<mo>\; *\; </mo>}}$ and g such that:

$f_{\bar{X},\lambda }(\cdot )={\displaystyle \underset{i=1}{\overset{m}{\sum }}}{b}_i{K}_{x_i}^{\mu }(\cdot )+{\displaystyle \underset{k=1}{\overset{l}{\sum }}}{b}_{m+k}\frac{{\partial }_x{K}_{x_{m+k}}^{\mu }(\cdot )}{\partial \stackrel{\to }{n}}.$ (21)

3) Algorithm (19) has unique solution $f_{\lambda }$ and there is a function $G_{\lambda \mathrm{,}{f}^{\mathrm{<mo>\; *\; </mo>}}}\left(x\right)$ depending upon $\lambda \mathrm{,}{f}^{\mathrm{<mo>\; *\; </mo>}}$ and a function $p_{\lambda \mathrm{,}g}\left(x\right)$ depending upon $\lambda$ and g such that:

$f_{\lambda }(\cdot )={\displaystyle {\int }_{B^d}}{G}_{\lambda \mathrm{,}{f}^{\mathrm{<mo>\; *\; </mo>}}}\left(x\right)K_x^{\mu }(\cdot )\text{d}{\rho }_{B^d}+{\displaystyle {\int }_{S^{d-1}}}{p}_{\lambda \mathrm{,}g}\left(x\right)\frac{{\partial }_x{K}_x^{\mu }(\cdot )}{\partial \stackrel{\to }{n}}\text{d}{\rho }_{S^{d-1}}\mathrm{.}$ (22)

Proof. See the proof in Section 4.

(20) shows that Algorithm (17) can be replaced by some coefficient regularized models and is a new topic, such kind of research can be found from literature [34] [35] .

We give the following error decomposition:

$\begin{array}{l}{\Vert f_{z,\lambda }-f^{*}\Vert }_{L^2\left({\rho }_{B^d}\right)}+{\Vert \frac{\partial f_{z,\lambda }}{\partial \stackrel{\to }{n}}-g\Vert }_{L^2\left({\rho }_{S^{d-1}}\right)}\\ \le {\Vert f_{z,\lambda }-f_{\bar{X}\mathrm{,}\lambda }\Vert }_{L^2\left({\rho }_{B^d}\right)}+{\Vert f_{\bar{X}\mathrm{,}\lambda }-f^{*}\Vert }_{L^2\left({\rho }_{B^d}\right)}\\ +{\Vert \frac{\partial f_{z,\lambda }}{\partial \stackrel{\to }{n}}-\frac{\partial f_{\bar{X},\lambda }}{\partial \stackrel{\to }{n}}\Vert }_{L^2\left({\rho }_{S^{d-1}}\right)}+{\Vert \frac{\partial f_{\bar{X},\lambda }}{\partial \stackrel{\to }{n}}-g\Vert }_{L^2\left({\rho }_{S^{d-1}}\right)}\\ \le {\Vert f_{z,\lambda }-f_{\bar{X}\mathrm{,}\lambda }\Vert }_{L^2\left({\rho }_{B^d}\right)}+{\Vert f_{\bar{X}\mathrm{,}\lambda }-f_{\lambda }\Vert }_{L^2\left({\rho }_{B^d}\right)}+{\Vert f_{\lambda }-f^{*}\Vert }_{L^2\left({\rho }_{B^d}\right)}\\ +{\Vert \frac{\partial f_{z,\lambda }}{\partial \stackrel{\to }{n}}-\frac{\partial f_{\bar{X}\mathrm{,}\lambda }}{\partial \stackrel{\to }{n}}\Vert }_{L^2\left({\rho }_{S^{d-1}}\right)}+{\Vert \frac{\partial f_{\bar{X},\lambda }}{\partial \stackrel{\to }{n}}-\frac{\partial f_{\lambda }}{\partial \stackrel{\to }{n}}\Vert }_{L^2\left({\rho }_{S^{d-1}}\right)}+{\Vert \frac{\partial f_{\lambda }}{\partial \stackrel{\to }{n}}-g\Vert }_{L^2\left({\rho }_{S^{d-1}}\right)}\end{array}$

$\begin{array}{l}\le \left({\Vert f_{z,\lambda }-f_{\bar{X}\mathrm{,}\lambda }\Vert }_{L^2\left({\rho }_{B^d}\right)}+{\Vert \frac{\partial f_{z,\lambda }}{\partial \stackrel{\to }{n}}-\frac{\partial f_{\bar{X},\lambda }}{\partial \stackrel{\to }{n}}\Vert }_{L^2\left({\rho }_{S^{d-1}}\right)}\right)\\ +\left({\Vert f_{\bar{X}\mathrm{,}\lambda }-f_{\lambda }\Vert }_{L^2\left({\rho }_{B^d}\right)}+{\Vert \frac{\partial f_{\bar{X},\lambda }}{\partial \stackrel{\to }{n}}-\frac{\partial f_{\lambda }}{\partial \stackrel{\to }{n}}\Vert }_{L^2\left({\rho }_{S^{d-1}}\right)}\right)+\sqrt{2}\sqrt{K\left(f^{*},g,\lambda \right)}\\ \le k\left({\Vert f_{z,\lambda }-f_{\bar{X}\mathrm{,}\lambda }\Vert }_{K^{\mu }}+{\Vert f_{\bar{X}\mathrm{,}\lambda }-f_{\lambda }\Vert }_{K^{\mu }}\right)+\sqrt{2}\sqrt{K\left(f^{*},g,\lambda \right)},\end{array}$ (23)

where in the last derivation, we have used (15) and

$K\left(f^{\mathrm{<mo>\; *\; </mo>}}\mathrm{,}g\mathrm{,}\lambda \right)=\underset{f\in {\mathcal{H}}_{K^{\mu }}^{\stackrel{\to }{n}}}{\inf }\left({\Vert f-f^{\mathrm{<mo>\; *\; </mo>}}\Vert }_{L^2\left({\rho }_{B^d}\right)}^2+{\displaystyle {\int }_{S^{d-1}}}{\left(\frac{\partial f\left(x\right)}{\partial \stackrel{\to }{n}}-g\left(x\right)\right)}^2\text{d}{\rho }_{S^{d-1}}+\lambda {\Vert f\Vert }_{K^{\mu }}^2\right)\mathrm{,}$

which controls the approximation errors. Then, to bound the error:

$\mathcal{E}\left(f_{z\mathrm{,}\lambda }\mathrm{,}{f}^{\mathrm{<mo>\; *\; </mo>}}\mathrm{,}g\right)={\Vert f_{z\mathrm{,}\lambda }-f^{\mathrm{<mo>\; *\; </mo>}}\Vert }_{L^2\left({\rho }_{B^d}\right)}+{\Vert \frac{\partial f_{z\mathrm{,}\lambda }}{\partial \stackrel{\to }{n}}-g\Vert }_{L^2\left({\rho }_{S^{d-1}}\right)}\mathrm{,}$

we need only to bound upper bounds for the sample errors ${\Vert f_{z\mathrm{,}\lambda }-f_{\bar{X}\mathrm{,}\lambda }\Vert }_{K^{\mu }}$ and ${\Vert f_{\bar{X}\mathrm{,}\lambda }-f_{\rho \mathrm{,}\lambda }\Vert }_{K^{\mu }}$ respectively.

2.4. Learning Rates

Theorem 2.1. Let the above Assumptions A and B hold and let $f_{z\mathrm{,}\lambda }$ be the solution of (17) and let $f^{\mathrm{<mo>\; *\; </mo>}}\in C\left(B^d\right)$ and $g\in L^2\left({\rho }_{S^{d-1}}\right)$ . Then, for any $\delta \in \left(\mathrm{0,1}\right)$ , with confidence $1-\delta$ , holds:

$\begin{array}{l}{\Vert f_{z,\lambda }-f^{*}\Vert }_{L^2\left({\rho }_{B^d}\right)}+{\Vert \frac{\partial f_{z,\lambda }}{\partial \stackrel{\to }{n}}-g\Vert }_{L^2\left({\rho }_{S^{d-1}}\right)}\\ =O(\frac{\sigma }{\lambda \sqrt{m\delta }}+\left(\frac{\sqrt{K\left(f^{*},g,\lambda \right)}}{{\lambda }^{\frac{3}{2}}\sqrt{m}}+\frac{{\Vert f^{*}\Vert }_{C\left(B^d\right)}}{\lambda \sqrt{m}}\right)\log \frac{4}{\delta }\\ +\frac{\sqrt{K\left(f^{*},g,\lambda \right)}}{\lambda \sqrt{l\delta }}+\sqrt{K\left(f^{*},g,\lambda \right)}).\end{array}$ (24)

If $g=\frac{\partial f^{\mathrm{<mo>\; *\; </mo>}}\left(x\right)}{\partial \stackrel{\to }{n}}$ for $x\in S^{d-1}$ , then

$\mathcal{E}\left(f_{z\mathrm{,}\lambda }\mathrm{,}{f}^{\mathrm{<mo>\; *\; </mo>}}\mathrm{,}g\right)={\Vert f_{z\mathrm{,}\lambda }-f^{\mathrm{<mo>\; *\; </mo>}}\Vert }_{H^{\stackrel{\to }{n}}\left({\rho }_{B^d}\right)}$

and in this case,

$K\left(f^{*},g,\lambda \right)=K\left(f^{*},\lambda \right)=\underset{f\in {\mathcal{H}}_{K^{\mu }}^{\stackrel{\to }{n}}}{\inf }\left({\Vert f-f^{*}\Vert }_{H^{\stackrel{\to }{n}}\left({\rho }_{B^d}\right)}^2+\lambda {\Vert f\Vert }_{K^{\mu }}^2\right),\lambda >0,$

where the norm ${\Vert \cdot \Vert }_{H^{\stackrel{\to }{n}}\left({\rho }_{B^d}\right)}$ is defined in Section 2.2. The decay for $K\left(f^{\mathrm{<mo>\; *\; </mo>}}\mathrm{,}\lambda \right)$ has recently been discussed in [36] .

We then have the following Corollary 2.1.

Corollary 2.1. Let Assumptions A and B hold and let $f_{z\mathrm{,}\lambda }$ be the solution of (17) and $f^{\mathrm{<mo>\; *\; </mo>}}\in C\left(B^d\right)$ . If $g\left(x\right)=\frac{\partial f^{\mathrm{<mo>\; *\; </mo>}}\left(x\right)}{\partial \stackrel{\to }{n}}$ for $x\in S^{d-1}$ . Then, for any $\delta \in \left(\mathrm{0,1}\right)$ , with confidence $1-\delta$ , holds:

$\begin{array}{l}{\Vert f_{z,\lambda }-f^{*}\Vert }_{H^{\stackrel{\to }{n}}\left({\rho }_{B^d}\right)}\\ =O\left(\frac{\sigma }{\lambda \sqrt{m\delta }}+\left(\frac{\sqrt{K\left(f^{*},\lambda \right)}}{{\lambda }^{\frac{3}{2}}\sqrt{m}}+\frac{{\Vert f^{*}\Vert }_{C\left(B^d\right)}}{\lambda \sqrt{m}}\right)\log \frac{4}{\delta }+\frac{\sqrt{K\left(f^{*},g,\lambda \right)}}{\lambda \sqrt{l\delta }}+\sqrt{K\left(f^{*},\lambda \right)}\right).\end{array}$ (25)

By (25), we know if $\lambda =\lambda \left(m\right){\downarrow }_{0^{+}}\left(m\to +\infty \right)$ is chosen in such a way that $\underset{\lambda \to 0^{+}}{lim}K\left(f^{\mathrm{<mo>\; *\; </mo>}}\mathrm{,}\lambda \right)=0$ , ${\lambda }^{\frac{3}{2}}\sqrt{m}\to +\infty$ when $m\to +\infty$ , then, with confidence $1-\delta$ , holds (since $m~l$ ):

$\underset{m\to +\infty }{lim}{\Vert f_{z\mathrm{,}\lambda }-f^{\mathrm{<mo>\; *\; </mo>}}\Vert }_{H^{\stackrel{\to }{n}}\left({\rho }_{B^d}\right)}=0.$ (26)

2.5. Further Discussions

We now give some explanation on the results and the assumptions.

1) There are three reasons encouraging us to choose $D=B^d$ and $\partial D=S^{d-1}$ to show the kernel-regularized regression model for solving the Neumann boundary problem (5).

i) When $D=B^d$ and $\partial D=S^{d-1}$ , we can easily give explicit representation for outward normal derivatives (see (9)). Therefore, we can extend the method used in this paper to the domains whose outward normal derivatives may be computed easily.

ii) By the statement in Section 2.1, we know a tool for us to construct a reproducing kernel Hilbert space is the orthonormal basis. We notice that function orthogonal basis theory has been developed not only on the unit ball $B^d$ (see [37] ) and the unit sphere $S^{d-1}$ (see [29] ), but also on the some Sobolev space associated with both $B^d$ and $S^{d-1}$ (see e.g. [30] ). These facts also encourage us to choose $D=B^d$ and $\partial D=S^{d-1}$ for problem (5).

iii) In learning theory, the learning rate estimate for the kernel-regularized learning algorithm are sum up to bound the error bounds, which belong to the scope of approximation theory. One can make use of the rich spherical approximation theory skills to bound the learning rates (see [24] ).

2) In the present paper, we give two Assumptions A and B. They are reasonable.

Since $p_k\left(x\mathrm{,}y\right)$ is a bivariate polynomial on $B^d\times B^d$ for a given k, $\frac{{\partial }_x{p}_k\left(x\mathrm{,}x\right)}{\partial \stackrel{\to }{n}}$ is still a polynomial whose sup can be attained on $S^{d-1}$ . The Assumption B is reasonable. By the same way, we know $d_k=\underset{x,y\in B^d}{\sup }{\partial }^{\alpha }{p}_k\left(x,y\right)$ can be attained. If we choose ${\lambda }_l={\text{e}}^{-k^{\beta }}\left(\beta >0\right)$ , then, we can have $K^{\mu }\in C^{\left(1\right)}\left(B^d\times B^d\right)$ . So, the Assumption A is reasonable as well.

3) The convergence in the present paper are uniform convergence (see (26)), which admits the reliability for the proposed method.

3. Lemmas

To give the feature description of the optimal solutions of Algorithms (17)-(19), we need the concept of Gâteaux derivative.

Let $\left(\mathcal{H}\mathrm{,}{\Vert \cdot \Vert }_{\mathcal{H}}\right)$ be a Hilbert space, $F\left(f\right)\mathrm{<mo>\; :\; </mo>}\mathcal{H}\to R\cup \left\{\mp \infty \right\}$ be a real function. We say F is Gâteaux differentiable at $f\in \mathcal{H}$ if there is a $\xi \in \mathcal{H}$ such that for any $g\in \mathcal{H}$ there holds:

$\underset{t\to 0}{lim}\frac{F\left(f+tg\right)-F\left(f\right)}{t}={\langle g\mathrm{,}\xi \rangle }_H$

and write ${\nabla }_{f}F\left(f\right)=\xi$ as the Gâteaux derivative of $F\left(f\right)$ at f.

To prove Theorem 2.1, we need some lemmas.

Lemma 3.1. There hold the following equations:

1) $f_{z\mathrm{,}\lambda }$ satisfies equation:

$\begin{array}{l}\frac{1}{m}{\displaystyle \underset{i=1}{\overset{m}{\sum }}}\left(f_{z,\lambda }\left(x_i\right)-y_{x_i}\right)K_{x_i}^{\mu }(\cdot )+\frac{1}{l}{\displaystyle \underset{k=1}{\overset{l}{\sum }}}\left(\frac{\partial f_{z,\lambda }\left(x_{m+k}\right)}{\partial \stackrel{\to }{n}}-g\left(x_{m+k}\right)\right)\frac{\partial K_{x_{m+k}}^{\mu }(\cdot )}{\partial \stackrel{\to }{n}}\\ +\lambda f_{z,\lambda }(\cdot )=0.\end{array}$ (27)

2) $f_{\bar{X}\mathrm{,}\lambda }$ satisfies equation:

$\begin{array}{l}\frac{1}{m}{\displaystyle \underset{i=1}{\overset{m}{\sum }}}\left(f_{\bar{X}\mathrm{,}\lambda }\left(x_i\right)-f^{*}\right)\left(x_i\right)K_{x_i}^{\mu }(\cdot )+\frac{1}{l}{\displaystyle \underset{k=1}{\overset{l}{\sum }}}\left(\frac{\partial f_{\bar{X}\mathrm{,}\lambda }\left(x_{m+k}\right)}{\partial \stackrel{\to }{n}}-g\left(x_{m+k}\right)\right)\frac{\partial K_{x_{m+k}}^{\mu }(\cdot )}{\partial \stackrel{\to }{n}}\\ +\lambda f_{\bar{X}\mathrm{,}\lambda }(\cdot )=0.\end{array}$ (28)

3) $f_{\lambda }$ satisfies equation:

$\begin{array}{l}{\displaystyle {\int }_{B^d}}\left(f_{\lambda }\left(x\right)-f^{*}\left(x\right)\right)K_x^{\mu }(\cdot )\text{d}{\rho }_{B^d}+{\displaystyle {\int }_{S^{d-1}}}\left(\frac{\partial f_{\lambda }\left(x\right)}{\partial \stackrel{\to }{n}}-g\left(x\right)\right)\frac{\partial K_x^{\mu }(\cdot )}{\partial \stackrel{\to }{n}}\text{d}{\rho }_{S^{d-1}}\\ +\lambda f_{\lambda }(\cdot )=0.\end{array}$ (29)

Proof. Proof of 1). Define ${\Omega }_z\left(f\right)$ as:

${\Omega }_z\left(f\right)={\mathcal{E}}_z\left(f\right)+\frac{1}{l}{\displaystyle \underset{k=1}{\overset{l}{\sum }}}{\left(\frac{\partial f\left(x_{k+m}\right)}{\partial \stackrel{\to }{n}}-g\left(x_{m+k}\right)\right)}^2+\lambda {\Vert f\Vert }_{K^{\mu }}^2\mathrm{,}f\in {\mathcal{H}}_{K^{\mu }}^{\stackrel{\to }{n}}\mathrm{.}$

Then,

$\begin{array}{l}\underset{t\to 0^{+}}{lim}\frac{{\Omega }_z\left(f+th\right)-{\Omega }_z\left(f\right)}{t}\\ =\underset{t\to 0^{+}}{lim}\frac{{\mathcal{E}}_z\left(f+th\right)-{\mathcal{E}}_z\left(f\right)}{t}+2\lambda {\langle h\mathrm{,}f\rangle }_{K^{\mu }}\mathrm{,}\\ +\frac{1}{l}{\displaystyle \underset{k=1}{\overset{l}{\sum }}}\underset{t\to 0^{+}}{lim}\frac{{\left(\frac{\partial f\left(x_{k+m}\right)}{\partial \stackrel{\to }{n}}+t\frac{\partial h\left(x_{k+m}\right)}{\partial \stackrel{\to }{n}}-g\left(x_{k+m}\right)\right)}^2-{\left(\frac{\partial f\left(x_{k+m}\right)}{\partial \stackrel{\to }{n}}-g\left(x_{k+m}\right)\right)}^2}{t}\mathrm{,}\end{array}$

where

$\begin{array}{c}\underset{t\to 0^{+}}{lim}\frac{{\mathcal{E}}_z\left(f+th\right)-{\mathcal{E}}_z\left(f\right)}{t}=\frac{1}{m}{\displaystyle \underset{i=1}{\overset{m}{\sum }}}\underset{t\to 0^{+}}{lim}\frac{{\left(f\left(x_i\right)+th\left(x_i\right)-y_{x_i}\right)}^2-{\left(f\left(x_i\right)-y_{x_i}\right)}^2}{t}\\ =\frac{2}{m}{\displaystyle \underset{i=1}{\overset{m}{\sum }}}\left(f\left(x_i\right)-y_{x_i}\right)h\left(x_i\right)\\ =\frac{2}{m}{\displaystyle \underset{i=1}{\overset{m}{\sum }}}\left(f\left(x_i\right)-y_{x_i}\right){\langle h\mathrm{,}{K}_{x_i}^{\mu }(\cdot )\rangle }_{K^{\mu }}\\ ={\langle h\mathrm{,}\frac{2}{m}{\displaystyle \underset{i=1}{\overset{m}{\sum }}}\left(f\left(x_i\right)-y_{x_i}\right)K_{x_i}^{\mu }(\cdot )\rangle }_{K^{\mu }}\end{array}$

and

$\begin{array}{l}\frac{1}{l}{\displaystyle \underset{k=1}{\overset{l}{\sum }}}\underset{t\to 0^{+}}{lim}\frac{{\left(\frac{\partial f\left(x_{k+m}\right)}{\partial \stackrel{\to }{n}}+t\frac{\partial h\left(x_{k+m}\right)}{\partial \stackrel{\to }{n}}-g\left(x_{k+m}\right)\right)}^2-{\left(\frac{\partial f\left(x_{k+m}\right)}{\partial \stackrel{\to }{n}}-g\left(x_{k+m}\right)\right)}^2}{t}\\ ={\langle h,\frac{2}{l}{\displaystyle \underset{k=1}{\overset{l}{\sum }}}\left(\frac{\partial f\left(x_{k+m}\right)}{\partial \stackrel{\to }{n}}-g\left(x_{k+m}\right)\right)\frac{\partial K_{x_{k+m}}^{\mu }(\cdot )}{\partial \stackrel{\to }{n}}\rangle }_{K^{\mu }}.\end{array}$ (30)

It follows

$\begin{array}{l}\underset{t\to 0^{+}}{lim}\frac{{\Omega }_z\left(f+th\right)-{\Omega }_z\left(f\right)}{t}\\ ={\langle h,\frac{2}{m}{\displaystyle \underset{i=1}{\overset{m}{\sum }}}\left(f\left(x_i\right)-y_{x_i}\right)K_{x_i}^{\mu }(\cdot )+\frac{2}{l}{\displaystyle \underset{k=1}{\overset{l}{\sum }}}\left(\frac{\partial f\left(x_{k+m}\right)}{\partial \stackrel{\to }{n}}-g\left(x_{k+m}\right)\right)\frac{\partial K_{x_{k+m}}^{\mu }(\cdot )}{\partial \stackrel{\to }{n}}+2\lambda f\rangle }_{K^{\mu }}.\end{array}$

By the definition of Gâteaux derivative, we have:

$\begin{array}{c}{\nabla }_f{\Omega }_z\left(f\right)=\frac{2}{m}{\displaystyle \underset{i=1}{\overset{m}{\sum }}}\left(f\left(x_i\right)-y_{x_i}\right)K_{x_i}^{\mu }(\cdot )\\ +\frac{2}{l}{\displaystyle \underset{k=1}{\overset{l}{\sum }}}\left(\frac{\partial f\left(x_{k+m}\right)}{\partial \stackrel{\to }{n}}-g\left(x_{k+m}\right)\right)\frac{\partial K_{x_{k+m}}^{\mu }(\cdot )}{\partial \stackrel{\to }{n}}+2\lambda f(\cdot ).\end{array}$

By Fermat’s rule (see 1) in Proposition A1) and the definition of $f_{z\mathrm{,}\lambda }$ , we have ${{\nabla }_f{\Omega }_z\left(f\right)|}_{f=f_{z,\lambda }}=0$ , i.e. (27) holds.

Lemma 3.2. There hold the inequality:

${\Vert f_{z\mathrm{,}\lambda }-f_{\bar{X}\mathrm{,}\lambda }\Vert }_{K^{\mu }}\le \frac{2}{\lambda m}{\Vert {\displaystyle \underset{i=1}{\overset{m}{\sum }}}{\eta }_{x_i}\left(y\right)K_{x_i}^{\mu }(\cdot )\Vert }_{K^{\mu }}$ (31)

and the inequality:

$\begin{array}{l}{\Vert f_{\bar{X}\mathrm{,}\lambda }-f_{\rho \mathrm{,}\lambda }\Vert }_{K^{\mu }}\\ \le \frac{2}{\lambda }({\Vert {\displaystyle {\int }_{B^d}}\left(f_{\lambda }\left(x\right)-f^{\mathrm{<mo>\; *\; </mo>}}\left(x\right)\right)K_x^{\mu }(\cdot )\text{d}{\rho }_{B^d}-\frac{1}{m}{\displaystyle \underset{i\mathrm{<mo>\; =\; </mo>1}}{\overset{m}{\sum }}}\left(f_{\lambda }\left(x_i\right)-f^{\mathrm{<mo>\; *\; </mo>}}\left(x_i\right)\right)K_{x_i}^{\mu }(\cdot )\Vert }_{K^{\mu }}\\ +{\Vert {\displaystyle {\int }_{S^{d-1}}}\left(\frac{\partial f\left(x\right)}{\partial \stackrel{\to }{n}}-g\left(x\right)\right)K_x^{\mu }(\cdot )\text{d}{\rho }_{S^{d-1}}-\frac{1}{l}{\displaystyle \underset{k\mathrm{<mo>\; =\; </mo>1}}{\overset{l}{\sum }}}\left(\frac{\partial f\left(x_{k+m}\right)}{\partial \stackrel{\to }{n}}-g\left(x_{m+k}\right)\right)K_{x_{m+k}}^{\mu }(\cdot )\Vert }_{K^{\mu }})\mathrm{.}\end{array}$ (32)

Proof. The definition of $f_{z\mathrm{,}\lambda }$ and the inequality (43) give:

$\begin{array}{l}0\ge {\Omega }_z\left(f_{z,\lambda }\right)-{\Omega }_z\left(f_{\bar{X}\mathrm{,}\lambda }\right)\\ \ge \frac{2}{m}{\displaystyle \underset{i=1}{\overset{m}{\sum }}}\left(f_{\bar{X}\mathrm{,}\lambda }\left(x_i\right)-y_{x_i}\right)\left(f_{z,\lambda }\left(x_i\right)-f_{\bar{X}\mathrm{,}\lambda }\left(x_i\right)\right)\\ +\frac{2}{l}{\displaystyle \underset{k=1}{\overset{l}{\sum }}}\left(\frac{\partial f_{\bar{X}\mathrm{,}\lambda }\left(x_{m+k}\right)}{\partial \stackrel{\to }{n}}-g\left(x_{m+k}\right)\right)\left(\frac{\partial f_{z,\lambda }\left(x_{m+k}\right)}{\partial \stackrel{\to }{n}}-\frac{\partial f_{\bar{X}\mathrm{,}\lambda }\left(x_{m+k}\right)}{\stackrel{\to }{n}}\right)\\ +\lambda \left({\Vert f_{z,\lambda }\Vert }_{K^{\mu }}^2-{\Vert f_{\bar{X}\mathrm{,}\lambda }\Vert }_{K^{\mu }}^2\right)\end{array}$

$\begin{array}{c}\ge \langle f_{z,\lambda }-f_{\bar{X}\mathrm{,}\lambda },\frac{2}{m}{\displaystyle \underset{i=1}{\overset{m}{\sum }}}\left(f_{\bar{X}\mathrm{,}\lambda }\left(x_i\right)-y_{x_i}\right)K_{x_i}^{\mu }(\cdot )\\ +{\frac{2}{l}{\displaystyle \underset{k=1}{\overset{l}{\sum }}}\left(\frac{\partial f_{\bar{X}\mathrm{,}\lambda }\left(x_{m+k}\right)}{\partial \stackrel{\to }{n}}-g\left(x_{m+k}\right)\right)\frac{\partial K_{x_{m+k}}^{\mu }(\cdot )}{\partial \stackrel{\to }{n}}\rangle }_{K^{\mu }}\\ +{\langle f_{z,\lambda }-f_{\bar{X}\mathrm{,}\lambda },2\lambda f_{\bar{X}\mathrm{,}\lambda }\rangle }_{K^{\mu }}+\lambda {\Vert f_{z,\lambda }-f_{\bar{X}\mathrm{,}\lambda }\Vert }_{K^{\mu }}^2,\end{array}$

where we have used (44). Since (28) and $y_{x_i}=f^{\mathrm{<mo>\; *\; </mo>}}\left(x_i\right)+{\eta }_{x_i}$ , we have:

$\begin{array}{c}0\ge {\Omega }_z\left(f_{z,\lambda }\right)-{\Omega }_z\left(f_{\bar{X},\lambda }\right)\\ \ge {\langle f_{z,\lambda }-f_{\bar{X},\lambda },\frac{2}{m}{\displaystyle \underset{i=1}{\overset{m}{\sum }}}{\eta }_{x_i}\left(y\right)K_{x_i}^{\mu }(\cdot )\rangle }_{K^{\mu }}+\lambda {\Vert f_{z,\lambda }-f_{\bar{X},\lambda }\Vert }_{K^{\mu }}^2.\end{array}$

It follows:

$\begin{array}{c}\lambda {\Vert f_{z,\lambda }-f_{\bar{X},\lambda }\Vert }_{K^{\mu }}^2\le {\langle f_{\bar{X},\lambda }-f_{z,\lambda },\frac{2}{m}{\displaystyle \underset{i=1}{\overset{m}{\sum }}}{\eta }_{x_i}\left(y\right)K_{x_i}^{\mu }(\cdot )\rangle }_{K^{\mu }}\\ \le {\Vert \frac{2}{m}{\displaystyle \underset{i=1}{\overset{m}{\sum }}}{\eta }_{x_i}\left(y\right)K_{x_i}^{\mu }(\cdot )\Vert }_{K^{\mu }}\times {\Vert f_{\bar{X},\lambda }-f_{z,\lambda }\Vert }_{K^{\mu }}.\end{array}$

(31) thus holds.

Proof of (32). Define ${\Omega }_{\bar{X}}\left(f\right)$ as:

${\Omega }_{\bar{X}}\left(f\right)={\mathcal{E}}_{\bar{X}}\left(f\right)+\frac{1}{l}{\displaystyle \underset{k=1}{\overset{l}{\sum }}}{\left(\frac{\partial f\left(x_{k+m}\right)}{\partial \stackrel{\to }{n}}-g\left(x_{m+k}\right)\right)}^2+\lambda {\Vert f\Vert }_{K^{\mu }}^2\mathrm{,}f\in {\mathcal{H}}_{K^{\mu }}^{\stackrel{\to }{n}}\mathrm{.}$

Then, by the definition of $f_{\bar{X}\mathrm{,}\lambda }$ and inequalities (43) and (44), we have:

$\begin{array}{c}0\ge {\Omega }_{\bar{X}}\left(f_{\bar{X},\lambda }\right)-{\Omega }_{\bar{X}}\left(f_{\lambda }\right)\\ \ge {\langle f_{\bar{X},\lambda }-f_{\lambda },\frac{2}{m}{\displaystyle \underset{i=1}{\overset{m}{\sum }}}\left(f_{\lambda }\left(x_i\right)-f^{*}\left(x_i\right)\right)K_{x_i}^{\mu }(\cdot )\rangle }_{K^{\mu }}\\ +{\langle f_{\bar{X},\lambda }-f_{\lambda },\frac{2}{l}{\displaystyle \underset{k=1}{\overset{l}{\sum }}}\left(\frac{\partial f_{\lambda }\left(x_{m+k}\right)}{\partial \stackrel{\to }{n}}-g\left(x_{m+k}\right)\right)\frac{\partial K_{x_{m+k}}^{\mu }(\cdot )}{\partial \stackrel{\to }{n}}\rangle }_{K^{\mu }}\\ +{\langle f_{\bar{X},\lambda }-f_{\lambda },2\lambda f_{\lambda }\rangle }_{K^{\mu }}+\lambda {\Vert f_{\bar{X}\mathrm{,}\lambda }-f_{\lambda }\Vert }_{K^{\mu }}^2.\end{array}$

By (29), we have:

$\begin{array}{l}0\ge \langle f_{\bar{X}\mathrm{,}\lambda }-f_{\lambda },{\displaystyle {\int }_{B^d}}\left(f_{\lambda }\left(x\right)-f^{*}\left(x\right)\right)K_x^{\mu }(\cdot )\text{d}{\rho }_{B^d}\begin{array}{c}\\ \end{array}\\ -\frac{1}{m}{\displaystyle \underset{i=1}{\overset{m}{\sum }}}\left(f_{\lambda }\left(x_i\right)-f^{*}\left(x_i\right)\right)K_{x_i}^{\mu }(\cdot )+{\displaystyle {\int }_{S^{d-1}}}\left(\frac{\partial f_{\lambda }\left(x\right)}{\partial \stackrel{\to }{n}}-g\left(x\right)\right)\frac{\partial K_x^{\mu }(\cdot )}{\partial \stackrel{\to }{n}}\text{d}{\rho }_{S^{d-1}}\\ -{\frac{1}{l}{\displaystyle \underset{k=1}{\overset{l}{\sum }}}\left(\frac{\partial f_{\lambda }\left(x_{m+k}\right)}{\partial \stackrel{\to }{n}}-g\left(x_{m+k}\right)\right)\frac{\partial K_{x_{m+k}}^{\mu }(\cdot )}{\partial \stackrel{\to }{n}}\rangle }_{K^{\mu }}+\lambda {\Vert f_{\bar{X}\mathrm{,}\lambda }-f_{\lambda }\Vert }_{K^{\mu }}^2.\end{array}$

It follows:

$\begin{array}{l}\lambda {\Vert f_{\bar{X}\mathrm{,}\lambda }-f_{\lambda }\Vert }_{K^{\mu }}^2\\ \le \langle f_{\bar{X}\mathrm{,}\lambda }-f_{\lambda },\frac{1}{m}{\displaystyle \underset{i=1}{\overset{m}{\sum }}}\left(f_{\lambda }\left(x_i\right)-f^{*}\left(x_i\right)\right)K_{x_i}^{\mu }(\cdot )-{\displaystyle {\int }_{B^d}}\left(f_{\lambda }\left(x\right)-f^{*}\left(x\right)\right)K_x^{\mu }(\cdot )\text{d}{\rho }_{B^d}\\ -{\displaystyle {\int }_{S^{d-1}}}\left(\frac{\partial f_{\lambda }\left(x\right)}{\partial \stackrel{\to }{n}}-g\left(x\right)\right)\frac{\partial K_x^{\mu }(\cdot )}{\partial \stackrel{\to }{n}}\text{d}{\rho }_{S^{d-1}}\\ +{\frac{1}{l}{\displaystyle \underset{k=1}{\overset{l}{\sum }}}\left(\frac{\partial f_{\lambda }\left(x_{m+k}\right)}{\partial \stackrel{\to }{n}}-g\left(x_{m+k}\right)\right)\frac{\partial K_{x_{m+k}}^{\mu }(\cdot )}{\partial \stackrel{\to }{n}}\rangle }_{K^{\mu }}\end{array}$

$\begin{array}{l}\le ({\Vert \frac{1}{m}{\displaystyle \underset{i=1}{\overset{m}{\sum }}}\left(f_{\lambda }\left(x_i\right)-f^{*}\left(x_i\right)\right)K_{x_i}^{\mu }(\cdot )-{\displaystyle {\int }_{B^d}}\left(f_{\lambda }\left(x\right)-f^{*}\left(x\right)\right)K_x^{\mu }(\cdot )\text{d}{\rho }_{B^d}\Vert }_{K^{\mu }}\\ \times {\Vert f_{\bar{X}\mathrm{,}\lambda }-f_{\lambda }\Vert }_{K^{\mu }}+\Vert {\displaystyle {\int }_{S^{d-1}}}\left(\frac{\partial f_{\lambda }\left(x\right)}{\partial \stackrel{\to }{n}}-g\left(x\right)\right)\frac{\partial K_x^{\mu }(\cdot )}{\partial \stackrel{\to }{n}}\text{d}{\rho }_{S^{d-1}}\\ -{\frac{1}{l}{\displaystyle \underset{k=1}{\overset{l}{\sum }}}\left(\frac{\partial f_{\lambda }\left(x_{m+k}\right)}{\partial \stackrel{\to }{n}}-g\left(x_{m+k}\right)\right)\frac{\partial K_{x_{m+k}}^{\mu }(\cdot )}{\partial \stackrel{\to }{n}}\Vert }_{K^{\mu }}).\end{array}$

Above inequality gives (32).

Lemma 3.3. There hold following inequalities.

1) For any $\delta \in \left(\mathrm{0,1}\right)$ , with confidence $1-\delta$ , holds:

${\Vert f_{z\mathrm{,}\lambda }-f_{\bar{X}\mathrm{,}\lambda }\Vert }_{K^{\mu }}\le \frac{2k\sigma }{\lambda \sqrt{m\delta }}\mathrm{.}$ (33)

2) For any $\delta \in \left(\mathrm{0,1}\right)$ , with confidence $1-\delta$ , holds:

${\Vert f_{\bar{X}\mathrm{,}\lambda }-f_{\lambda }\Vert }_{K^{\mu }}=\left(\frac{2k^2\sqrt{K\left(f^{\mathrm{<mo>\; *\; </mo>}}\mathrm{,}g\mathrm{,}\lambda \right)}}{\lambda \sqrt{m\lambda }}+\frac{2k{\Vert f^{\mathrm{<mo>\; *\; </mo>}}\Vert }_{C\left(B^d\right)}}{\lambda \sqrt{m}}\right)\log \frac{2}{\delta }+\frac{k\sqrt{K\left(f^{\mathrm{<mo>\; *\; </mo>}}\mathrm{,}g\mathrm{,}\lambda \right)}}{\sqrt{l\delta }}\mathrm{.}$ (34)

Proof. Proof of (33). The definition of ${\Vert \cdot \Vert }_{K^{\mu }\left(B^d\right)}$ and (14) give:

$\begin{array}{c}{\Vert \frac{1}{m}{\displaystyle \underset{i=1}{\overset{m}{\sum }}}{\eta }_{x_i}\left(y\right)K_{x_i}^{\mu }(\cdot )\Vert }_{K^{\mu }}^2={\langle \frac{1}{m}{\displaystyle \underset{i=1}{\overset{m}{\sum }}}{\eta }_{x_i}\left(y\right)K_{x_i}^{\mu }(\cdot ),\frac{1}{m}{\displaystyle \underset{j=1}{\overset{m}{\sum }}}{\eta }_{x_j}\left(y\right)K_{x_j}^{\mu }(\cdot )\rangle }_{K^{\mu }}\\ =\frac{1}{m^2}{\displaystyle \underset{i,j=1}{\overset{m}{\sum }}}{\eta }_{x_i}\left(y\right){\eta }_{x_j}\left(y\right)K^{\mu }\left(x_i,x_j\right).\end{array}$

By Markov inequality, we have:

$P\left({\Vert f_{z\mathrm{,}\lambda }-f_{\bar{X}\mathrm{,}\lambda }\Vert }_{K^{\mu }}>\epsilon \right)\le \frac{E\left({\Vert f_{z\mathrm{,}\lambda }-f_{\bar{X}\mathrm{,}\lambda }\Vert }_{K^{\mu }}^2\right)}{{\epsilon }^2}\mathrm{.}$ (35)

Then, by (31), we have:

$\begin{array}{c}{\Vert f_{z,\lambda }-f_{\bar{X}\mathrm{,}\lambda }\Vert }_{K^{\mu }}^2\le \frac{4}{{\lambda }^2{m}^2}{\Vert {\displaystyle \underset{i=1}{\overset{m}{\sum }}}{\eta }_{x_i}\left(y\right)K_{x_i}^{\mu }(\cdot )\Vert }_{K^{\mu }}^2\\ \le \frac{4}{{\lambda }^2{m}^2}{\langle {\displaystyle \underset{i=1}{\overset{m}{\sum }}}{\eta }_{x_i}\left(y\right)K_{x_i}^{\mu }(\cdot ),{\displaystyle \underset{j=1}{\overset{m}{\sum }}}{\eta }_{x_j}\left(y\right)K_{x_j}^{\mu }(\cdot )\rangle }_{K^{\mu }}\\ =\frac{4}{{\lambda }^2{m}^2}{\displaystyle \underset{i,j=1}{\overset{m}{\sum }}}{\eta }_{x_i}\left(y\right){\eta }_{x_j}\left(y\right)K^{\mu }\left(x_i,x_j\right).\end{array}$

Since $E_{\rho \left(\cdot \mathrm{<mo>\; |\; </mo>}x\right)}\left({\eta }_x\right)=0$ , we have:

$E\left({\Vert f_{z\mathrm{,}\lambda }-f_{\bar{X}\mathrm{,}\lambda }\Vert }_{K^{\mu }}^2\right)\le \frac{4k^2}{{\lambda }^{2}m}{\displaystyle {\int }_{B^d}}{\eta }_x^2\text{d}{\rho }_{B^d}=\frac{4k^2{\sigma }^2}{{\lambda }^{2}m}\mathrm{.}$

By (35), we have:

$P\left({\Vert f_{z\mathrm{,}\lambda }-f_{\bar{X}\mathrm{,}\lambda }\Vert }_{K^{\mu }}\le \epsilon \right)\ge 1-\frac{4k^2{\sigma }^2}{{\lambda }^{2}m{\epsilon }^2}\mathrm{.}$

Taking $\delta =\frac{4k^2{\sigma }^2}{{\lambda }^{2}m{\epsilon }^2}$ . We have $\epsilon =\frac{2k\sigma }{\lambda \sqrt{m\delta }}$ and (33) thus holds.

We now show (34). Take

$A={\Vert {\displaystyle {\int }_{B^d}}\left(f_{\lambda }\left(x\right)-f^{*}\left(x\right)\right)K_x^{\mu }(\cdot )\text{d}{\rho }_{B^d}-\frac{1}{m}{\displaystyle \underset{i=1}{\overset{m}{\sum }}}\left(f_{\lambda }\left(x_i\right)-f^{*}\left(x_i\right)\right)K_{x_i}^{\mu }(\cdot )\Vert }_{K^{\mu }}$

and

$\begin{array}{c}B=\Vert {\displaystyle {\int }_{S^{d-1}}}\left(\frac{\partial f_{\lambda }\left(x\right)}{\partial \stackrel{\to }{n}}-g\left(x\right)\right)\frac{\partial K_x^{\mu }(\cdot )}{\partial \stackrel{\to }{n}}\text{d}{\rho }_{S^{d-1}}\\ -{\frac{1}{l}{\displaystyle \underset{k=1}{\overset{l}{\sum }}}\left(\frac{\partial f_{\lambda }\left(x_{m+k}\right)}{\partial \stackrel{\to }{n}}-g\left(x_{m+k}\right)\right)\frac{\partial K_{x_{m+k}}^{\mu }(\cdot )}{\partial \stackrel{\to }{n}}\Vert }_{K^{\mu }}\mathrm{.}\end{array}$

Then,

${\Vert f_{\bar{X}\mathrm{,}\lambda }-f_{\rho \mathrm{,}\lambda }\Vert }_{K^{\mu }}\le \frac{2}{\lambda }\left(A+B\right)\mathrm{,}$ (36)

where

$\begin{array}{c}A\le {\Vert {\displaystyle {\int }_{B^d}}{f}_{\lambda }\left(x\right)K_x^{\mu }(\cdot )\text{d}{\rho }_{B^d}-\frac{1}{m}{\displaystyle \underset{i=1}{\overset{m}{\sum }}}{f}_{\lambda }\left(x_i\right)K_{x_i}^{\mu }(\cdot )\Vert }_{K^{\mu }}\\ +{\Vert {\displaystyle {\int }_{B^d}}{f}^{*}\left(x\right)K_x^{\mu }(\cdot )\text{d}{\rho }_{B^d}-\frac{1}{m}{\displaystyle \underset{i=1}{\overset{m}{\sum }}}{f}^{*}\left(x_i\right)K_{x_i}^{\mu }(\cdot )\Vert }_{K^{\mu }}.\end{array}$

Since

$\begin{array}{c}{\Vert f_{\lambda }\left(x\right)K_x^{\mu }(\cdot )\Vert }_{K^{\mu }}=\left|f_{\lambda }\left(x\right)\right|\sqrt{K_x^{\mu }\left(x\right)}\le k\left|f_{\lambda }\left(x\right)\right|\\ \le k^2{\Vert f\Vert }_{K^{\mu }}\le \frac{k^2\sqrt{K\left(f^{*},g,\lambda \right)}}{\sqrt{\lambda }}\end{array}$

and ${\Vert f^{\mathrm{<mo>\; *\; </mo>}}\left(x\right)K_x^{\mu }(\cdot )\Vert }_{K^{\mu }}\le k\left|f^{\mathrm{<mo>\; *\; </mo>}}\left(x\right)\right|\le k{\Vert f^{\mathrm{<mo>\; *\; </mo>}}\Vert }_{C\left(B^d\right)}$ , we have by (47) that, with confidence $1-\delta$ , holds:

$A\le \left(\frac{2k^2\sqrt{K\left(f^{\mathrm{<mo>\; *\; </mo>}}\mathrm{,}g\mathrm{,}\lambda \right)}}{\lambda \sqrt{m\lambda }}+\frac{2k{\Vert f^{\mathrm{<mo>\; *\; </mo>}}\Vert }_{C\left(B^d\right)}}{\lambda \sqrt{m}}\right)\log \frac{2}{\delta }\mathrm{.}$ (37)

On the other hand, take $\xi \left(x\right)=\left(\frac{\partial f_{\lambda }\left(x\right)}{\partial \stackrel{\to }{n}}-g\left(x\right)\right)$ . Then,

$B={\Vert {\displaystyle {\int }_{S^{d-1}}}\xi \left(x\right)\frac{\partial K_x^{\mu }(\cdot )}{\partial \stackrel{\to }{n}}\text{d}{\rho }_{S^{d-1}}-\frac{1}{l}{\displaystyle \underset{k=1}{\overset{l}{\sum }}}\xi \left(x_{m+k}\right)\frac{\partial K_{x_{m+k}}^{\mu }(\cdot )}{\partial \stackrel{\to }{n}}\Vert }_{K^{\mu }}\mathrm{.}$

By the definition of ${\Vert \cdot \Vert }_{K^{\mu }}$ , we have:

$\begin{array}{c}{B}^2=\langle {\displaystyle {\int }_{S^{d-1}}}\xi \left(x\right)\frac{\partial K_x^{\mu }(\cdot )}{\partial \stackrel{\to }{n}}\text{d}{\rho }_{S^{d-1}}-\frac{1}{l}{\displaystyle \underset{k=1}{\overset{l}{\sum }}}\xi \left(x_{m+k}\right)\frac{\partial K_{x_{m+k}}^{\mu }(\cdot )}{\partial \stackrel{\to }{n}},\\ {\displaystyle {\int }_{S^{d-1}}}\xi \left(u\right)\frac{\partial K_u^{\mu }(\cdot )}{\partial \stackrel{\to }{n}}\text{d}{\rho }_{S^{d-1}}-{\frac{1}{l}{\displaystyle \underset{i=1}{\overset{l}{\sum }}}\xi \left(x_{m+i}\right)\frac{\partial K_{x_{m+i}}^{\mu }(\cdot )}{\partial \stackrel{\to }{n}}\rangle }_{K^{\mu }}\\ ={\displaystyle {\int }_{S^{d-1}}}{\displaystyle {\int }_{S^{d-1}}}\xi \left(x\right)\xi \left(u\right){\langle \frac{\partial K_x^{\mu }(\cdot )}{\partial \stackrel{\to }{n}},\frac{\partial K_u^{\mu }(\cdot )}{\partial \stackrel{\to }{n}}\rangle }_{K^{\mu }}\text{d}{\rho }_{S^{d-1}}\left(x\right)\text{d}{\rho }_{S^{d-1}}\left(u\right)\\ -2{\displaystyle {\int }_{S^{d-1}}}\xi \left(x\right)\left(\frac{1}{l}{\displaystyle \underset{i=1}{\overset{l}{\sum }}}\xi \left(x_{m+i}\right){\langle \frac{\partial K_x^{\mu }(\cdot )}{\partial \stackrel{\to }{n}},\frac{\partial K_{x_{m+i}}^{\mu }(\cdot )}{\partial \stackrel{\to }{n}}\rangle }_{K^{\mu }}\right)\text{d}{\rho }_{S^{d-1}}\left(x\right)\\ +\frac{1}{l^2}{\displaystyle \underset{k=1}{\overset{l}{\sum }}}{\xi }^2\left(x_{m+k}\right){\langle \frac{\partial K_{x_{m+k}}^{\mu }(\cdot )}{\partial \stackrel{\to }{n}},\frac{\partial K_{x_{m+k}}^{\mu }(\cdot )}{\partial \stackrel{\to }{n}}\rangle }_{K^{\mu }}\\ +\frac{1}{l^2}{\displaystyle \underset{k,i=1,k\ne i}{\overset{l}{\sum }}}\xi \left(x_{m+i}\right)\xi \left(x_{m+j}\right){\langle \frac{\partial K_{x_{m+k}}^{\mu }(\cdot )}{\partial \stackrel{\to }{n}},\frac{\partial K_{x_{m+i}}^{\mu }(\cdot )}{\partial \stackrel{\to }{n}}\rangle }_{K^{\mu }}.\end{array}$

Since (14), we have:

${\langle \frac{\partial K_x^{\mu }(\cdot )}{\partial \stackrel{\to }{n}},\frac{\partial K_u^{\mu }(\cdot )}{\partial \stackrel{\to }{n}}\rangle }_{K^{\mu }}=\frac{{\partial }_u}{\partial \stackrel{\to }{n}}\frac{{\partial }_x}{\partial \stackrel{\to }{n}}{K}_x^{\mu }\left(u\right),$

${\langle \frac{\partial K_x^{\mu }(\cdot )}{\partial \stackrel{\to }{n}},\frac{\partial K_{x_{m+i}}^{\mu }(\cdot )}{\partial \stackrel{\to }{n}}\rangle }_{K^{\mu }}={\frac{{\partial }_u}{\partial \stackrel{\to }{n}}\frac{{\partial }_x}{\partial \stackrel{\to }{n}}{K}_x^{\mu }\left(u\right)|}_{u=x_{m+i}}$

and

${\langle \frac{\partial K_{x_{m+k}}^{\mu }(\cdot )}{\partial \stackrel{\to }{n}},\frac{\partial K_{x_{m+k}}^{\mu }(\cdot )}{\partial \stackrel{\to }{n}}\rangle }_{K^{\mu }}={\frac{{\partial }_u}{\partial \stackrel{\to }{n}}\frac{{\partial }_u}{\partial \stackrel{\to }{n}}{K}_u^{\mu }\left(u\right)|}_{u=x_{m+k}},$

${\langle \frac{\partial K_{x_{m+k}}^{\mu }(\cdot )}{\partial \stackrel{\to }{n}},\frac{\partial K_{x_{m+i}}^{\mu }(\cdot )}{\partial \stackrel{\to }{n}}\rangle }_{K^{\mu }}={\frac{{\partial }_u}{\partial \stackrel{\to }{n}}\frac{{\partial }_x}{\partial \stackrel{\to }{n}}{K}_x^{\mu }\left(u\right)|}_{u=x_{m+i},x=x_{m+k}}.$

It follows that:

$\begin{array}{c}{B}^2={\displaystyle {\int }_{S^{d-1}}}{\displaystyle {\int }_{S^{d-1}}}\xi \left(x\right)\xi \left(u\right)\frac{{\partial }_u}{\partial \stackrel{\to }{n}}\frac{{\partial }_x}{\partial \stackrel{\to }{n}}{K}_x^{\mu }\left(u\right)\text{d}{\rho }_{S^{d-1}}\left(x\right)\text{d}{\rho }_{S^{d-1}}\left(u\right)\\ -2{\displaystyle {\int }_{S^{d-1}}}\xi \left(x\right)\left(\frac{1}{l}{\displaystyle \underset{i=1}{\overset{l}{\sum }}}\xi \left(x_{m+i}\right){\frac{{\partial }_u}{\partial \stackrel{\to }{n}}\frac{{\partial }_x}{\partial \stackrel{\to }{n}}{K}_x^{\mu }\left(u\right)|}_{u=x_{m+i}}\right)\text{d}{\rho }_{S^{d-1}}\left(x\right)\\ +\frac{1}{l^2}{\displaystyle \underset{k=1}{\overset{l}{\sum }}}{\xi }^2\left(x_{m+k}\right){\frac{{\partial }_u}{\partial \stackrel{\to }{n}}\frac{{\partial }_u}{\partial \stackrel{\to }{n}}{K}_u^{\mu }\left(u\right)|}_{u=x_{m+k}}\\ +\frac{1}{l^2}{\displaystyle \underset{k,i=1,k\cancel{=}i}{\overset{l}{\sum }}}\xi \left(x_{m+i}\right)\xi \left(x_{m+j}\right){\frac{{\partial }_u}{\partial \stackrel{\to }{n}}\frac{{\partial }_x}{\partial \stackrel{\to }{n}}{K}_x^{\mu }\left(u\right)|}_{u=x_{m+i},x=x_{m+k}}.\end{array}$

Since $\left(x_{m+1}\mathrm{,}{x}_{m+2}\mathrm{,}\cdots \mathrm{,}{x}_{m+l}\right)$ are i.i.d. according to ${\rho }_{{\rho }_{S^{d-1}}}$ , we have:

$\begin{array}{c}E\left(B^2\right)=\frac{1}{l}({\displaystyle {\int }_{S^{d-1}}}{\xi }^2\left(x\right)\frac{{\partial }_x}{\partial \stackrel{\to }{n}}\frac{{\partial }_x}{\partial \stackrel{\to }{n}}{K}_x^{\mu }\left(x\right)\text{d}{\rho }_{S^{d-1}}\left(x\right)\\ -{\displaystyle {\int }_{S^{d-1}}}{\displaystyle {\int }_{S^{d-1}}}\xi \left(x\right)\xi \left(u\right)\frac{{\partial }_u}{\partial \stackrel{\to }{n}}\frac{{\partial }_x}{\partial \stackrel{\to }{n}}{K}_x^{\mu }\left(u\right)\text{d}{\rho }_{S^{d-1}}\left(x\right)\text{d}{\rho }_{S^{d-1}}\left(u\right))\\ \le \frac{1}{l}\left({\displaystyle {\int }_{S^{d-1}}}{\xi }^2\left(x\right)\frac{{\partial }_x}{\partial \stackrel{\to }{n}}\frac{{\partial }_x}{\partial \stackrel{\to }{n}}{K}_x^{\mu }\left(x\right)\text{d}{\rho }_{S^{d-1}}\left(x\right)\right),\end{array}$

where we have used the fact that:

${\displaystyle {\int }_{S^{d-1}}}{\displaystyle {\int }_{S^{d-1}}}\xi \left(x\right)\xi \left(u\right)\frac{{\partial }_u}{\partial \stackrel{\to }{n}}\frac{{\partial }_x}{\partial \stackrel{\to }{n}}{K}_x^{\mu }\left(u\right)\text{d}{\rho }_{S^{d-1}}\left(x\right)\text{d}{\rho }_{S^{d-1}}\left(u\right)\ge 0$

since $\frac{{\partial }_u}{\partial \stackrel{\to }{n}}\frac{{\partial }_x}{\partial \stackrel{\to }{n}}{K}_x^{\mu }\left(u\right)$ is a positive definition function about u and x. According to Assumption B, we have:

$\left|\frac{{\partial }_u}{\partial \stackrel{\to }{n}}\frac{{\partial }_x}{\partial \stackrel{\to }{n}}{K}_x^{\mu }\left(u\right)\right|\le k^2\mathrm{,}$

It follows that:

$E\left(B^2\right)\le \frac{k^2}{l}\left({\displaystyle {\int }_{S^{d-1}}}{\xi }^2\left(x\right)\text{d}{\rho }_{S^{d-1}}\left(x\right)\right)\mathrm{.}$

By Markov inequality, we have:

$\begin{array}{c}P\left(B>\epsilon \right)\le \frac{E\left(B^2\right)}{{\epsilon }^2}\\ \le \frac{k^2}{l{\epsilon }^2}\left({\displaystyle {\int }_{S^{d-1}}}{\xi }^2\left(x\right)\text{d}{\rho }_{S^{d-1}}\left(x\right)\right)\\ \le \frac{k^2}{l{\epsilon }^2}K\left(f^{*},g,\lambda \right).\end{array}$

Take $\delta =\frac{k^2}{l{\epsilon }^2}K\left(f^{\mathrm{<mo>\; *\; </mo>}}\mathrm{,}g\mathrm{,}\lambda \right)$ . Then, $\epsilon =\frac{k\sqrt{K\left(f^{\mathrm{<mo>\; *\; </mo>}}\mathrm{,}g\mathrm{,}\lambda \right)}}{\sqrt{l\delta }}$ . It follows that with confidence $1-\delta$ :

$B\le \frac{k\sqrt{K\left(f^{\mathrm{<mo>\; *\; </mo>}}\mathrm{,}g\mathrm{,}\lambda \right)}}{\sqrt{l\delta }}\mathrm{.}$ (38)

Collecting (38), (37) and (36), we arrive at (34).

4. Proofs

Proof of Proposition 2.1. Proof of 1). For any $f\left(x\right)={\displaystyle \underset{k=0}{\overset{\infty }{\sum }}}{\displaystyle \underset{l=1}{\overset{a_k^d}{\sum }}}{a}_{k\mathrm{,}l}\left(f\right)Q_{k\mathrm{,}l}\left(x\right)\in {\mathcal{H}}_{K^{\mu }}^{\stackrel{\to }{n}}$ . We rewrite $K^{\mu }\left(x\mathrm{,}y\right)$ as:

$K^{\mu }\left(x,y\right)=K_y^{\mu }\left(x\right)={\displaystyle \underset{k=0}{\overset{\infty }{\sum }}}{\displaystyle \underset{l=1}{\overset{a_k^d}{\sum }}}\left({\lambda }_{k,l}{Q}_{k,l}\left(y\right)\right)Q_{k,l}\left(x\right).$

Then,

${\langle f,K_y^{\mu }\rangle }_{K^{\mu }}={\displaystyle \underset{k=0}{\overset{\infty }{\sum }}}{\displaystyle \underset{l=1}{\overset{a_k^d}{\sum }}}\frac{\left({\lambda }_{k,l}{Q}_{k,l}\left(y\right)\right)a_{k,l}\left(f\right)}{{\lambda }_{k,l}}=f\left(y\right),$

which yields (13).

Proof of 2). Since (9) and (46), we have:

$\begin{array}{c}\frac{\partial f\left(x\right)}{\partial \stackrel{\to }{n}}={\langle f,{\displaystyle \underset{i=1}{\overset{d}{\sum }}}{x}^i\frac{\partial K\left(x,\cdot \right)}{\partial x^i}\left(x\right)\rangle }_{K^{\mu }}\\ ={\langle f,\frac{\partial }{\partial \stackrel{\to }{n}}K\left(x,\cdot \right)\rangle }_{K^{\mu }},x=\left(x^1,\cdots ,x^d\right)\in S^{d-1}.\end{array}$

(14) thus holds.

Proof of 3). By (14), we have:

$\left|\frac{\partial f\left(x\right)}{\partial \stackrel{\to }{n}}\right|=\left|{\langle f,\frac{\partial K_x^{\mu }(\cdot )}{\partial \stackrel{\to }{n}}\rangle }_{K^{\mu }}\right|\le {\Vert f\Vert }_{K^{\mu }}{\Vert \frac{\partial K_x^{\mu }(\cdot )}{\partial \stackrel{\to }{n}}\Vert }_{K^{\mu }}\le k{\Vert f\Vert }_{K^{\mu }}.$

By the same method, we have by (13) that:

$\begin{array}{c}\left|f\left(x\right)\right|=\left|{\langle f,K_x^{\mu }(\cdot )\rangle }_{K^{\mu }}\right|\\ \le {\Vert f\Vert }_{K^{\mu }}{\Vert K_x^{\mu }(\cdot )\Vert }_{K^{\mu }}\\ \le k{\Vert f\Vert }_{K^{\mu }}.\end{array}$

(15) thus holds.

Proof of Proposition 2.2. By (27), we have:

$\begin{array}{c}{f}_{z,\lambda }(\cdot )=\frac{1}{\lambda m}{\displaystyle \underset{i=1}{\overset{m}{\sum }}}\left(y_{x_i}-f_{z,\lambda }\left(x_i\right)\right)K_{x_i}^{\mu }(\cdot )\\ +\frac{1}{\lambda l}{\displaystyle \underset{k=1}{\overset{l}{\sum }}}\left(g\left(x_{m+k}\right)-\frac{\partial f_{z,\lambda }\left(x_{m+k}\right)}{\partial \stackrel{\to }{n}}\right)\frac{\partial K_{x_{m+k}}^{\mu }(\cdot )}{\partial \stackrel{\to }{n}}.\end{array}$ (39)

Taking $a_i=\frac{1}{\lambda m}\left(y_{x_i}-f_{z\mathrm{,}\lambda }\left(x_i\right)\right)$ and $a_{m+k}=\frac{1}{\lambda l}\left(g\left(x_{m+k}\right)-\frac{\partial f_{z\mathrm{,}\lambda }\left(x_{m+k}\right)}{\partial \stackrel{\to }{n}}\right)$ into (40), we have (20).

By (29), we have:

$\begin{array}{c}{f}_{\bar{X}\mathrm{,}\lambda }(\cdot )=\frac{1}{\lambda m}{\displaystyle \underset{i=1}{\overset{m}{\sum }}}\left(f^{*}\left(x_i\right)-f_{\bar{X},\lambda }\left(x_i\right)\right)K_{x_i}^{\mu }(\cdot )\\ +\frac{1}{\lambda l}{\displaystyle \underset{k=1}{\overset{l}{\sum }}}\left(g\left(x_{m+k}\right)-\frac{\partial f_{\bar{X},\lambda }\left(x_{m+k}\right)}{\stackrel{\to }{n}}\right)\frac{\partial K_{x_{m+k}}^{\mu }(\cdot )}{\partial \stackrel{\to }{n}}.\end{array}$ (40)

Taking $b_i=\frac{1}{\lambda m}\left(f^{\mathrm{<mo>\; *\; </mo>}}\left(x_i\right)-f_{\bar{X}\mathrm{,}\lambda }\left(x_i\right)\right)$ and $b_{m+k}=\frac{1}{\lambda l}\left(g\left(x_{m+k}\right)-\frac{\partial f_{\bar{X}\mathrm{,}\lambda }\left(x_{m+k}\right)}{\stackrel{\to }{n}}\right)$ into (40), we have (21).

By (29), we have:

$\begin{array}{l}{f}_{\lambda }(\cdot )=\frac{1}{\lambda }{\displaystyle {\int }_{B^d}}\left(f^{*}\left(x\right)-f_{\lambda }\left(x\right)\right)K_x^{\mu }(\cdot )\text{d}{\rho }_{B^d}\\ +\frac{1}{\lambda }{\displaystyle {\int }_{S^{d-1}}}\left(g\left(x\right)-\frac{\partial f_{\lambda }\left(x\right)}{\stackrel{\to }{n}}\right)\frac{\partial K_x^{\mu }(\cdot )}{\partial \stackrel{\to }{n}}\text{d}{\rho }_{S^{d-1}}.\end{array}$ (41)

Taking $G_{\lambda \mathrm{,}{f}^{\mathrm{<mo>\; *\; </mo>}}}\left(x\right)=\frac{1}{\lambda }\left(f^{\mathrm{<mo>\; *\; </mo>}}\left(x\right)-f_{\lambda }\left(x\right)\right)$ and $p_{\lambda \mathrm{,}g}\left(x\right)=\frac{1}{\lambda }\left(g\left(x\right)-\frac{\partial f_{\lambda }\left(x\right)}{\partial \stackrel{\to }{n}}\right)$ into (41), we have (22).

Proof of Theorem 2.1. Collecting (33), (34) and (23), we have:

$\begin{array}{l}{\Vert f_{z,\lambda }-f^{*}\Vert }_{L^2\left({\rho }_{B^d}\right)}+{\Vert \frac{\partial f_{z,\lambda }}{\partial \stackrel{\to }{n}}-g\Vert }_{L^2\left({\rho }_{S^{d-1}}\right)}\\ \le \frac{2k^2\sigma }{\lambda \sqrt{m\delta }}+k\left(\frac{\sqrt{K\left(f^{*},g,\lambda \right)}}{{\lambda }^{\frac{3}{2}}\sqrt{m}}+\frac{{\Vert f^{*}\Vert }_{C\left(B^d\right)}}{\lambda \sqrt{m}}\right)\log \frac{4}{\delta }\\ +\frac{k\sqrt{K\left(f^{*},g,\lambda \right)}}{\sqrt{l\delta }}+\sqrt{2}\sqrt{K\left(f^{*},g,\lambda \right)}.\end{array}$ (42)

(42) yields (24).

Founding

Supported partially by the NSF (Project No. 61877039), the NSFC/RGC Joint Research Scheme (Project No. 12061160462 and N_CityU102/20) of China.

Appendices

Appendix 1. Gâteaux Derivative and the Convex Function

Following Proposition A1 can be found from the Proposition 17.4, Proposition 17.10 and Proposition 17.12 of [38] .

Proposition A1. Let $\left(\mathcal{H}\mathrm{,}{\Vert \cdot \Vert }_{\mathcal{H}}\right)$ be a Hilbert space and $F\left(f\right)\mathrm{<mo>\; :\; </mo>}\mathcal{H}\to R\cup \left\{\mp \infty \right\}$ be a function defined on $\mathcal{H}$ . Then,

1) If $F\left(f\right)$ is a convex function, then, $F\left(f\right)$ attains minimal value at $f_0$ if and only if ${\nabla }_{f}F\left(f_0\right)=0$ .

2) If $F\left(f\right)\mathrm{<mo>\; :\; </mo>}\mathcal{H}\to R\cup \left\{\mp \infty \right\}$ is a Gâteaux differentiable function, then $F\left(f\right)$ is a convex on $\mathcal{H}$ if and only if for any $f\mathrm{,}g\in \mathcal{H}$ , there holds:

$F\left(g\right)-F\left(f\right)\ge {\langle g-f\mathrm{,}{\nabla }_{f}F\left(f\right)\rangle }_H\mathrm{.}$

In particular, we have:

$x^2-y^2\ge 2y\left(x-y\right)\mathrm{,}\forall x\mathrm{,}y\in R\mathrm{.}$ (43)

3) For function $F\left(f\right)={\Vert f\Vert }_{\mathcal{H}}^2$ , there holds ${\nabla }_{f}F\left(f\right)=2f$ and there holds equality:

${\Vert f\Vert }_H^2-{\Vert g\Vert }_H^2={\langle f-g\mathrm{,}2g\rangle }_H+{\Vert f-g\Vert }_H^2\mathrm{,}$ (44)

i.e.

${\Vert f\Vert }_H^2-{\Vert g\Vert }_H^2={\langle f-g\mathrm{,}{\nabla }_{g}F\left(g\right)\rangle }_H+{\Vert f-g\Vert }_H^2\mathrm{.}$

Appendix 2. Derivatives Reproducing Property

Let $\Omega \subset R^d$ be a compact subset which is the closure of its nonempty interior ${\Omega }^0$ . Let $K\left(x\mathrm{,}y\right)$ be a Mercer kernel on $\Omega \times \Omega$ having the expansion (see e.g. [39] ):

$K\left(x,y\right)={\displaystyle \underset{k=0}{\overset{+\infty }{\sum }}}{\lambda }_k{\phi }_k\left(x\right){\phi }_k\left(y\right),x,y\in \Omega ,$ (45)

where the convergence is absolute (for each $x\mathrm{,}y\in \Omega$ ) and uniform on $\Omega \times \Omega$ . Then, we have a proposition.

Proposition A2. Let $K\left(x\mathrm{,}y\right)$ be a Mercer kernel of form (45) and $K\in C^{\left(1\right)}\left(\Omega \times \Omega \right)$ . If ${\mathcal{H}}_K$ is a reproducing kernel Hilbert space such that:

$f\left(x\right)={\langle f\mathrm{,}K\left(\cdot \mathrm{,}x\right)\rangle }_{{\mathcal{H}}_K}\mathrm{,}f\in H_K\mathrm{,}x\in \Omega \mathrm{.}$

Then,

${\partial }_x^{\alpha }f\left(x\right)={\langle f,{\partial }_x^{\alpha }K\left(\cdot ,x\right)\rangle }_{{\mathcal{H}}_K^{\stackrel{\to }{n}}\left({\rho }_{\Omega }\right)},\left|\alpha \right|\le 1,f\in {\mathcal{H}}_K,x\in \Omega ,$ (46)

where $\left|\alpha \right|={\displaystyle \underset{i=1}{\overset{d}{\sum }}}{\alpha }_i\le 1$ .

Proof. It can be found from Theorem 1 in [12] , or see (v) in Theorem 4.7 in [40] .

Appendix 3. A Probability Inequality

Proposition A4. [41] Let $\xi$ be a random variable taking values in a real separable Hilbert space H on a probability space $\left(\Omega \mathrm{,}\mathcal{F}\mathrm{,}P\right)$ . Assume that there are a positive constant L such that ${\Vert \xi \Vert }_H\le L$ . Then, for all $n\ge 1$ and $0<\eta <1$ , it holds, with confidence $1-\eta$ , that:

${\Vert \frac{1}{n}{\displaystyle \underset{i=1}{\overset{n}{\sum }}}\xi \left({\Omega }_i\right)-E\left(\xi \right)\Vert }_H\le \frac{4L}{\sqrt{n}}\log \frac{2}{\eta }.$ (47)

Conflicts of Interest

The authors declare no conflicts of interest regarding the publication of this paper.

References

[1]	Atkinson, K., Hansen, O. and Chien, D. (2011) A Spectral Method for Elliptic Equations: The Neumann Problem. Advances in Computational Mathematics, 34, 295-317. https://doi.org/10.1007/s10444-010-9154-3
[2]	Atkinson, K., Chien, D. and Hansen, O. (2010) A Spectral Method for Elliptic Equation: The Dirichlet Problem. Advances in Computational Mathematics, 33, 169-189. https://doi.org/10.1007/s10444-009-9125-8
[3]	Atkinson, K. and Hansen, O. (2010) A Spectral Method for the Eigenvalue Problem for Elliptic Equations. Electronic Transactions on Numerical Analysis, 37, 386-412.
[4]	Li, X. (2009) Approximation of Potential Integral by Radial Bases for Solutions of Helmholtz Equation. Advances in Computational Mathematics, 30, 201-230. https://doi.org/10.1007/s10444-008-9065-8
[5]	Li, X. (2008) Rate of Convergence of the Method of Fundamental Solutions and Hyperinterpolation for Modified Helmholtz Equations on the Unit Ball. Advances in Computational Mathematics, 29, 393-413. https://doi.org/10.1007/s10444-007-9056-1
[6]	Li, X. (2008) Convergence of the Method of Fundamental Solutions for Poisson’s Equation on the Unit Sphere. Advances in Computational Mathematics, 28, 269-282. https://doi.org/10.1007/s10444-006-9022-3
[7]	Cialenco, I., Fasshauer, G.E. and Ye, Q. (2012) Approximation of Stochastic Partial Differential Equations by a Kernel-Based Collocation Method. International Journal of Computer Mathematics, 89, 2543-2561. https://doi.org/10.1080/00207160.2012.688111
[8]	Ding, L.L., Liu, Z.Y. and Xu, Q.Y. (2021) Multilevel RBF Collocation Method for the Fourth-Order Thin Plate Problem. International Journal of Wavelets, Multiresolution and Information Processing, 19, Article ID: 2050079. https://doi.org/10.1142/S0219691320500794
[9]	Fasshauer, G.E. and Ye, Q. (2013) Kernel-Based Collocation Methods versus Galerkin Finite Element Methods for Approximating Elliptic Stochastic Partial Differential Equation. In: Griebel, M. and Schweitzer, M., Eds., Meshfree Methods for Partial Differential Equations VI, Springer, Berlin, 155-170. https://doi.org/10.1007/978-3-642-32979-1_10
[10]	Fasshauer, G.E. and Ye, Q. (2012) A Kernel-Based Collocation Method for Elliptic Partial Differential Equations with Random Coefficients. In: Dick, J., Kuo, F., Peters, G. and Sloan, I., Eds., Monte Carlo and Quasi-Monte Carlo Methods 2012, Springer, Berlin, 331-347. https://doi.org/10.1007/978-3-642-41095-6_14
[11]	Ye, Q. (2014) Approximation of Nonlinear Stochastic Partial Differential Equations by a Kernel-Based Collocation Method. International Journal of Applied Nonlinear Science, 1, 156-172. https://doi.org/10.1504/IJANS.2014.061018
[12]	Zhou, D.X. (2008) Derivative Reproducing Properties for Kernel Methods in Learning Theory. Journal of Computational and Applied Mathematics, 220, 456-463. https://doi.org/10.1016/j.cam.2007.08.023
[13]	Bao, K.J., Qian, X., Liu, Z.Y. and Song, S.B. (2022) An Operator Learning Approach via Function—Valued Reproducing Kernel Hilbert Space for Diferential Equations. arXiv: 2202.09488.
[14]	Mo, Y. and Qian, T. (2014) Support Vector Machine Adapted Tikhonov Regularization Method to Solve Dirichlet Problem. Applied Mathematics and Computation, 245, 509-519. https://doi.org/10.1016/j.amc.2014.07.089
[15]	Sheng, B.H., Zhou, D.P. and Wang, S.H. (2022) The Kernel Regularized Learning Algorithm for Solving Laplace Equationn with Dirichlet Boundary. International Journal of Wavelets, Multiresolution and Information Processing, 20, Article ID: 2250031. https://doi.org/10.1142/S021969132250031X
[16]	Stepaniants, G. (2023) Learning Partial Differential Equations in Reproducing Kernel Hilbert Spaces. Journal of Machine Learning Research, 24, 1-72
[17]	Harosko, D.D. and Triebel, H. (2008) Distributions, Sobolev Spaces, Elliptic Equations. European Mathematical Society, Helsinki. https://doi.org/10.4171/042
[18]	Belkin, M. and Niyogi, P. (2004) Semi-Supervised Learning on Riemannian Manifolds. Machine Learning, 56, 209-239. https://doi.org/10.1023/B:MACH.0000033120.25363.1e
[19]	Belkin, M., Niyogi, P. and Sindhwani, V. (2006) Manifold Regularization: A Geometric Framework for Learning from Labeled and Unlabeled Examples. Journal of Machine Learning Research, 7, 2399-2434.
[20]	Niyogi, P. (2013) Manifold Regularization and Semi-Supervised Learning: Some Theoretical Analysis, Journal of Machine Learning Research, 14, 1229-1250.
[21]	Sheng, B.H. and Zhu, H.C. (2018) The Convergence Rate of Semi-Supervised Regression with Quadratic Loss. Applied Mathematics and Computation, 321, 11-24. https://doi.org/10.1016/j.amc.2017.10.033
[22]	Smale, S. and Zhou, D.X. (2004) Shannon Sampling and Function Reconstruction from Point Values. Bulletin of the American Mathematical Society, 41, 279-305. https://doi.org/10.1090/S0273-0979-04-01025-0
[23]	Cucker, F. and Smale, S. (2002) On the Mathematical Foundations of Learning Theory. Bulletin of the American Mathematical Society, 39, 1-49. https://doi.org/10.1090/S0273-0979-01-00923-5
[24]	Cucker, F. and Zhou, D.X. (2007) Learning Theory: An Approximation Theory Viewpoint. Cambridge University Press, Cambridge. https://doi.org/10.1017/CBO9780511618796
[25]	Chen, H., Zhou, Y.C., Tang, Y.Y., Li, L.Q. and Pan, Z.B. (2013) Convergence Rate of the Semi-Supervised Greedy Algorithm. Neural Networks, 44, 44-50. https://doi.org/10.1016/j.neunet.2013.03.001
[26]	Sheng, B.H. and Xiang, D.H. (2017) The Performance of Semi-Supervised Laplacian Regularized Regression with Least Square Loss. International Journal of Wavelets, Multiresolution and Information Processing, 15, Article ID: 1750016. https://doi.org/10.1142/S0219691317500163
[27]	Sheng, B.H., Xiang, D.H. and Ye, P.X. (2015) Convergence Rate of Semi-Supervised Gradient Learning. International Journal of Wavelets, Multiresolution and Information Processing, 13, Article ID: 1550021. https://doi.org/10.1142/S0219691315500216
[28]	Sheng, B.H. and Zhang, H.Z. (2020) Performance Analysis of the LapRSSLG Algorithmin Learning Theory. Analysis and Applications, 18, 79-108. https://doi.org/10.1142/S0219530519410033
[29]	Wang, K.Y. and Li, L.Q. (2000) Harmonic Analysis and Approximation on the Unit Sphere. Science Press, Beijing.
[30]	Delgado, A.M., Fernández, L., Lubinsky, D., Pérez, T.E. and Piñar, M.A. (2016) Sobolev Orthogonal Polynomials on the Unit Ball via Outward Normal Derivatives. Journal of Mathematical Analysis and Applications, 440, 716-740. https://doi.org/10.1016/j.jmaa.2016.03.041
[31]	Jordao, T. and Menegatto, V.A. (2012) Reproducing Properties of Differentiable Mercer-Like Kernels on the Sphere. Numerical Functional Analysis and Optimization, 33, 1221-1243. https://doi.org/10.1080/01630563.2012.660590
[32]	Castro, M.H., Menegatto, V.A. and Oliveira, C.P. (2013) Laplace-Beltrami Differentiability of Positive Definite Kernels on the Sphere. Acta Mathematica Sinica, English Series, 29, 93-104. https://doi.org/10.1007/s10114-012-1067-2
[33]	Ferreira, J.C. and Menegatto, V.A. (2013) Positive Definiteness, Reproducing Kernel Hilbert Spaces and Beyond. Annals of Functional Analysis, 4, 64-88. https://doi.org/10.15352/afa/1399899838
[34]	Sun, H.W. and Wu, Q. (2011) Least Square Regression with Independent Kernels and Coefficient Regularization. Applied and Computational Harmonic Analysis, 30, 96-109. https://doi.org/10.1016/j.acha.2010.04.001
[35]	Zhang, J., Wang, J.L. and Sheng, B.H. (2011) Learning from Regularized Regression Algorithms with P-Order Markov Chain Sampling. Applied Mathematics: A Journal of Chinese Universities, 226, 295-306. https://doi.org/10.1007/s11766-011-2701-y
[36]	Sheng, B.H. and Wang, J.L. (2024) Moduli of Smoothness, K-Functionals and Jackson-Type Inequalities Associated with Kernel Function Approximation in Learning Theory. Analysis and Applications. https://doi.org/10.1142/S021953052450009X
[37]	Dai, F. and Xu, Y. (2013) Approximation Theory and Harmonic Analysis on Spheres and Balls. Springer-Verlag, New York. https://doi.org/10.1007/978-1-4614-6660-4
[38]	Bauschke, H.H. and Combettes, P.L. (2010) Convex Analysis and Monotone Operator Theory in Hilbert Spaces. Springer-Verlag, New York. https://doi.org/10.1007/978-1-4419-9467-7
[39]	Aronszajn, N. (1950) Theory of Reproducing Kernels. Transactions of the American Mathematical Society, 68, 337-404. https://doi.org/10.1090/S0002-9947-1950-0051437-7
[40]	Ferreira, J.C. and Menegatto, V.A. (2012) Reproducing Properties of Differentiable Mercer-Like Kernels. Mathematische Nachrichten, 285, 959-973. https://doi.org/10.1002/mana.201100072
[41]	Smale, S. and Zhou, D.X. (2007) Learning Theory Estimates via Integral Operators and Their Applications. Constructive Approximation, 26, 153-172. https://doi.org/10.1007/s00365-006-0659-y