The Behaviour of the Dispersion Matrix of the Information Matrix Test under the Wrong Logistic Regression Model ()
Estimation of Parameters
1. Introduction
The IMT is a test for general misspecification, produced by [1] who pointed out that the properties of the Maximum likelihood estimator and the information matrix can be exploited to yield a family of useful tests for model mis-specification. The idea of the IMT is to compare two different estimators of the information matrix to assess model fit. The IMT is based on the information matrix equality that obtains when the model specification is correct. This equality implies the asymptotic equivalence of the Hessian and the score forms of Fisher’s information matrix [2]. As [1], points out, the IMT is designed to detect the failure of this equality and the failure implies the model misspecification. [3] discussed the information matrix test and showed that it is useful with binary data models. Many researchers, [4] [5] and [6] pointed out the behaviour of the asymptotic distribution of IMT statistic and dispersion matrix. The idea of the
information matrix test is to compare
and
, as these
differ when the model is mis-specified but not when the model is correct. [7], pointed out, can be estimated the covariance matrix of IMT, dependent upon the IMT of [1], which can be estimated without the computation of analytic third derivatives of the density function. [4], discussed that, the IMT is sensitive to non-normality. Moreover, he proposed a simple computation procedure which employs the Outer Product of the Gradient (OPG) covariance matrix estimator of IMT statistic. However, [5] argue that, such a procedure maybe give unreliable inferences, related to the stochastic nature of the covariance matrix estimator which uses high sample moments to estimate high population moments. [6] purposed a simple calculation procedure for the test statistic, for general binary data models, which employs the ML covariance matrix estimator instead the OPG estimator. Moreover, [8], computed and examined IMT and found it had good power for logistic model.
Basic Idea of the IMT
Let us consider the density function
for individual observation and the data are independent, identically distribution so we have
and we consider
to be the logarithm of a density function of x dependent upon p parameters
, so the log-likelihood function in this case is
Now, as we defined the idea of the IMT to compare two different matrix of expected the first and second partial derivatives of the
, we have
(1)
So, according to the ML method, we have
Differentiating (1) again we get
(2)
So
(3)
When the model is mis-specified, the above quantity will be not necessarily equal zero.
Asymptotic Distribution of
The asymptotic distribution of estimated parameters and the behaviour of the MLE under the wrong model discussed by [9] and more investigated considered by [10]. [11], pointed out the estimation the parameters of a given regression model. In the limit for each value of the parameter vector
,
where
denoted to the true model and
is the fitted model. Also, consider the Kullback-Leibler divergence (KL) from the true to the approximating model conditional on X, under the wrong model. In this case
, where
is the least false value (LF). Note that the least false value
minimizes the KL divergence, because the derivative of the KL is
Also, if we need define
and
these matrixes are identical when
for all Y. As explained in [11], the distribution of the
, in this case from the central limit theorem there is convergence in distribution
where,
, which is leads to
So, we can say, the asymptotic MLE distribution under the null hypotheses H0, in this case
where,
is the true value. And the asymptotic distribution of
under alternative hypotheses H1 is
So, that is meaning (
) if and only if when fitted the correct model (i.e. under H0).
2. The IMT under Missing Covariates for Logistic Regression Model
In this part, we apply the procedure of the IMT statistic under missing covariates for a logistic regression model. If
is a p-dimensional vector of covariates draw from normal distribution and
is binary with
(4)
In the following we treat the simple case where the fitted model is
(5)
for a scalar
and that the true model has
(6)
where
is also a scalar. We have the log-likelihood function contribution for the ith element
is
(7)
and so,
and note that we only consider fitting the model with
, even if the true model also includes
(i.e.
). From this we get:
Also,
using,
as
is
, and so we get that
(8)
3. An Alternative Formulae of Variance
In this part we are interested to find a formulae of the variance of d statistic, even when the model is mis-specified. To perform the IMT we need to find the mean and variance of
Under H0
, and so the IMT could be written as
which will have a
-distribution on
d.f. as T is asymptotically Normal. However, the test statistic has to be evaluated at the MLE
and this introduces a complication. The MLE
is the solution to
The expression for T is
and this is clearly going to be highly correlation with S. Therefore, the appropriate variance for the IMT is
. As T and S are sums of independent elements, the Central limit Theorem implies that
is asymptotically Normal and so we can use
(9)
To work out
, so, in this case we can write
and similarly
3.1. The Variance of IMT under Missing Covariates for Logistic Regression Model
We now need to find expressions for
,
and
We already have that
and
so, the variance is
(10)
and we have
(11)
taking expectation
we obtain
(12)
and,
(13)
Now we need to compute
. In fact
, not only if the model is correct but also when evaluated at the least false value
(under wrong model), so in this case
and we have
then,
(14)
Now we will work out
, as before, since
, so
and note that
where,
is
under the true model. So,
(15)
Hence, the required variance (9)
(16)
and we have expressions for each component from (12), (13), (14) and (15) We need to evaluate these components by simulation.
3.2. The Dispersion Matrix under Wrong Model
In fact, may be some elements of the covariance matrix of the IMT are linear combinations of others leading to singularity of the estimated covariance matrix, this point discussed by [1] and [12]. We are interested to compute the
, even when the wrong model has been fitted. We will compute each of the components of this variance separately. We see from Section 3.1 that we need to evaluate, e.g.
and also,
This cannot be done analytically so we simulate 5000 values of X and replace the
by the mean of these 5000 values. In evaluating
we use the values of the parameters
,
and
. What do we use for
? We need to evaluate
at the least false values
and
for
and
. So, e.g, the first element of
is found by simulation from
where,
(17)
(18)
and X draw from bivariate normal distribution with
, and
. The formulae of the least false values
and
has been discussed and calculated by [10].
4. Empirical Variance of IMT
The expression in (16) is the variance V of d at
but we need an estimate,
. If we have a sample
how can we estimate V consistently? One candidate would be to compute
and
where,
is the fitted value from the model with just
. Now compute
and
Then use
(19)
as an estimate of V, we will assess this by simulation.
5. Simulation Study
This simulation examines the correctness of the form of the dispersion matrix V in (16) and (19). To achieve the aim of this simulation, we will consider a logistic regression model which has two covariates draw from bivariate normal distribution with mean zero and covariance matrix
as:
and the fitted model is
・ Apply in two cases of logistic model,
・ The fitted is the true logistic model (i.e.
)
・ The fitted model is mis-specified (i.e.
).
・ Use variance (
) and correlation
.
・ We choose some different components of parameters
,
and
to calculate
.
・ We compute the least false values
and
by formulae to calculate
.
・ We compute the true variance by simulating
and take the variance to be
.
・ We compute the theoretical variance
at the least false value and calculate
and
as described in section 3.2.
・ Finally, for each simulation we compute the empirical variance
and take the mean over the simulations.
・ We make comparison between the diagonal elements of dispersion matrix
vs.
respectively.
・ Apply on different sample size
and
number of simulations.
6. Results and Discussion
The results were reported in tables, which show the diagonal elements of the variance matrix:
denotes the empirical variance,
denotes the theoretical variance and
denotes the true variance. The true parameters appear as
,
, and
;
and
denote to the rank of the covariance matrix
empirical and theoretical respectively. The Ratio
and
are
,
respectively.
denotes the standard deviation over a sample
where
is the true model. In our simulation we consider two covariates, so in this case the dispersion matrix of d is a 3 × 3 dimensional matrix.
Firstly, we consider the results under true logistic model, Table 1, shows the results of simulation, which appeared the diagonal elements of matrix V, the empirical version and theoretical form comparing with true variance, which use
in case of
by sample size
. Table 2, reported the results by sample size
, with equal variance
. We can see clearly, that all diagonal elements appeared small in value in two different cases of sample size. The first element was much closer to zero than of the rest. In almost cases the results appeared reasonable ratio which is meaning the theoretical variance and empirical variance are close to the true value. There are some slightly strange ratio almost in case of sample size
, the reason may be affected by small value of standard deviation of
, otherwise the ratio is close to one. In case of sample size
, the behaviour of results shows almost the same pattern, with the ratio close to one and that is meaning the formulae of the variance works well. In a few cases with small values of
which affected on the ratio where the first two elements were more sensitive. Overall, we have reasonable results to say that, the alternative formulae of variance works well and the two first elements still more sensitive which appeared tend to zero.
Secondly, we consider the results when the missing covariate logistic model has been fitted. That is meaning when the variance of IMT computed under H1 and uses the least false values. Table 3, shows the results of sample size
. Table 4, shows the results of sample size 1000. In general, the behaviour of ratio
Table 1. Simulation results of the variance (
), (
) and (
) in case of fitted true model, with sample size
and
.
Table 2. Simulation results of the variance (
), (
) and (
) in case of fitted true model, with sample size
and
.
Table 3. Simulation results of the variance (
), (
) and (
) in case of fitted missing covariates model, with sample size
and
.
Table 4. Simulation results of the variance (
), (
) and (
) in case of fitted missing covariates model, with sample size
and
.
appeared the same behaviour which found in case of
, the two cases of different sample size appeared reasonable ratio which is close to one. A few cases shows low ratio, the reason is as discussed before concerning to the small value of
.
7. Conclusion
This paper carried out to investigate the behaviour of IMT and compute the covariance matrix under the wrong logistic regression model. As result, we can see that the alternative formula of the variance appeared reasonable results under the true and missing covariate model. As we computed the final form of the variance of IMT, we can see clearly it is dependent on
. As we know, we made some notes on the first two elements of
, which may be quite close to zero under true model and use the least false value, the
related to the log likelihood functions. So, these elements leading to singularity of the estimated covariance matrix, and have effect on the behaviour of the dispersion matrix of the IMT.
Acknowledgements
I am very grateful to Professor J. N. S. Matthews, School of Mathematics and Statistics, Newcastle University for academic supporting and Dr. Hamza M. A. Boauod, (FCOPHTH) (SA), Consultant Ophthalmologist, Eye Department, Klerksdorp Hospital. South Africa for his financial support. Also thank the referees, associate editor and joint editor for their helpful comments and additional references.