《实用非参数统计》课程教学资源（阅读材料）A Review on Empirical Likelihood Methods for Regression.pdf

inference.The regression models considered in this review cover parametric,nonpara- metric and semiparametric regression models.In addition to the case of completely observed data,we also accommodate missing and censored data in this review. The EL method (Owen,1988,1990)owns its broad usage and fast research de- velopment to a number of important advantages.Generally speaking,it combines the reliability of nonparametric methods with the effectiveness of the likelihood approach. It yields confidence regions that respect the boundaries of the support of the target parameter.The regions are invariant under transformations and behave often better than confidence regions based on asymptotic normality when the sample size is small. Moreover,they are of natural shape and orientation since the regions are obtained by contouring a log likelihood ratio,and they often do not require the estimation of the variance,as the studentization is carried out internally via the optimization procedure. The EL method turns out appealing not only in getting confidence regions,but it also has its unique attractions in parameter estimation and formulating goodness-of-fit tests. 2 Parametric regression Suppose that we observe a sample of independent observations(i)where each Yi is regarded as the response of a d-dimensional design (covariate)variable Xi The preliminary interest here is in the conditional mean function (regression function) of Yi given Xi.One distinguishes between the design Xi being either fixed or random. Despite regression is conventionally associated with fixed designs,for ease of presen- tation,we will concentrate on random designs.The empirical likelihood analysis for fixed designs can be usually extended by regularizing the random designs. Consider first the following parametric regression model: Yi=m(Xi;B)+i,for i=1,...,n, (1) where m(r;B)is the known regression function with an unknown p-dimensional (p<n) parameter BE RP,the errors ei are independent random variables such that E(eilXi)= 0 and Var(eiXi)=o2(Xi)for some function ()Hence,the errors can be het- eroscedastic.We require,like in all empirical likelihood formulations,that the errors e;have finite conditional variance,which is a minimum condition needed by the em- pirical likelihood method to ensure a limiting chi-square distribution for the empirical likelihood ratio. The parametric regression function includes as special cases(i)the linear regression with m(x;B)=B:(ii)the generalized linear model (McCullagh and Nelder,1989) with m(;B)=G(r B)and a2(r)=ov{G(B)}for a known link function G,a known variance function V(),and an unknown constant o>0.Note that for these two special cases p d. In the absence of model information on the conditional variance,the least squares (LS)regression estimator of B is obtained by minimizing the sum of least squares 5n(a)=:∑-m(X:a2. =1 The LS estimator of B is Bis =arg infg Sn(B).When the regression function m(;B) is smooth enough with respect to B,Bis will be a solution of the following estimating

2 inference. The regression models considered in this review cover parametric, nonparametric and semiparametric regression models. In addition to the case of completely observed data, we also accommodate missing and censored data in this review. The EL method (Owen, 1988, 1990) owns its broad usage and fast research development to a number of important advantages. Generally speaking, it combines the reliability of nonparametric methods with the effectiveness of the likelihood approach. It yields confidence regions that respect the boundaries of the support of the target parameter. The regions are invariant under transformations and behave often better than confidence regions based on asymptotic normality when the sample size is small. Moreover, they are of natural shape and orientation since the regions are obtained by contouring a log likelihood ratio, and they often do not require the estimation of the variance, as the studentization is carried out internally via the optimization procedure. The EL method turns out appealing not only in getting confidence regions, but it also has its unique attractions in parameter estimation and formulating goodness-of-fit tests. 2 Parametric regression Suppose that we observe a sample of independent observations {(X T i , Yi) T } n i=1, where each Yi is regarded as the response of a d-dimensional design (covariate) variable Xi . The preliminary interest here is in the conditional mean function (regression function) of Yi given Xi . One distinguishes between the design Xi being either fixed or random. Despite regression is conventionally associated with fixed designs, for ease of presentation, we will concentrate on random designs. The empirical likelihood analysis for fixed designs can be usually extended by regularizing the random designs. Consider first the following parametric regression model: Yi = m(Xi ; β) + εi , for i = 1, . . . , n, (1) where m(x; β) is the known regression function with an unknown p-dimensional (p < n) parameter β ∈ R p , the errors εi are independent random variables such that E(εi |Xi) = 0 and Var(εi |Xi) = σ 2 (Xi) for some function σ(·). Hence, the errors can be heteroscedastic. We require, like in all empirical likelihood formulations, that the errors ǫi have finite conditional variance, which is a minimum condition needed by the empirical likelihood method to ensure a limiting chi-square distribution for the empirical likelihood ratio. The parametric regression function includes as special cases (i) the linear regression with m(x; β) = x T β; (ii) the generalized linear model (McCullagh and Nelder, 1989) with m(x; β) = G(x T β) and σ 2 (x) = σ 2 0V {G(x T β)} for a known link function G, a known variance function V (·), and an unknown constant σ 2 0 > 0. Note that for these two special cases p = d. In the absence of model information on the conditional variance, the least squares (LS) regression estimator of β is obtained by minimizing the sum of least squares Sn(β) =: Xn i=1 {Yi − m(Xi ; β)} 2 . The LS estimator of β is βˆ ls = arg infβ Sn(β). When the regression function m(x; β) is smooth enough with respect to β, βˆ ls will be a solution of the following estimating

3 equation: Om(:(Y-m(X::))0. (2) 83 Suppose that Bo is the true parameter value such that it is the unique value to make Em(=0.Let p,n be a set of probability weights allocated to the data.The empirical likelihood(EL)for B,in the spirit of Owen(1988) and (1991),is Ln(a=maxΠpa, (3) =1 where the maximization is subject to the constraints ∑n=1aud (4) = .Om(XB)(Y:-m(X:B))=0. (5) 83 The empirical likelihood,as conveyed by (3),is essentially a constrained profile like- lihood,with a trivial constraint (4)indicating the p,'s are probability weights.The constraint (5)is the most important one as it defines the nature of the parameters This formulation is similar to the original one given in Owen(1988,1990)for the mean parameter,say u,of Xi.There the second constraint,reflecting the nature of u,was given by∑-1pi(X-)=0. In getting the empirical likelihood at each candidate parameter value B,the above optimization problem as given in (3),(4)and(5)has to be solved for the optimal Pi's.It may be surprising in first instance that the above optimization problem can admit a solution as there are n pi's to be determined with only p+1 constraints.As the objective function In(B)is concave,and the constraints are linear in the Pi's,the optimization problem does admit unique solutions. The algorithm for computing Ln(B)at a candidate B is as follows.If the convex hull of the set of points (depending on {mm(in RPcontains the origin(zero)in RP,then the EL optimization problem for Ln(B)admits a solution. If the zero of RP is not contained in the convex hull of the points for the given B,then Ln(B)does not admit a finite solution as some weights pi are forced to take negative values;see Owen (1988;1990)for a discussion on this aspect.Tsao (2004)studied the probability of the EL not admitting a finite solution and the dependence of this probability on dimensionality. By introducing the Lagrange multipliers Ao ER and AIE RP,the constrained op- timization problem (3)-(5)can be translated into an unconstrained one with objective function T(p,0,A1)=∑1og,)+o(∑P-1)+T∑p Om(XY:-m(XB)(6) 03 where p=(p1,n)T.Differentiating T(p,o,with respect to each pi and set- ting the derivative to zero,it can be shown after some algebra that Ao =-n and by

3 equation: Xn i=1 ∂m(Xi ; β) ∂β {Yi − m(Xi ; β)} = 0. (2) Suppose that β0 is the true parameter value such that it is the unique value to make E[ ∂m(Xi;β) ∂β {Yi−m(Xi ; β)}|Xi ] = 0. Let p1, · · · , pn be a set of probability weights allocated to the data. The empirical likelihood (EL) for β, in the spirit of Owen (1988) and (1991), is Ln(β) = maxYn i=1 pi , (3) where the maximization is subject to the constraints Xn i=1 pi = 1 and (4) Xn i=1 pi ∂m(Xi ; β) ∂β {Yi − m(Xi ; β)} = 0. (5) The empirical likelihood, as conveyed by (3), is essentially a constrained profile likelihood, with a trivial constraint (4) indicating the pi ’s are probability weights. The constraint (5) is the most important one as it defines the nature of the parameters. This formulation is similar to the original one given in Owen (1988, 1990) for the mean parameter, say µ, of Xi . There the second constraint, reflecting the nature of µ, was given by Pn i=1 pi(Xi − µ) = 0. In getting the empirical likelihood at each candidate parameter value β, the above optimization problem as given in (3), (4) and (5) has to be solved for the optimal pi ’s. It may be surprising in first instance that the above optimization problem can admit a solution as there are n pi ’s to be determined with only p + 1 constraints. As the objective function Ln(β) is concave, and the constraints are linear in the pi ’s, the optimization problem does admit unique solutions. The algorithm for computing Ln(β) at a candidate β is as follows. If the convex hull of the set of points (depending on β) { ∂m(Xi;β) ∂β {Yi−m(Xi ; β)}}n i=1 in R p contains the origin (zero) in R p , then the EL optimization problem for Ln(β) admits a solution. If the zero of R p is not contained in the convex hull of the points for the given β, then Ln(β) does not admit a finite solution as some weights pi are forced to take negative values; see Owen (1988; 1990) for a discussion on this aspect. Tsao (2004) studied the probability of the EL not admitting a finite solution and the dependence of this probability on dimensionality. By introducing the Lagrange multipliers λ0 ∈ R and λ1 ∈ R p , the constrained optimization problem (3)-(5) can be translated into an unconstrained one with objective function T(p, λ0, λ1) = Xn i=1 log(pi) +λ0( Xn i=1 pi −1) +λ T 1 Xn i=1 pi ∂m(Xi ; β) ∂β {Yi − m(Xi ; β)}, (6) where p = (p1, · · · , pn) T . Differentiating T(p, λ0, λ1) with respect to each pi and setting the derivative to zero, it can be shown after some algebra that λ0 = −n and by

4 defining λ = −nλ1, we find that the optimal pi ’s are given by: pi = 1 n 1 1 + λT ∂m(Xi;β) ∂β {Yi − m(Xi ; β)} , where, from the structural constraint (5), λ satisfies Xn i=1 ∂m(Xi;β) ∂β {Yi − m(Xi ; β)} 1 + λT ∂m(Xi;β) ∂β {Yi − m(Xi ; β)} = 0. (7) Substituting the optimal weights into the empirical likelihood in (3), we get Ln(β) = Yn i=1 1 n 1 1 + λT ∂m(Xi;β) ∂β {Yi − m(Xi ; β)} and the log empirical likelihood is ℓn(β) =: log{Ln(β)} = − Xn i=1 log{1 + λ T ∂m(Xi ; β) ∂β {Yi − m(Xi ; β)} − n log(n). (8) The computing intensive nature of the empirical likelihood is clear from the above discussions. Indeed, to evaluate the EL at a β, one needs to solve the non-linear equation (7) for the λ which depends on β. An alternative computational approach, as given in Owen (1990) is to translate the optimization problem (3)-(5) with respect to the EL weights {pi} n i=1 to its dual problem with respect to λ. The dual problem to (3)-(5) involves minimizing an objective function Q(λ) =: − Xn i=1 log{1 + λ T ∂m(Xi ; β) ∂β {Yi − m(Xi ; β)}, which is the first term in the empirical likelihood ratio in (8), subject to 1 + λ T ∂m(Xi ; β) ∂β {Yi − m(Xi ; β)} ≥ 1/n for each i = 1, . . . , n. (9) The constraint (9) comes from 0 ≤ pi ≤ 1 for each i, whereas the gradient of Q(λ) is the function on the left hand side of (7). Let D = λ : 1 + λ T ∂m(Xi ; β) ∂β {Yi − m(Xi ; β)} ≥ 1/n for each i = 1, . . . , n . Then, the dual problem becomes the problem of minimizing Q(λ) over the set D. It can be verified that D is convex, closed and compact. Hence, there is a unique minimum within D. As suggested in Owen (1990), the set D can be removed by modifying the log(x) function in Q(λ) by a log∗ (x) such that log∗ (x) = log(x) for x ≥ 1/n; and log∗ (x) = −n 2x 2 /2 + 2nx − 3/2 − log(n) for x < 1/n, which is the quadratic function that matches log(x) and its first two derivatives at x = 1/n. We note that the profile likelihood Qn i=1 pi achieves its maximum n −n when all the weights pi equal n −1 for i = 1, · · · , n. Thus, if there exists a β, say βˆ, which solves (7) with λ = 0, namely Xn i=1 ∂m(Xi ; β) ∂β {Yi − m(Xi ; β)} = 0, (10)

then the EL attains its maximum Ln(B)=n-n at B.In the parametric regression we are considering.the number of parameters and the number of equations in(10)are the same.Hence,(10)has a solution B with probability approaching one in large samples. There are inference situations where the number of estimating equations is larger than the number of parameters (strictly speaking,dimension of the parameter space),for instance the Generalized Method of Moments in econometrics (Hansen,1982).Here, more model information is accounted for by imposing more moment restrictions,leading to more estimating equations than the number of parameters in the model.In statistics, they appear in the form of extra model information.In these so-called over-identified situations,the maximum EL,still using the notation Ln(B),may be different from n-.See Qin and Lawless(1994)for a discussion on this issue. Following the convention of the standard parametric likelihood,we can define from (8)the log EL ratio rn(8)=-2l0g(Ln(B)/n()=2 log1+xm((Y-m(X:B))).(11) 83 =1 Wilks'theorem(Wilks,1938)is a key property of the parametric likelihood ratio.If we replace the EL Ln(B)by the corresponding parametric likelihood,say Lpn(B),and use rpn(B)to denote the parametric likelihood ratio,according to Wilks'theorem,under certain regularity conditions, rpm(o)4X话asn一oo (12) This property is maintained by the EL,as is demonstrated in Owen (1990)for the mean parameter,Owen(1991)for linear regression,and many other situations(Qin and Lawless,1994,Molanes Lopez,Van Keilegom and Veraverbeke,2009).In the context of parametric regression, rn(8o)4xp as noo. (13) This can be viewed as a nonparametric version of Wilks'theorem,and it is quite re- markable for the empirical likelihood to achieve such a property under a nonparametric setting with much less parametric distributional assumptions.We call this analogue of sharing the Wilks'theorem the first order analogue between the parametric and the empirical likelihood. To appreciate why the nonparametric version of Wilks'theorem is valid,we would like to present a few steps of derivation that offer some insights into the nonparametric likelihood.Typically,the first step in a study on EL is considering an expansion for A defined in (7)at Bo,the true value of B,and determining its order of magnitude.It can be shown that for the current parametric regression, X=Op(n-1/2). (14) Such a rate for A is obtained in the original papers of Owen (1988,1990)for the mean parameter (which can be treated as a trivial case of regression without covariates), in Owen (1991)for linear regression,and in Qin and Lawless (1994)and Molanes Lopez,Van Keilegom and Veraverbeke (2009)for the more general case of estimating equations

5 then the EL attains its maximum Ln(βˆ) = n −n at βˆ. In the parametric regression we are considering, the number of parameters and the number of equations in (10) are the same. Hence, (10) has a solution βˆ with probability approaching one in large samples. There are inference situations where the number of estimating equations is larger than the number of parameters (strictly speaking, dimension of the parameter space), for instance the Generalized Method of Moments in econometrics (Hansen, 1982). Here, more model information is accounted for by imposing more moment restrictions, leading to more estimating equations than the number of parameters in the model. In statistics, they appear in the form of extra model information. In these so-called over-identified situations, the maximum EL, still using the notation Ln(βˆ), may be different from n −n . See Qin and Lawless (1994) for a discussion on this issue. Following the convention of the standard parametric likelihood, we can define from (8) the log EL ratio rn(β) = −2 log{Ln(β)/Ln(βˆ)} = 2Xn i=1 log{1 + λ T ∂m(Xi ; β) ∂β (Yi − m(Xi ; β))}. (11) Wilks’ theorem (Wilks, 1938) is a key property of the parametric likelihood ratio. If we replace the EL Ln(β) by the corresponding parametric likelihood, say Lpn(β), and use rpn(β) to denote the parametric likelihood ratio, according to Wilks’ theorem, under certain regularity conditions, rpn(β0) d→ χ 2 p as n → ∞. (12) This property is maintained by the EL, as is demonstrated in Owen (1990) for the mean parameter, Owen (1991) for linear regression, and many other situations (Qin and Lawless, 1994, Molanes L´opez, Van Keilegom and Veraverbeke, 2009). In the context of parametric regression, rn(β0) d→ χ 2 p as n → ∞. (13) This can be viewed as a nonparametric version of Wilks’ theorem, and it is quite remarkable for the empirical likelihood to achieve such a property under a nonparametric setting with much less parametric distributional assumptions. We call this analogue of sharing the Wilks’ theorem the first order analogue between the parametric and the empirical likelihood. To appreciate why the nonparametric version of Wilks’ theorem is valid, we would like to present a few steps of derivation that offer some insights into the nonparametric likelihood. Typically, the first step in a study on EL is considering an expansion for λ defined in (7) at β0, the true value of β, and determining its order of magnitude. It can be shown that for the current parametric regression, λ = Op(n −1/2 ). (14) Such a rate for λ is obtained in the original papers of Owen (1988, 1990) for the mean parameter (which can be treated as a trivial case of regression without covariates), in Owen (1991) for linear regression, and in Qin and Lawless (1994) and Molanes L´opez, Van Keilegom and Veraverbeke (2009) for the more general case of estimating equations