Chapter 11: M-Estimators and Z-Estimators (Part I)

In this chapter, we study the consistency and asymptotic normality of M-estimators (introduced by Peter J. Huber) and Z-estimators.

Maximum likelihood estimators (MLEs) and moment estimators (ME) can be viewed as special cases of M-estimators and Z-estimators, respectively.

• Suppose the parameter (or “functional”) \(\theta\) of interest is attached to the distribution of the observations.

• The sequence of observations follows the distribution:

\[ X_n := (X_1, \dots, X_n) \sim p_\theta(X) \]

---

1. Basic Definitions and Examples

1.1 M-Estimators

Definition 12.1 (M-estimator)

The goal of an M-estimator is to find an estimator \(\hat{\theta}_n := \hat{\theta}_n(X_1, \dots, X_n)\) that maximizes some type of random (empirical) criterion function:

\[ \theta \mapsto M_n(\theta) \]

For example, the usual form is the sample average:

\[ M_n(\theta) = \frac{1}{n} \sum_{i=1}^n m_\theta(X_i) \]

1.2 Z-Estimators

Often, the maximization problem can be solved by setting the derivative (or gradient) equal to zero. This naturally leads to the concept of Z-estimators.

Definition 12.2 (Z-estimator)

A Z-estimator satisfies the following estimating equations:

\[ \Psi_n(\theta) = \frac{1}{n} \sum_{i=1}^n \psi_\theta(X_i) = 0 \]

More generally, a Z-estimator is an estimator \(\hat{\theta}_n\) that is a root of the above estimating equation.

Some remarks on M- and Z-estimators:

• M- and Z-estimators do not require the observations to be independent and identically distributed (IID) or to have an independent structure.

• The minimization problem of the objective function \(-M_n(\theta)\) may be non-convex.

• In practice, Z-estimators are usually solved by numerical optimization algorithms, such as (quasi-)Newton methods, gradient descent, stochastic gradient descent, etc. (especially when facing non-convex problems).

---

1.3 Classical Examples

Example 1: Maximum Likelihood Estimation and Pseudo-MLE

• MLE: Let \(X_1, \dots, X_n \sim p_\theta\). The maximum likelihood estimator maximizes the likelihood function \(\prod_{i=1}^n p_\theta(X_i)\), which is equivalent to maximizing the log-likelihood function:

\[ M_n(\theta) := \frac{1}{n} \sum_{i=1}^n \log p_\theta(X_i) \]

• Pseudo-MLE: In this case, the \(X_i\) may be dependent, but we still borrow and maximize the above log-likelihood criterion function.

Example 2: Location Estimators

The sample mean and sample median can both be viewed as Z-estimators; they respectively satisfy the following estimating equations:

• Sample mean:

\[ \Psi_n(\theta) = \frac{1}{n} \sum_{i=1}^n (X_i - \theta) = 0 \]

• Sample median:

\[ \Psi_n(\theta) = \frac{1}{n} \sum_{i=1}^n \text{sign}(X_i - \theta) = 0 \]

Example 3: Quantiles

For a distribution function \(F\), the quantile function is defined as:

\[ F^{-1}(p) := \inf\{x \in \mathbb{R} : F(x) \ge p\} \]

• Median: \(\text{med}(X) = F^{-1}(0.5)\)

• General quantile: The parameter can be obtained by minimizing an expected loss, i.e., \(\theta_0 = \arg\min_\theta E \rho_\tau(X - \theta) = F^{-1}(\tau)\).

Introduce the check function as the loss function:

\[ \rho_\tau(y) = y(\tau - I_{(y<0)}) \]

Thus, the \(\tau\)-sample quantile \(\hat{\theta}\) can be viewed both as an M-estimator and as a Z-estimator:

• As an M-estimator criterion function:

\[ M_n(\theta) := \frac{1}{n} \sum_{i=1}^n \rho_\tau(X_i - \theta) \]

• As a Z-estimator estimating equation:

\[ \Psi_n(\theta) := \frac{1}{n} \sum_{i=1}^n \left( (1-\tau) 1\{X_i < \theta\} - \tau 1\{X_i > \theta\} \right) = 0 \]

For small sample size \(n\), van der Vaart's book gives an alternative definition of the quantile \(\hat{\theta}\) by solving the inequality: \(-1 < n\Psi_n(\theta) < 1\).

Example 4: Huber Estimator

The Huber estimator originates from robust statistics; it aims to control the influence of extreme outliers on the estimation result. It is based on the Huber \(\psi\) function:

\[ \psi(x) = [x]_{-k}^k := \begin{cases} -k & \text{if } x \le -k \\ x & \text{if } |x| \le k \\ k & \text{if } x \ge k \end{cases} \]

The Huber estimator solves the following estimating equation:

\[ \Psi_n(\theta) = \frac{1}{n} \sum_{i=1}^n \psi(X_i - \theta) = 0 \]

When \(k\) is large, the behavior of the Huber estimator is closer to the non-robust sample mean; when \(k\) is small, its behavior is closer to the highly robust sample median. Therefore, it achieves an excellent balance between the mean and the median.

(Note: The empirical Z-estimator equation image often appears step-like (e.g., the empirical equation for quantiles), while its population expectation equation is a smooth curve.)

---

2. Consistency of M-Estimators

For an estimator \(\hat{\theta}_n\), we wish to prove that as the sample size tends to infinity, it converges in probability to the true parameter: \(\hat{\theta}_n \xrightarrow{p} \theta_0\), where \(\theta, \theta_0 \in \Theta\), and \(\Theta\) is equipped with a distance metric \(d\).

Assume \(\hat{\theta}_n\) maximizes the random (empirical) criterion function:

\[ \hat{\theta}_n = \arg\max_{\theta \in \Theta} M_n(\theta) = \arg\min_{\theta \in \Theta} -M_n(\theta) \]

where \(-M_n(\theta) =: L(P_n, P)\) can be viewed as the empirical loss function.

Definition 12.3 (True parameter)

The true parameter \(\theta_0\) is usually defined as the maximizer of the deterministic (true) criterion function \(M(\theta) =: E m_\theta(X)\):

\[ \theta_0 = \arg\max_{\theta \in \Theta} E m_\theta(X) = \arg\min_{\theta \in \Theta} E [-m_\theta(X)] \]

By the Law of Large Numbers (LLN), for any given \(\theta\), we have \(M_n(\theta) \xrightarrow{p} M(\theta)\). However, it is important to note that this pointwise convergence is usually not uniform over the entire parameter space \(\Theta\)!

To prove consistency, given an arbitrary random function \(\theta \mapsto M_n(\theta)\), we consider a sequence of estimators \(\{\hat{\theta}_n\}\) that satisfies the following nearly maximization condition:

\[ M_n(\hat{\theta}_n) \ge \sup_{\theta \in \Theta} M_n(\theta) - o_P(1) \ge M_n(\theta_0) - o_P(1) \]

(For example, this condition holds when \(-M_n(\theta)\) is strongly convex, i.e., if and only if the second derivative \(\ddot{M}_n(\theta) \ge O(1)I_p > 0, \forall \theta \in \Theta\).)

2.1 Basic Theorem for Consistency of M-Estimators

Theorem 12.1 (Consistency of M-Estimators)

Let \(M_n\) be a sequence of random functions and \(M\) a fixed deterministic function. If for every \(\epsilon > 0\) the following three conditions hold:

• C1. Uniform convergence:

\[ \sup_{\theta \in \Theta} |M_n(\theta) - M(\theta)| \xrightarrow{p} 0 \]

• C2. Well-separation / Identifiability:

\[ \sup_{\theta: d(\theta, \theta_0) \ge \epsilon} M(\theta) < M(\theta_0) \]

• C3. Nearly maximization: The sequence \(\{\hat{\theta}_n\}\) satisfies the aforementioned nearly maximization condition.

Then we must have \(\hat{\theta}_n \xrightarrow{p} \theta_0\).

Proof of Theorem 12.1 (click to expand)

By the well-separation assumption C2, for any \(\epsilon > 0\) there exists \(\eta > 0\) such that:

\[ M(\theta) < M(\theta_0) - \eta \quad \text{for all } \theta \text{ with } d(\theta, \theta_0) \ge \epsilon. \]

Replacing \(\theta\) by \(\hat{\theta}_n\), we obtain the set inclusion: \(\{d(\hat{\theta}_n, \theta_0) \ge \epsilon\} \subseteq \{M(\hat{\theta}_n) < M(\theta_0) - \eta\}\).

Therefore, in probability:

\[ P\{d(\hat{\theta}_n, \theta_0) \ge \epsilon\} \le P\{M(\theta_0) - M(\hat{\theta}_n) > \eta\} \quad \text{--- (12.2)} \]

Next, we show that the right-hand side of (12.2) tends to 0. Combine conditions C3 and C1. From C3 (nearly maximization):

\[ M_n(\hat{\theta}_n) \ge M_n(\theta_0) - o_P(1) = M(\theta_0) - o_P(1) \quad \text{--- (12.3)} \]

Note that the last equality in (12.3) holds because C1 implies pointwise convergence in probability: \(M_n(\theta_0) \xrightarrow{p} M(\theta_0)\).

Further using (12.3) and C1:

\[ M(\theta_0) - M(\hat{\theta}_n) \le M_n(\hat{\theta}_n) - M(\hat{\theta}_n) + o_P(1) \le \sup_{\theta \in \Theta} |M_n(\theta) - M(\theta)| + o_P(1) \]

By the uniform convergence C1, the supremum term above is \(o_p(1)\). Hence, overall \(M(\theta_0) - M(\hat{\theta}_n) = o_P(1)\).

Letting \(n \rightarrow \infty\) and substituting into (12.2), this implies the limiting probability is 0, and thus:

\[ d(\hat{\theta}_n, \theta_0) \xrightarrow{p} 0 \]

The proof is complete. \(\square\)

(Note: If the objective function \(M(\theta)\) is very flat near the maximum, or if there exist local maxima arbitrarily close to the global maximum, then the well-separation condition C2 will not hold.)

2.2 A Corollary under Modified Conditions

Corollary 12.2 (Consistency of M-Estimators under Modified Conditions)

Assume the following conditions hold:

• (i) The uniform convergence condition C1 of Theorem 12.1;

• (ii) Unique maximization: \(M(\theta) := E m_\theta(X)\) attains a unique maximum at \(\theta_0\);

• (iii) Compactness: The parameter space \(\Theta\) is compact;

• (iv) Continuity: The deterministic function \(M(\theta)\) is continuous.

Then, for any estimator \(\hat{\theta}_n\) satisfying the nearly maximization condition \((v) \ M_n(\hat{\theta}_n) \ge M_n(\theta_0) - o_P(1)\), we have \(\hat{\theta}_n \xrightarrow{p} \theta_0\).

Proof of Corollary 12.2 (click to expand)

For any given \(\delta > 0\), define the exterior set of the neighborhood \(B_\delta^c(\theta_0) := \{\theta : d(\theta, \theta_0) \ge \delta\}\). By conditions (ii)-(iv) (a continuous function attains its maximum on a compact set), there exists a point \(\theta^* \in \Theta \cap B_\delta^c(\theta_0)\) such that:

\[ \sup_{\theta \in \Theta \cap B_\delta^c(\theta_0)} M(\theta) = M(\theta^*) < M(\theta_0) \]

Next we show that for any \(\epsilon > 0\), when \(n\) is sufficiently large, the following chain of inequalities holds with probability approaching 1 (wpal):

\[ M(\hat{\theta}_n) > M_n(\hat{\theta}_n) - \epsilon/3 > M_n(\theta_0) - 2\epsilon/3 > M(\theta_0) - \epsilon \quad \text{--- (12.4)} \]

We analyze each inequality in (12.4) one by one:

First ">" (based on C1): Define the event \(E_1 := \{M(\hat{\theta}_n) \le M_n(\hat{\theta}_n) - \epsilon/3\}\). Since \(\sup_{\theta \in \Theta} |M_n(\theta) - M(\theta)| \xrightarrow{p} 0\), we must have \(P(E_1) = o(1)\).

Second ">" (based on condition v): Define the event \(E_2 := \{M_n(\hat{\theta}_n) \le M_n(\theta_0) - \epsilon/3\}\). Because nearly maximization requires \(M_n(\hat{\theta}_n) \ge M_n(\theta_0) - o_P(1)\), we have \(P(E_2) = o(1)\).

Third ">" (based on C1): Similar to the first term, define \(E_3 := \{M_n(\theta_0) - 2\epsilon/3 \le M(\theta_0) - \epsilon\} = \{M_n(\theta_0) \le M(\theta_0) - \epsilon/3\}\). Again by pointwise convergence, \(P(E_3) = o(1)\).

Now set \(\epsilon = M(\theta_0) - M(\theta^*) > 0\). Substituting this \(\epsilon\) into the head and tail of the inequality chain yields that with probability approaching 1:

\[ M(\hat{\theta}_n) > M(\theta^*) \quad \text{wpal} \]

More precisely:

\[ P(E_1^c \cap E_2^c \cap E_3^c) \ge 1 - \sum_{i=1}^3 P(E_i) \rightarrow 1 \]

Note that because \(\theta^*\) is the maximizer on the boundary region \(\Theta \cap B_\delta^c(\theta_0)\), if \(M(\hat{\theta}_n) > M(\theta^*)\) then it must be that \(\hat{\theta}_n\) lies inside the central neighborhood:

\[ \{M(\hat{\theta}_n) > M(\theta^*)\} \subseteq \{d(\hat{\theta}_n, \theta_0) < \delta\} \]

Hence, letting \(n \rightarrow \infty\):

\[ 1 \leftarrow P\{M(\hat{\theta}_n) > M(\theta^*)\} \le P\{d(\hat{\theta}_n, \theta_0) < \delta\} \le 1 \]

This proves \(d(\hat{\theta}_n, \theta_0) \xrightarrow{p} 0\). \(\square\)

---

3. Consistency of Z-Estimators

Since Z-estimators are solved by setting the objective function equal to zero (rather than maximizing), their consistency theorem is a natural generalization of that for M-estimators.

Theorem 12.3 (Consistency of Z-Estimators)

Let \(\Psi_n\) be a sequence of random vector-valued functions and \(\Psi\) a fixed vector-valued function. If for every \(\epsilon > 0\) the following conditions hold:

• C1. Uniform convergence**:

\[ \sup_{\theta \in \Theta} \|\Psi_n(\theta) - \Psi(\theta)\| \xrightarrow{p} 0 \]

• C2. Well-separation (Identifiability)**:

\[ \inf_{\theta: d(\theta, \theta_0) \ge \epsilon} \|\Psi(\theta)\| > 0 = \|\Psi(\theta_0)\| \]

• C3. Nearly zero condition**: The sequence of estimators \(\{\hat{\theta}_n\}\) satisfies \(\Psi_n(\hat{\theta}_n) = o_P(1)\).

Then we must have \(\hat{\theta}_n \xrightarrow{p} \theta_0\).

Proof sketch: This theorem follows directly from the consistency theorem for M-estimators. Simply construct the criterion functions \(M_n(\theta) = -\|\Psi_n(\theta)\|\) and \(M(\theta) = -\|\Psi(\theta)\|\). It can be seen that the nearly zero condition in Z-estimators corresponds exactly to the nearly maximization condition in M-estimators.

3.1 Weakening the Uniform Convergence Condition

In practice, the uniform convergence \(C1^*\) required in Theorem 12.3 is often too stringent and difficult to verify (it usually requires empirical process theory to establish rigorous guarantees). Lemma 12.4 provides a way to prove consistency without requiring uniform convergence.

Lemma 12.4 (Consistency via Monotonicity/Continuity, vdv p47)

Let \(\Theta\) be a subset of the real line, \(\Psi_n\) a random function, \(\Psi\) a fixed function, and for each given \(\theta\), we have pointwise convergence: \(\Psi_n(\theta) \xrightarrow{p} \Psi(\theta)\).

Assume one of the following conditions holds:

• (a1) Continuous and unique root: Each mapping \(\theta \mapsto \Psi_n(\theta)\) is continuous and has exactly one zero \(\hat{\theta}_n\); or
• (a2) Monotone and bounded: The mapping is non-decreasing (or non-increasing), and \(\Psi_n(\hat{\theta}_n) = o_P(1)\);

And assume:

• (b) Sign alternation of the true root: Let \(\theta_0\) satisfy that for every \(\epsilon > 0\), \(\Psi(\theta_0 - \epsilon) < 0 < \Psi(\theta_0 + \epsilon)\).

Then \(\hat{\theta}_n \xrightarrow{p} \theta_0\).

Proof of Lemma 12.4 (click to expand)

Case 1: Continuous and unique root (when a1 holds)

If the mapping \(\theta \mapsto \Psi_n(\theta)\) is continuous and has a unique zero at \(\hat{\theta}_n\), then the event \(\{\Psi_n(\theta_0 - \epsilon) < 0 \text{ and } \Psi_n(\theta_0 + \epsilon) > 0\}\) must imply that the zero lies between them, i.e., \(\{\theta_0 - \epsilon < \hat{\theta}_n < \theta_0 + \epsilon\}\).

In probability terms, this is expressed as:

\[ P(\Psi_n(\theta_0 - \epsilon) < 0, \Psi_n(\theta_0 + \epsilon) > 0) \le P(\theta_0 - \epsilon < \hat{\theta}_n < \theta_0 + \epsilon) \]

Since for the given \(\theta_0 \pm \epsilon\) we have pointwise convergence in probability \(\Psi_n(\theta_0 \pm \epsilon) \xrightarrow{p} \Psi(\theta_0 \pm \epsilon)\), and by condition (b), the probability on the left tends to 1 as \(n \rightarrow \infty\). Therefore, the probability on the right must also tend to 1, which proves the consistency of \(\hat{\theta}_n\).

Case 2: Non-decreasing and nearly zero (when a2 holds)

Suppose the mapping \(\theta \mapsto \Psi_n(\theta)\) is non-decreasing. First note:

\[ \{\Psi_n(\theta_0 - \epsilon) < -\eta\} = \{\Psi_n(\theta_0 - \epsilon) < -\eta, \hat{\theta}_n \le \theta_0 - \epsilon\} \cup \{\Psi_n(\theta_0 - \epsilon) < -\eta, \hat{\theta}_n \ge \theta_0 - \epsilon\} \]

Due to monotonicity of \(\Psi_n\), if \(\hat{\theta}_n \ge \theta_0 - \epsilon\), then \(\Psi_n(\hat{\theta}_n) \ge \Psi_n(\theta_0 - \epsilon) < -\eta\). Let \(E_- := \{\Psi_n(\hat{\theta}_n) < -\eta\}\). By the condition \(\Psi_n(\hat{\theta}_n) = o_P(1)\), we have \(P(E_-) = o(1)\) for any \(\eta > 0\). Therefore, the above union can be bounded as:

\[ \subseteq E_- \cup \{\hat{\theta}_n \ge \theta_0 - \epsilon\} \]

Similarly, we can define the event for the right tail \(E_+ := \{\Psi_n(\hat{\theta}_n) > \eta\}\), and also \(P(E_+) = o(1)\).

Hence, for any given \(\epsilon, \eta > 0\):

\[ P(\Psi_n(\theta_0 - \epsilon) < -\eta, \Psi_n(\theta_0 + \epsilon) > \eta) \le P(\theta_0 - \epsilon < \hat{\theta}_n < \theta_0 + \epsilon) + P(E_-) + P(E_+) \]

Rearranging:

\[ P(\Psi_n(\theta_0 - \epsilon) < -\eta, \Psi_n(\theta_0 + \epsilon) > \eta) = P(\theta_0 - \epsilon < \hat{\theta}_n < \theta_0 + \epsilon) + o(1) \]

Choosing an appropriate threshold, let \(2\eta = \min\{-\Psi(\theta_0 - \epsilon), \Psi(\theta_0 + \epsilon)\} > 0\). By pointwise convergence, the probability on the left tends to 1, so the probability on the right also tends to 1, establishing consistency. \(\square\)

3.2 Example: Applying Lemma 12.4 to Prove Consistency of the Median

Sample median \(\hat{\theta}_n\) can be viewed as a zero of the equation \(\theta \mapsto \Psi_n(\theta) = n^{-1} \sum_{i=1}^n \text{sign}(X_i - \theta)\).

By the Law of Large Numbers (LLN), for each fixed \(\theta\), we have:

\[ \Psi_n(\theta) \xrightarrow{p} \Psi(\theta) = E[\text{sign}(X - \theta)] = P(X > \theta) - P(X < \theta) \]

Now we verify the conditions of Lemma 12.4:

• Check condition (a): The function \(\theta \mapsto \Psi_n(\theta)\) is clearly non-increasing (the lemma symmetrically applies to the non-increasing case by reversing the sign).

• Check condition (b): We need to show \(-\Psi(\theta_0 - \epsilon) < 0 < -\Psi(\theta_0 + \epsilon)\). Examine the value of the limit function near \(\theta_0\):

\[ \Psi(\theta_0 - \epsilon) = P(X > \theta_0 - \epsilon) - P(X < \theta_0 - \epsilon) = 1 - 2P(X < \theta_0 - \epsilon) \]

At the population median \(\theta_0\), we have:

\[ \Psi(\theta_0) = P(X > \theta_0) - P(X < \theta_0) = 0 \implies P(X > \theta_0) = P(X < \theta_0) = 0.5 \]

Similarly:

\[ \Psi(\theta_0 + \epsilon) = P(X > \theta_0 + \epsilon) - P(X < \theta_0 + \epsilon) = 1 - 2P(X < \theta_0 + \epsilon) \]

If the distribution of \(X\) is continuous and the population median is uniquely determined, then we must have:

\[ P(X < \theta_0 - \epsilon) < 0.5 < P(X < \theta_0 + \epsilon) \quad \text{for all } \epsilon > 0. \]

Therefore, both conditions (a2) and (b) are satisfied. Applying Lemma 12.4, we directly conclude that the sample median converges consistently to the population median, i.e., \(\hat{\theta}_n \xrightarrow{p} \theta_0\). This avoids the need for complex empirical process theory.

4. Wald Consistency of M-Estimators

4.1 Semi-continuity

Semi-continuity is a topological property weaker than continuity.

• Upper semi-continuous (u.s.c.): A real-valued function \(f\) is said to be upper semi-continuous at \(x_0\) if:

\[ \limsup_{x \rightarrow x_0} f(x) \le f(x_0) \]

• Lower semi-continuous (l.s.c.): A real-valued function \(f\) is said to be lower semi-continuous at \(x_0\) if \(-f\) is upper semi-continuous, i.e.,

\[ \liminf_{x \rightarrow x_0} f(x) \ge f(x_0) \]

In statistical inference, upper semi-continuous M-functions are often used to establish Wald consistency conditions.

---

4.2 Wald Consistency Conditions

Let \(P m_\theta := E m_\theta(X)\). Typically, the map \(\theta \mapsto P m_\theta\) has a unique global maximum at \(\theta_0\). However, in the framework of Wald consistency, we allow multiple global maxima:

\[ \Theta_0 := \{\theta_0 \in \Theta : P m_{\theta_0} = \sup_\theta P m_\theta\} \ne \emptyset \]

Here, \(P m\) attains its maximum on the set \(\Theta_0\).

Theorem 12.4 (Wald Consistency of M-Estimators)

For every compact set \(K \subset \Theta\), the Wald consistency conditions ensuring \(P(d(\hat{\theta}_n, \Theta_0) \ge \epsilon, \hat{\theta}_n \in K) \rightarrow 0\) are:

• W1. Upper semi-continuity condition: The map \(\theta \mapsto m_\theta(x)\) is upper semi-continuous for almost every \(x\).

• W2. Uniform bound on small balls: For any small ball \(U \subset \Theta\), assume \(x \mapsto \sup_{\theta \in U} m_\theta(x)\) is measurable and satisfies the following dominated convergence (W-condition):

\[ E \sup_{\theta \in U} m_\theta(X) < \infty \]

• W3. Approximate maximization on compact sets: The sequence \(\{\hat{\theta}_n\}\) satisfies:

\[ M_n(\hat{\theta}_n) \ge M_n(\theta_0) - o_P(1) \]

for some \(\theta_0 \in \Theta_0\).

Proof of Theorem 12.4 (click to expand)

Step 1: Construct a cover

Let \(B = \{\theta \in K : d(\theta, \Theta_0) \ge \epsilon\}\). Our goal is to show:

\[ P\{\hat{\theta}_n \in B\} \rightarrow 0 \]

If the function \(\theta \mapsto P m_\theta\) is identically \(-\infty\), then \(\Theta_0 = \Theta\) holds trivially. Hence, we may assume there exists \(\theta_0 \in \Theta_0\) such that \(E m_{\theta_0} > -\infty\). By condition W2, \(E|m_{\theta_0}(X)| < \infty\).

Fix some \(\theta \in K\), and let \(U_l \ni \theta\) be a decreasing sequence of open balls containing \(\theta\) with diameters converging to 0. Write \(m_U(x) := \sup_{\theta \in U} m_\theta(x)\). Clearly, for each \(l\), the sequence:

\[ m_\theta(x) \le m_{U_l}(x) = \sup_{\theta \in U_l} m_\theta(x) \]

is decreasing in \(l\). Taking the limit gives:

\[ m_\theta(x) \le \lim_{U_l \rightarrow \theta} m_{U_l}(x) \le m_\theta(x) \quad a.s. \]

where the last inequality follows from the upper semi-continuity of \(m_\theta(x)\) (condition W1).

Step 2: Apply the monotone convergence theorem

Since \(m_{U_l} \downarrow m_\theta\) a.e., applying the monotone convergence theorem to \(\{-m_{U_l}\}\) yields:

\[ E m_{U_l}(X) \rightarrow E m_\theta(X) \]

Thus, for all \(\theta \notin \Theta_0\), because \(\theta_0\) maximizes \(E m_\theta(X)\), we must have:

\[ E m_{U_l}(X) \rightarrow E m_\theta(X) < E m_{\theta_0}(X) \]

Therefore, there exists a sufficiently small open ball \(U_k\) covering \(\theta\) such that:

\[ E m_{U_k}(X) < E m_{\theta_0}(X), \quad \forall \theta \notin \Theta_0, \exists k \in \mathbb{N} \]

By compactness of \(K\), the compact set \(B\) can be covered by finitely many such balls \(U_{\theta_1}, \dots, U_{\theta_p}\). Combining with the Law of Large Numbers (LLN), we have:

\[ \sup_{\theta \in B} P_n m_\theta \le \max_{j=1,\dots,p} P_n m_{\theta_j} \le \max_{j=1,\dots,p} P_n \sup_{\theta \in U_{\theta_j}} m_\theta \]

\[ \xrightarrow{a.s.} \max_{j=1,\dots,p} E \sup_{\theta \in U_{\theta_j}} m_\theta(X) < E m_{\theta_0}(X) \]

Step 3: Probability measure limiting bound

On the other hand, by the approximate maximization assumption, the event \(\{\hat{\theta}_n \in B\}\) implies the following event:

\[ \sup_{\theta \in B} P_n m_\theta \ge P_n m_{\hat{\theta}_n} \ge P_n m_{\theta_0} - o_P(1) = E m_{\theta_0}(X) - o_P(1) \]

where the equality follows from applying the law of large numbers to the single point \(P_n m_{\theta_0}\). Therefore, we have the event inclusion:

\[ \{\hat{\theta}_n \in B\} \subseteq \left\{ \sup_{\theta \in B} P_n m_\theta \ge E m_{\theta_0}(X) - o_P(1) \right\} \]

Consequently,

\[ P\{\hat{\theta}_n \in B\} \le P\left\{ \sup_{\theta \in B} P_n m_\theta \ge E m_{\theta_0}(X) - o_P(1) \right\} \rightarrow 0 \]

The proof is complete. \(\square\)

---

4.3 Example: Location MLE for the Cauchy Distribution

Consider the Cauchy distribution \(\text{Cauchy}(\theta)\), with probability density function (pdf):

\[ f_\theta(x) = \frac{1}{\pi \{1 + (x - \theta)^2\}} \]

The maximum likelihood estimator (MLE) for the location parameter \(\theta\) based on i.i.d. samples \(X_1, \dots, X_n \sim \text{Cauchy}(\theta)\) maximizes the log-likelihood function \(\theta \mapsto P_n m_\theta\):

\[ m_\theta(x) = -\log(1 + (x - \theta)^2) \]

Since the original parameter space \(\mathbb{R}\) is not compact, to apply Wald's theorem, we can extend the parameter space to its compactification:

\[ \overline{\mathbb{R}} := \mathbb{R} \cup \{\pm\infty\} \]

Now we verify the Wald consistency conditions one by one:

• W1. Upper semi-continuity: Since \(m_\theta(x)\) is continuous on the entire extended space, the upper semi-continuity condition holds trivially.

• W2. Uniform bound on small balls: For any open ball \(U\), we have:

\[ E \sup_{\theta \in U} m_\theta(X) = \int \sup_{\theta \in U} \frac{-\log\{1 + (x - \theta)^2\}}{\pi \{1 + (x - \theta)^2\}} dx < \infty \]

and the limit as \(\theta \rightarrow \pm\infty\) behaves well:

\[ m_{-\infty}(x) = \lim_{\theta \rightarrow -\infty} \sup m_\theta(x) = -\infty \]

\[ m_{\infty}(x) = \lim_{\theta \rightarrow \infty} \sup m_\theta(x) = -\infty \]

• W3. Approximate maximization on compact sets: This holds by definition of the estimator.

We equip \(\Theta = \overline{\mathbb{R}}\) with the following distance metric:

\[ d(\theta_1, \theta_2) = |\arctan \theta_1 - \arctan \theta_2| \]

By the identifiability condition of maximum likelihood, the set of maximizers is a singleton \(\Theta_0 = \{\theta_0\}\). Taking the compact set \(K = \overline{\mathbb{R}}\) and applying Wald's theorem directly yields \(d(\hat{\theta}_n, \Theta_0) \xrightarrow{p} 0\), i.e., the Cauchy MLE is consistent.

---

5. Asymptotic Normality of Z-Estimators

5.1 Theorem on Asymptotic Normality

Theorem 12.5 (Asymptotic Normality of Z-Estimators)

Assume the following standard conditions hold:

• (C1) Measurability: For each \(\theta\) in the parameter space (an open subset of Euclidean space), the map \(x \mapsto \psi_\theta(x)\) is a measurable vector-valued function.

• (C2) Local Lipschitz condition: For any \(\theta_1\) and \(\theta_2\) in a neighborhood of the true parameter \(\theta_0\), there exists a measurable envelope function \(\dot{\psi}(x)\) with \(P[\dot{\psi}(X)]^2 < \infty\) such that:

\[ \|\psi_{\theta_1}(x) - \psi_{\theta_2}(x)\| \le \dot{\psi}(x) \|\theta_1 - \theta_2\| \]

• (C3) Moment and differentiability conditions: Assume \(P\|\psi_{\theta_0}\|^2 < \infty\), and the map \(\theta \mapsto P \psi_\theta\) is differentiable at the zero \(\theta_0\), with non-singular derivative matrix \(V_{\theta_0}\):

\[ V_{\theta_0} := \left. \frac{\partial}{\partial \theta} P[\psi_\theta(X)] \right|_{\theta = \theta_0} \]

• (C4) Consistency: The estimator satisfies \(P_n \psi_{\hat{\theta}_n} = o_P(n^{-1/2})\) and \(\hat{\theta}_n \xrightarrow{p} \theta_0\).

Then the estimator satisfies asymptotic normality:

\[ \sqrt{n}(\hat{\theta}_n - \theta_0) = -V_{\theta_0}^{-1} \frac{1}{\sqrt{n}} \sum_{i=1}^n \psi_{\theta_0}(X_i) + o_P(1) \xrightarrow{d} N\left(0, V_{\theta_0}^{-1} P \psi_{\theta_0} \psi_{\theta_0}^T (V_{\theta_0}^{-1})^T\right) \]

---

5.2 Empirical Processes and Donsker Classes

To rigorously prove the asymptotic normality of M-estimators and Z-estimators, we need tools from empirical process theory.

• Basic setup: Let \(X_1, \dots, X_n\) be random samples from an unknown distribution \(P\) on a measurable space \((\mathcal{X}, \mathcal{A})\). Define the empirical distribution measure:

\[ P_n = n^{-1} \sum_{i=1}^n \delta_{X_i} \]

• Operator notation: For a measurable function \(f: \mathcal{X} \mapsto \mathbb{R}\), we denote the expectation under \(P_n\) and \(P\) respectively as:

\[ P_n f = \frac{1}{n} \sum_{i=1}^n f(X_i), \quad P f = \int f dP \]

• Empirical process linear functional: Define the empirical process acting on a function \(f\) as:

\[ \mathbb{G}_n f = \sqrt{n}(P_n f - P f) = \sqrt{n}\left( \frac{1}{n} \sum_{i=1}^n f(X_i) - E f(X) \right) \]

Definition 12.4 (Donsker Class of Functions)

A measurable function class \(\mathcal{F}\) is called a P-Donsker class if, for all \(f \in \mathcal{F}\), the sequence \(\mathbb{G}_n f\) converges weakly in \(\ell^\infty(\mathcal{F})\) (the space of bounded linear functionals) to a tight limiting Gaussian process.

• Robust Lipschitz Donsker criterion: Let \(\mathcal{F} = \{f_\theta : \theta \in \Theta\}\) be a function class indexed by a bounded subset \(\Theta \subset \mathbb{R}^d\). If there exists a measurable function \(m(x)\) such that \(f_\theta(x)\) satisfies a global Lipschitz condition:

\[ |f_{\theta_1}(x) - f_{\theta_2}(x)| \le m(x) \|\theta_1 - \theta_2\|, \quad \forall \theta_1, \theta_2 \in \Theta \]

and if \(P|m|^r < \infty\) for some \(r > 0\), then \(\mathcal{F}\) is P-Donsker.

---

5.3 Key Random Function Lemma

Lemma 12.6 (van der Vaart Lemma 19.24)

Assume the following conditions hold:

• (a) \(\mathcal{F}\) is a P-Donsker measurable function class.

• (b) \(\{\hat{f}_n\}\) is a sequence of random functions taking values in \(\mathcal{F}\) and converging in \(L_2(P)\) to some deterministic function \(f_0\):

\[ \int (\hat{f}_n(x) - f_0(x))^2 dP(x) = P(\hat{f}_n - f_0)^2 \xrightarrow{p} 0 \]

Then:

\[ \mathbb{G}_n(\hat{f}_n - f_0) \xrightarrow{p} 0 \quad \text{and} \quad \mathbb{G}_n \hat{f}_n \xrightarrow{d} \mathbb{G}_P f_0 \]

Extension of Lemma 12.6 to Z-Estimators (click to expand)

In the framework of the theorem, we set \(\hat{f}_n := \psi_{\hat{\theta}_n}\) and \(f_0 := \psi_{\theta_0}\). By the Lipschitz condition:

\[ P(\psi_{\hat{\theta}_n} - \psi_{\theta_0})^2 \le P(\dot{\psi})^2 \|\hat{\theta}_n - \theta_0\|^2 \rightarrow 0 \]

When \(\hat{\theta}_n \xrightarrow{p} \theta_0\) and the envelope moment condition \(P(\dot{\psi})^2 < \infty\) holds, the above condition is satisfied. Therefore, because \(\mathcal{F} = \{\psi_\theta : \theta \in \Theta\}\) is P-Donsker, Lemma 12.6 directly implies:

\[ \mathbb{G}_n(\hat{f}_n - f_0) = \sqrt{n}(P_n \psi_{\hat{\theta}_n} - P \psi_{\hat{\theta}_n}) - \sqrt{n}(P_n \psi_{\theta_0} - P \psi_{\theta_0}) \xrightarrow{p} 0 \]

Rearranging gives:

\[ \sqrt{n}(P_n \psi_{\hat{\theta}_n} - P \psi_{\hat{\theta}_n}) = \sqrt{n}(P_n \psi_{\theta_0} - P \psi_{\theta_0}) + o_P(1) \]

The proof is complete. \(\square\)

---

5.4 Complete Proof of Theorem 12.5

Asymptotic expansion proof of Theorem 12.5 (click to expand)

From the derivation in Section 5.3, we have established the asymptotic equivalence of the empirical process:

\[ \mathbb{G}_n \psi_{\hat{\theta}_n} - \mathbb{G}_n \psi_{\theta_0} = \mathbb{G}_n(\psi_{\hat{\theta}_n} - \psi_{\theta_0}) \xrightarrow{p} 0 \quad \text{--- (12.13)} \]

Meanwhile, expand the definition of the empirical process: \(\mathbb{G}_n \psi_{\hat{\theta}_n} = \sqrt{n}(P_n \psi_{\hat{\theta}_n} - P \psi_{\hat{\theta}_n})\). Since \(\theta_0\) is the true population zero satisfying \(P \psi_{\theta_0} = 0\), we can rewrite the above as:

\[ \mathbb{G}_n \psi_{\hat{\theta}_n} = \sqrt{n}(P_n \psi_{\hat{\theta}_n} - P \psi_{\hat{\theta}_n}) = \sqrt{n} P_n \psi_{\hat{\theta}_n} + \sqrt{n}(P \psi_{\theta_0} - P \psi_{\hat{\theta}_n}) \]

By the approximate zero condition (C4), \(\sqrt{n} P_n \psi_{\hat{\theta}_n} = o_P(1)\). Substituting this yields:

\[ \mathbb{G}_n \psi_{\hat{\theta}_n} = \sqrt{n}(P \psi_{\theta_0} - P \psi_{\hat{\theta}_n}) + o_P(1) \quad \text{--- (12.14)} \]

Combining (12.14) with (12.13), we can eliminate the empirical linear term at the estimator location:

\[ \mathbb{G}_n \psi_{\theta_0} = \mathbb{G}_n \psi_{\hat{\theta}_n} + o_P(1) = \sqrt{n}(P \psi_{\theta_0} - P \psi_{\hat{\theta}_n}) + o_P(1) \]

Next, perform a first-order Taylor expansion of the deterministic expectation function \(P \psi_\theta\) around \(\theta_0\):

\[ \frac{1}{\sqrt{n}} \sum_{i=1}^n \psi_{\theta_0}(X_i) + o_P(1) = -\sqrt{n} \left. \frac{\partial}{\partial \theta} P \psi_\theta(X) \right|_{\theta = \theta_0} (\hat{\theta}_n - \theta_0) + \sqrt{n} o_P(\|\hat{\theta}_n - \theta_0\|) \]

Substituting the matrix \(V_{\theta_0}\):

\[ \sqrt{n} V_{\theta_0}(\hat{\theta}_n - \theta_0) = -\mathbb{G}_n \psi_{\theta_0} + \sqrt{n} o_P(\|\hat{\theta}_n - \theta_0\|) \quad \text{--- (12.15)} \]

Because the central limit theorem (CLT) guarantees \(\mathbb{G}_n \psi_{\theta_0} = O_P(1)\), equation (12.15) implies stochastic boundedness: \(\sqrt{n} \|V_{\theta_0}(\hat{\theta}_n - \theta_0)\| = O_P(1)\). Using non-singularity of the matrix:

\[ \sqrt{n} \|\hat{\theta}_n - \theta_0\| \le \|V_{\theta_0}^{-1}\| \sqrt{n} \|V_{\theta_0}(\hat{\theta}_n - \theta_0)\| = O_P(1) + o_P(\sqrt{n} \|\hat{\theta}_n - \theta_0\|) \]

This proves the \(\sqrt{n}\)-consistency, i.e., \(\sqrt{n} \|\hat{\theta}_n - \theta_0\| = O_P(1)\).

Substituting this rate back into the remainder term in (12.15), the higher-order term becomes \(o_P(1)\):

\[ \sqrt{n}(\hat{\theta}_n - \theta_0) = -V_{\theta_0}^{-1} \frac{1}{\sqrt{n}} \sum_{i=1}^n \psi_{\theta_0}(X_i) + o_P(1) \]

Finally, applying the multivariate central limit theorem and Slutsky's theorem yields the asymptotic normality. \(\square\)

---

6. Asymptotic Normality of M-Estimators

Analogous to Z-estimators, we can establish asymptotic normality of M-estimators by imposing smoothness and differentiability conditions on the criterion function.

Theorem 12.6 (Asymptotic Normality of M-Estimators)

Assume the following standard conditions hold:

• Measurability: For each \(\theta\) in the open parameter space, the map \(x \mapsto m_\theta(x)\) is measurable, and \(\theta \mapsto m_\theta(x)\) is differentiable at the true point \(\theta_0\) for \(P\)-almost every \(x\), with first derivative (score function) denoted \(\dot{m}_{\theta_0}(x)\).

• Lipschitz condition: There exists a square-integrable control function \(\dot{m}(x)\) (\(P[\dot{m}(X)]^2 < \infty\)) such that in a neighborhood of \(\theta_0\):

\[ |m_{\theta_1}(x) - m_{\theta_2}(x)| \le \dot{m}(x) \|\theta_1 - \theta_2\| \]

• Moment and second-order differentiability: Assume the map \(\theta \mapsto P m_\theta\) admits the following second-order Taylor expansion at the global maximizer \(\theta_0\):

\[ P m_\theta = P m_{\theta_0} + \frac{1}{2}(\theta - \theta_0)^T V_{\theta_0}(\theta - \theta_0) + o(\|\theta - \theta_0\|^2) \]

where the symmetric Hessian matrix \(V_{\theta_0}\) is non-singular:

\[ V_{\theta_0} = \left. \frac{\partial^2}{\partial \theta^2} P[m_\theta(X)] \right|_{\theta = \theta_0} \]

• Consistency and approximate maximization rate: The estimator satisfies \(P_n m_{\hat{\theta}_n} \ge \sup_\theta P_n m_\theta - o_P(n^{-1})\) and \(\hat{\theta}_n \xrightarrow{p} \theta_0\).

Then the M-estimator satisfies asymptotic normality:

\[ \sqrt{n}(\hat{\theta}_n - \theta_0) = -V_{\theta_0}^{-1} \frac{1}{\sqrt{n}} \sum_{i=1}^n \dot{m}_{\theta_0}(X_i) + o_P(1) \xrightarrow{d} N\left(0, V_{\theta_0}^{-1} P \dot{m}_{\theta_0} \dot{m}_{\theta_0}^T V_{\theta_0}^{-1}\right) \]

---

7. Classical Application Examples

7.1 Asymptotic Distribution of the Sample Median (via M-Estimation Theory)

The sample median can be obtained by maximizing the following criterion function:

\[ \theta \mapsto -\frac{1}{n} \sum_{i=1}^n |X_i - \theta| \]

Assume the population distribution function \(F(x)\) is differentiable at the true population median \(\theta_0 = F^{-1}(1/2)\), and its density satisfies \(f(\theta_0) > 0\).

We use the centered M-estimation function: \(m_\theta(x) = |x - \theta| - |x|\). By the triangle inequality, for any \(\theta_1, \theta_2\):

\[ ||x - \theta_1| - |x - \theta_2|| \le |\theta_1 - \theta_2| \]

This shows the function satisfies a Lipschitz condition controlled by a constant, where \(\dot{m}(x) \equiv 1\), which obviously satisfies the square-integrability condition.

Moreover, this map is differentiable almost everywhere except at \(x = \theta\), with first derivative (score function):

\[ \dot{m}_{\theta_0}(x) = -\text{sign}(x - \theta_0) \]

Its squared expectation is:

\[ E(\dot{m}_{\theta_0}(X))^2 = E(\text{sign}^2(X - \theta_0)) = 1 \]

Using integration by parts, we can derive the analytic form of the deterministic objective function:

\[ P m_\theta = E(m_\theta(X)) = \int |x - \theta| dF(x) - \int |x| dF(x) \]

\[ = \theta F(0) + \int_{(0, \theta]} 2(\theta - x) dF(x) - \theta(1 - F(0)) = 2\int_0^\theta F(x) dx - \theta \]

Taking first and second derivatives of the objective function:

\[ \frac{d P m_\theta}{d \theta} = 2F(\theta) - 1, \quad \frac{d^2 P m_\theta}{d \theta^2} = 2f(\theta) \]

Therefore, the second-order matrix evaluated at \(\theta_0\) is \(V_{\theta_0} = 2f(\theta_0)\).

Since the asymptotic variance takes the sandwich form \(V_{\theta_0}^{-1} P[\dot{m}_{\theta_0} \dot{m}_{\theta_0}^T] V_{\theta_0}^{-1} = \frac{1}{(2f(\theta_0)) \cdot 1 \cdot (2f(\theta_0))} = \frac{1}{4f^2(\theta_0)}\), substituting into Theorem 12.6 directly yields the classic asymptotic normality result for the median:

\[ \sqrt{n}(\hat{\theta}_n - \theta_0) \xrightarrow{d} N\left(0, \frac{1}{4f^2(\theta_0)}\right) \]

---

7.2 Nonlinear Least Squares Estimation

Suppose observed samples \((X_1, Y_1), \dots, (X_n, Y_n)\) consist of covariates \(X\) and response \(Y\), with data generating mechanism:

\[ Y = f_{\theta_0}(X) + e, \quad E(e|X) = 0, \quad \text{Var}(e|X) = \sigma^2(X) < \infty \]

The nonlinear least squares estimator (LSE) minimizes the sum of squared residuals:

\[ \hat{\theta}_n = \arg\min_\theta \sum_{i=1}^n (Y_i - f_\theta(X_i))^2 \]

This is equivalent to a special M-estimator with loss function \(m_\theta(x, y) = -(y - f_\theta(x))^2\). The limit of the empirical objective function under the law of large numbers is:

\[ \theta \mapsto P m_\theta = -E(Y - f_\theta(X))^2 = -E[Y - f_{\theta_0}(X) + (f_{\theta_0}(X) - f_\theta(X))]^2 \]

\[ = -E(f_{\theta_0}(X) - f_\theta(X))^2 - E e^2 \quad \text{--- (12.16)} \]

From (12.16), if the model satisfies an identifiability condition (i.e., \(\theta \ne \theta_0 \implies P(f_\theta(X) \ne f_{\theta_0}(X)) > 0\)), then the true parameter \(\theta_0\) is indeed a global maximizer of the objective function, guaranteeing consistency of the estimator.

Now we verify the key Lipschitz condition for asymptotic normality:

\[ |m_{\theta_1}(x,y) - m_{\theta_2}(x,y)| = |(y - f_{\theta_1}(x))^2 - (y - f_{\theta_2}(x))^2| \]

\[ = |f_{\theta_1}(x) - f_{\theta_2}(x)| \cdot |2y - f_{\theta_1}(x) - f_{\theta_2}(x)| \]

Assume the regression function is well-behaved in first order: \(|f_{\theta_1}(x) - f_{\theta_2}(x)| \le g(x) \|\theta_2 - \theta_1\|\), and the function values are uniformly bounded \(|f_\theta(x)| \le c(x)\). Then we can construct a square-integrable global Lipschitz envelope:

\[ \dot{m}(x,y) := g(x)[2|y| + 2c(x)] \]

Perform a Taylor expansion of the objective function (12.16) to obtain the Hessian matrix \(V_{\theta_0}\):

\[ P m_\theta = P m_{\theta_0} - \int [f_\theta(x) - f_{\theta_0}(x)]^2 \rho(x) dx \]

\[ = P m_{\theta_0} - \int [(\theta - \theta_0)^T \dot{f}_{\theta_0}(x) + o(\|\theta - \theta_0\|)]^2 \rho(x) dx \]

\[ = P m_{\theta_0} - \frac{1}{2}(\theta - \theta_0)^T \left[ 2 \int \dot{f}_{\theta_0}(x) \dot{f}_{\theta_0}^T(x) p(x) dx \right] (\theta - \theta_0) + o(\|\theta - \theta_0\|^2) \]

Thus, the Hessian matrix and the corresponding first-order score function are:

\[ V_{\theta_0} = -2 P(\dot{f}_{\theta_0} \dot{f}_{\theta_0}^T), \quad \dot{m}_{\theta_0}(x,y) = 2(y - f_{\theta_0}(x)) \dot{f}_{\theta_0}(x) = 2 e \dot{f}_{\theta_0}(x) \]

Plugging into the asymptotic normality formula for M-estimators, under the assumption of independence between the error \(e\) and covariates \(X\), the middle sandwich matrix simplifies to:

\[ V_{\theta_0}^{-1} P(\dot{m}_{\theta_0} \dot{m}_{\theta_0}^T) V_{\theta_0}^{-1} = [2P \dot{f}_{\theta_0} \dot{f}_{\theta_0}^T]^{-1} \cdot 4 P(e^2 \dot{f}_{\theta_0} \dot{f}_{\theta_0}^T) \cdot [2P \dot{f}_{\theta_0} \dot{f}_{\theta_0}^T]^{-1} \]

\[ = [P \dot{f}_{\theta_0} \dot{f}_{\theta_0}^T]^{-1} E(\sigma^2(X)) [P \dot{f}_{\theta_0} \dot{f}_{\theta_0}^T]^{-1} \]

This determines the asymptotic distribution of the nonlinear least squares estimator.

---

7.3 Binary Regression Model

Suppose observed samples \((X_1, Y_1), \dots, (X_n, Y_n)\), where \(X_i\) is a \(k\)-dimensional covariate vector and \(Y_i \in \{0, 1\}\) is a binary response variable, with conditional probability model:

\[ P_\theta(Y_i = 1 | X_i = x) = \Psi(\theta^T x) \]

Here \(\Psi: \mathbb{R} \mapsto [0, 1]\) is a known continuously differentiable monotonic link function.

• Logistic Regression: Take \(\Psi(t) = \frac{1}{1 + e^{-t}}\).

• Probit Model: Take \(\Psi(t) = \Phi(t)\) (standard normal distribution function).

The maximum likelihood estimator for this model maximizes the conditional log-likelihood function:

\[ \theta \mapsto \prod_{i=1}^n p_\theta(Y_i | X_i) := \prod_{i=1}^n \Psi(\theta^T X_i)^{Y_i} (1 - \Psi(\theta^T X_i))^{1 - Y_i} \]

To ensure identifiability of the parameter, we require that the covariates are non-degenerate, i.e., the distribution of \(X_i\) does not concentrate on any \((k-1)\)-dimensional affine subspace of \(\mathbb{R}^k\). To this end, we assume the range of \(X_i\) is bounded and the matrix \(E X X^T\) is non-singular.

Differentiating the log-likelihood gives the corresponding Z-estimator score equation:

\[ \psi_\theta(x, y) := \dot{l}_\theta(y|x) = \frac{y - \Psi(\theta^T x)}{\Psi(\theta^T x)[1 - \Psi(\theta^T x)]} \Psi'(\theta^T x) x \]

When the parameter \(\theta\) varies over a compact set, this score function is uniformly bounded in \((x,y)\), so the conditions are naturally satisfied. Compute the corresponding Fisher information matrix:

\[ I_\theta = E \left[ \frac{\Psi'(\theta^T X)^2}{\Psi(\theta^T X)[1 - \Psi(\theta^T X)]} X X^T \right] \]

Applying the asymptotic normality theorem for Z-estimators, the maximum likelihood estimator converges to the following normal limiting distribution:

\[ \sqrt{n}(\hat{\theta}_n - \theta) \xrightarrow{d} N(0, I_\theta^{-1}) \]

💬 Comments