๐ Detailed Solutions to Homework Exercises: Large Sample Theory Assignment 3
Exercise 1: Second-Order Delta Method
Problem: Let \(X_1, ..., X_n \stackrel{i.i.d}{\sim} Bernoulli(\theta)\).
(a) When \(\theta = 1/2\), find the asymptotic distribution of \(\hat{\theta}_n = \overline{X}_n\).
(b) The Kullback-Leibler (KL) divergence from \(P_\theta\) to \(P_\eta\) is defined as:
where \(p_\theta, p_\eta\) are the probability mass (density) functions of \(P_\theta, P_\eta\) respectively. Prove that:
Solution (click to expand)
(a) Asymptotic distribution
Since \(X_1, ..., X_n\) are i.i.d. Bernoulli random variables, we have:
The estimator is the sample mean \(\hat{\theta}_n = \overline{X}_n = \frac{1}{n}\sum_{i=1}^n X_i\). By the classical Central Limit Theorem (CLT),
Substituting \(\theta = 1/2\), the mean is \(1/2\) and the variance is \((1/2)(1 - 1/2) = 1/4\). Therefore,
Equivalently, we can write it in standard normal form:
(b) Asymptotic distribution of the KL divergence
The KL divergence from \(P_\theta\) to \(P_{\hat{\theta}_n}\) is defined as:
For the Bernoulli distribution, the probability mass function is \(p_t(x) = t^x (1-t)^{1-x}\). To apply the Delta method, we treat the KL divergence as a function \(g(t)\) of \(t = \hat{\theta}_n\):
Since the expectation is taken under the true \(\theta\) (i.e., \(\mathbb{E}_\theta[X] = \theta\)), this becomes:
Our goal is to find the asymptotic distribution of \(n g(\hat{\theta}_n) = n(g(\hat{\theta}_n) - g(\theta))\) (since \(g(\theta) = D_{KL}(P_\theta || P_\theta) = 0\)).
First, compute the first and second derivatives of \(g(t)\):
Evaluated at the true parameter \(t = \theta\):
Since the first derivative is zero, we need to use the second-order Delta method. Compute the second derivative:
Evaluated at \(t = \theta\):
From the CLT, \(\sqrt{n}(\hat{\theta}_n - \theta) \xrightarrow{d} \mathcal{N}(0, \sigma^2)\) with asymptotic variance \(\sigma^2 = \theta(1-\theta)\).
According to the second-order Delta method, since \(g'(\theta) = 0\) and \(g''(\theta) \neq 0\), we have:
Substituting \(\sigma^2\) and \(g''(\theta)\):
This simplifies perfectly to 1. Hence,
Exercise 2: Hodges (1951) Superefficiency Estimator
Problem: Suppose \(X_1, X_2, \dots, X_n\) are i.i.d. observations from \(\mathcal{N}(\theta, 1)\) with \(\theta \in \mathbb{R}\). Then \(\overline{X}_n = n^{-1}\sum_{i=1}^n X_i\) is the MLE of \(\theta\). Show that when \(\theta \neq 0\),
Solution (click to expand)
Proof:
Since \(X_1, X_2, \dots, X_n \stackrel{i.i.d}{\sim} \mathcal{N}(\theta, 1)\), the exact distribution of the sample mean is:
Standardize \(\overline{X}_n\) to a standard normal random variable \(Z\):
Now rewrite the target probability using \(Z\):
To show that this probability converges to 0 when \(\theta \neq 0\), consider two cases:
Case 1: \(\theta > 0\)
Examine the upper bound of \(Z\):
As \(n \rightarrow \infty\), \(n^{-1/4} \rightarrow 0\). Since \(\theta > 0\), \((n^{-1/4} - \theta) \rightarrow -\theta < 0\). Therefore,
The probability is bounded above by the CDF of the standard normal at the upper bound:
As \(n \rightarrow \infty\), \(P(Z \le n^{1/4} - \sqrt{n}\theta) \rightarrow \Phi(-\infty) = 0\).
Case 2: \(\theta < 0\)
Examine the lower bound of \(Z\):
As \(n \rightarrow \infty\), \(n^{-1/4} \rightarrow 0\). Since \(\theta < 0\), \((n^{-1/4} + \theta) \rightarrow \theta < 0\). Therefore,
The probability is bounded above by the right tail probability of the standard normal:
As \(n \rightarrow \infty\), \(P(Z \ge -n^{1/4} - \sqrt{n}\theta) \rightarrow 1 - \Phi(+\infty) = 0\).
In both cases where \(\theta \neq 0\), the probability vanishes. Hence, as \(n \rightarrow \infty\),
Exercise 3: Irregular Maximum Likelihood Estimator
Problem: Let \(X_1, X_2, \dots, X_n\) be a random sample from the uniform distribution \(U(0, \theta)\) (\(\theta > 0\)).
(a) Obtain the MLE \(\hat{\theta}_n\) of \(\theta\) and show that \(\sqrt{n}(\hat{\theta}_n - \theta) \xrightarrow{p} 0\).
(b) Find an appropriate scaling factor such that the asymptotic distribution is non-degenerate, and find that asymptotic distribution.
Solution (click to expand)
(a) Derive the MLE and prove convergence
The likelihood function is:
where \(I(\cdot)\) is the indicator function. To maximize \(L(\theta)\), we need to minimize \(\theta^n\) subject to \(\theta \ge \max_{1 \le i \le n} X_i\).
Since \(\theta^n\) is strictly increasing in \(\theta\) for \(\theta > 0\), the value that maximizes the likelihood is the smallest possible \(\theta\) satisfying the constraint, i.e., the maximum order statistic:
To show \(\sqrt{n}(\hat{\theta}_n - \theta) \xrightarrow{p} 0\), analyze the bias \(\theta - \hat{\theta}_n\). Clearly \(\hat{\theta}_n \le \theta\) almost surely, so \(\theta - \hat{\theta}_n \ge 0\). Use Markov's inequality.
First compute the expectation of \(\hat{\theta}_n\). The PDF of \(X_{(n)}\) for \(0 < x < \theta\) is:
Thus the expectation is:
The expected bias is:
For any \(\epsilon > 0\), by Markov's inequality:
As \(n \rightarrow \infty\), \(\frac{\sqrt{n}}{n+1} \rightarrow 0\). The probability tends to 0, which means:
(b) Appropriate scaling and non-degenerate asymptotic distribution
From (a) we have \(\sqrt{n}(\theta - \hat{\theta}_n) \xrightarrow{p} 0\), so \(\hat{\theta}_n\) converges to \(\theta\) at a rate faster than \(\sqrt{n}\). The appropriate scaling factor here is \(n\).
We study the asymptotic distribution of the nonโnegative random variable \(Y_n = n(\theta - \hat{\theta}_n)\) by deriving the limit of its CDF. For any \(x > 0\):
Since \(\hat{\theta}_n = \max_{1 \le i \le n} X_i\), the sample maximum is strictly less than some value iff every individual observation is strictly less than that value. Because \(X_i\) are i.i.d. \(U(0, \theta)\),
Taking the limit as \(n \rightarrow \infty\) using the standard limit definition of the exponential:
Therefore, the asymptotic CDF of \(n(\theta - \hat{\theta}_n)\) is:
This is exactly the CDF of an exponential distribution with mean \(\theta\) (rate parameter \(1/\theta\)).
Conclusion, the non-degenerate asymptotic distribution is:
Exercise 4: Different Convergence Rates of MLEs
Problem: Suppose \(X_1, ..., X_n \stackrel{i.i.d}{\sim} Exp(\mu, \sigma)\) with density \(f(x) = \frac{1}{\sigma}e^{-\frac{x-\mu}{\sigma}}\), where \(x \ge \mu\). Let \((\hat{\mu}, \hat{\sigma})\) be the MLE of \((\mu, \sigma)\). Find the asymptotic distributions of \(\hat{\mu}\) and \(\hat{\sigma}\) respectively. (Marginal distributions, not joint.)
Solution (click to expand)
Step 1: Derive the MLEs of \(\mu\) and \(\sigma\)
The likelihood function is:
Under the constraint \(\mu \le X_{(1)}\), the log-likelihood is:
First maximize with respect to \(\mu\). Note that:
Thus the log-likelihood is strictly increasing in \(\mu\). To maximize it, we must make \(\mu\) as large as possible subject to \(\mu \le X_{(1)}\). Hence, the MLE of \(\mu\) is the sample minimum:
Next, differentiate with respect to \(\sigma\) and set to zero:
Solving gives \(\hat{\sigma} = \overline{X}_n - \mu\). Substituting \(\hat{\mu} = X_{(1)}\) yields the MLE of \(\sigma\):
Step 2: Asymptotic distribution of \(\hat{\mu}\)
We use the CDF method for \(\hat{\mu} = X_{(1)}\). The appropriate scaling factor is \(n\). For any \(x > 0\):
Since \(X_i \sim Exp(\mu, \sigma)\), for \(t \ge \mu\), \(P(X_i > t) = e^{-(t-\mu)/\sigma}\). Therefore:
Substituting back into the product:
This means the limiting CDF is \(1 - e^{-x/\sigma}\), which is the CDF of an exponential distribution with mean \(\sigma\) (rate \(1/\sigma\)). Hence:
Step 3: Asymptotic distribution of \(\hat{\sigma}\)
We need the asymptotic distribution of \(\hat{\sigma} = \overline{X}_n - X_{(1)}\).
First, the moments of \(X_1\) are: \(E[X_1] = \mu + \sigma\) and \(Var(X_1) = \sigma^2\). By the standard CLT for \(\overline{X}_n\):
Rewrite the standardized estimator \(\sqrt{n}(\hat{\sigma} - \sigma)\):
From Step 2, \(n(X_{(1)} - \mu) \xrightarrow{d} Exp(\sigma)\), which implies \(n(X_{(1)} - \mu) = O_p(1)\). Consequently,
That is, \(\sqrt{n}(X_{(1)} - \mu) \xrightarrow{p} 0\).
By Slutskyโs theorem, since the second term converges in probability to 0, the asymptotic distribution is entirely determined by the first term. Hence:
Exercise 5: Inconsistent MLE and the Neyman-Scott Problem
Problem: Let \(X_{ij}\) (\(i=1,...,n\), \(j=1,...,k\)) be independent with \(X_{ij} \sim \mathcal{N}(\mu_i, \sigma^2)\). This is essentially a balanced oneโway ANOVA design; assume \(k\) is fixed and \(n \rightarrow \infty\), so that the number of parameters grows with the sample size. Show that the MLE \(\hat{\sigma}^2\) of the common variance satisfies:
Solution (click to expand)
Step 1: Derive the MLEs of \(\mu_i\) and \(\sigma^2\)
The likelihood function is:
Taking logarithms gives:
To find the MLE of \(\mu_i\), take the derivative and set it to zero:
Next, differentiate with respect to \(\sigma^2\) and set to zero:
Substituting \(\hat{\mu}_i = \overline{X}_{i.}\) gives the MLE of the common variance:
Step 2: Prove convergence in probability
Let \(Y_i = \sum_{j=1}^k (X_{ij} - \overline{X}_{i.})^2\). For each group \(i\), because \(X_{ij} \sim \mathcal{N}(\mu_i, \sigma^2)\) for \(j=1,...,k\), the scaled sum of squares follows a chiโsquare distribution:
Hence \(Y_i \sim \sigma^2 \chi_{k-1}^2\). Since samples across groups (\(i=1,...,n\)) are independent, the sequence \(Y_1, Y_2, ..., Y_n\) consists of i.i.d. random variables.
The expectation of \(Y_i\) is \(E[Y_i] = \sigma^2(k-1)\).
When \(n \rightarrow \infty\) (while \(k\) is fixed), apply the Weak Law of Large Numbers (WLLN) to \(\{Y_i\}_{i=1}^n\):
Now rewrite the estimator \(\hat{\sigma}^2\) in terms of this sequence:
By the continuous mapping theorem (or basic properties of convergence in probability), multiplying by the constant \(1/k\) yields:
This completes the proof. This result illustrates the famous Neyman-Scott problem: the MLE of the common variance is inconsistent because the number of nuisance parameters (\(\mu_1, ..., \mu_n\)) grows at the same rate as the sample size \(n\).
Exercise 6: Consistency and Asymptotic Normality of Sample Quantiles
Problem: (a) Define the check function \(\rho_\tau(y) = y(\tau - I(y < 0))\) as the loss function, and let the quantile function of a distribution \(F\) be \(F^{-1}(p) := \inf\{x \in \mathbb{R} : F(x) \ge p\}\). Prove that:
(b) Verify the consistency of the sample quantile defined by the following Zโestimator:
Hint: Use Theorem 5.7 or Lemma 5.10 from van der Vaart [1] (add regularity conditions if necessary).
(c) (van der Vaart [1] Ex. 5.11) Assuming consistency, use Theorem 5.23 to prove the asymptotic normality of the sample quantile.
Solution (click to expand)
(a) Show that the minimizer of the expected loss is the population quantile
Assume a regularity condition: the random variable \(X\) has a continuous density \(f(x)\) and distribution function \(F(x)\). Define the objective function \(M(\theta) = E[\rho_\tau(X-\theta)]\). Using the definition \(\rho_\tau(y) = y(\tau - I(y < 0))\), write the expectation as an integral:
Split the integral:
Differentiate with respect to \(\theta\) using Leibniz's rule. Note that when \(x=\theta\), the term \((x-\theta)\) vanishes, so boundary terms disappear:
Computing the integrals:
Set the derivative to zero to find the minimizer:
By the definition of the generalized inverse distribution function (quantile function) \(F^{-1}(\tau) = \inf\{x \in \mathbb{R} : F(x) \ge \tau\}\), we conclude:
(b) Verify consistency of the sample quantile
We will use Lemma 5.10 to prove consistency of \(\hat{\theta}_n\).
Lemma 5.10 (van der Vaart)
Let \(\Psi_n(\theta)\) be random functions and \(\Psi(\theta)\) a fixed function such that for each \(\theta\), \(\Psi_n(\theta)\) converges in probability to \(\Psi(\theta)\). Suppose the map \(\theta \mapsto \Psi_n(\theta)\) is nondecreasing and \(\Psi_n(\hat{\theta}_n) = o_p(1)\). If \(\theta_0\) satisfies that for every \(\epsilon > 0\), \(\Psi(\theta_0 - \epsilon) < 0 < \Psi(\theta_0 + \epsilon)\), then \(\hat{\theta}_n \xrightarrow{p} \theta_0\).
We verify the conditions of this lemma one by one:
-
Condition 1: Monotonicity of \(\Psi_n(\theta)\)
As \(\theta\) increases, the indicator \(I(X_i < \theta)\) is nondecreasing, while \(I(X_i > \theta)\) is nonincreasing (so \(-\tau I(X_i > \theta)\) is also nondecreasing). Since \((1-\tau) > 0\) and \(\tau > 0\), the map \(\theta \mapsto \Psi_n(\theta)\) is a positive linear combination of nondecreasing functions, hence it is nondecreasing.
-
Condition 2: Convergence in probability to a fixed function \(\Psi(\theta)\)
By the Weak Law of Large Numbers, for any fixed \(\theta\), \(\Psi_n(\theta)\) converges in probability to its expectation. Assume \(X\) is continuous (so \(P(X=\theta)=0\)):
\[ \Psi(\theta) = E[\Psi_n(\theta)] = (1-\tau)P(X < \theta) - \tau P(X > \theta) = (1-\tau)F(\theta) - \tau(1-F(\theta)) = F(\theta) - \tau \]Thus \(\Psi_n(\theta) \xrightarrow{p} \Psi(\theta)\) holds.
-
Condition 3: Unique zero crossing (add a regularity condition)
The lemma requires that for any \(\epsilon > 0\), \(\Psi(\theta_0 - \epsilon) < 0 < \Psi(\theta_0 + \epsilon)\). For this we add a regularity condition: assume the distribution function \(F(x)\) is strictly increasing in a neighborhood of the true parameter \(\theta_0 = F^{-1}(\tau)\). Under this condition, for any \(\epsilon > 0\),
\[ F(\theta_0 - \epsilon) < F(\theta_0) = \tau < F(\theta_0 + \epsilon) \]Subtracting \(\tau\) gives exactly:
\[ \Psi(\theta_0 - \epsilon) < 0 < \Psi(\theta_0 + \epsilon) \]Since \(\theta \mapsto \Psi_n(\theta)\) is nondecreasing, the sample quantile satisfies \(\Psi_n(\hat{\theta}_n) = 0\) (or \(o_p(1)\)), and the true parameter \(\theta_0\) uniquely crosses the zero of the limiting function \(\Psi(\theta)\), all conditions of Lemma 5.10 are satisfied. Hence the sample quantile is consistent:
\[ \hat{\theta}_n \xrightarrow{p} \theta_0 \]
(c) Asymptotic normality of the sample quantile
To prove asymptotic normality, we apply Theorem 5.23. To match the notation of the textbook, we denote the quantile as \(p\) (which corresponds to \(\tau\) in the previous part).
Theorem 5.23 (Asymptotic normality of Mโestimators)
Suppose the map \(\theta \mapsto m_\theta(x)\) is differentiable at \(\theta_0\) for \(P\)-almost every \(x\), with derivative \(\dot{m}_{\theta_0}(x)\). Moreover, assume a Lipschitz condition: \(|m_{\theta_1}(x) - m_{\theta_2}(x)| \le \dot{m}(x)|\theta_1 - \theta_2|\), where the envelope satisfies \(P\dot{m}^2 < \infty\). Also assume that \(\theta \mapsto P m_\theta\) admits a secondโorder Taylor expansion at the maximum \(\theta_0\) with a nonโsingular symmetric second derivative matrix \(V_{\theta_0}\). If \(\hat{\theta}_n \xrightarrow{p} \theta_0\) and approximately maximizes the empirical objective function, then:
\(\sqrt{n}(\hat{\theta}_n - \theta_0) = -V_{\theta_0}^{-1} \frac{1}{\sqrt{n}} \sum_{i=1}^n \dot{m}_{\theta_0}(X_i) + o_p(1)\).
The sample \(p\)-quantile \(\hat{\theta}_n = \mathbb{F}_n^{-1}(p)\) minimizes the empirical objective function \(\frac{1}{n}\sum \rho_p(X_i - \theta)\). To fit the "maximization" framework of Theorem 5.23, define the objective function as:
We verify the conditions of Theorem 5.23 at the true parameter \(\theta_0 = F^{-1}(p)\):
-
Condition 1: \(P\)-almost everywhere differentiability
The map \(\theta \mapsto m_\theta(x)\) is differentiable everywhere except at \(\theta = x\). Because the distribution \(F\) is differentiable at \(\theta_0\) with density \(f(\theta_0) > 0\), the probability of an observation exactly equal to \(\theta_0\) is zero, i.e., \(P(X=\theta_0)=0\). Hence \(m_\theta(x)\) is differentiable at \(\theta_0\) for \(P\)-almost every \(x\), with derivative:
\[ \dot{m}_{\theta_0}(x) = \frac{\partial}{\partial \theta} [-(x-\theta)p + (x-\theta)I(x < \theta)] \Big|_{\theta=\theta_0} = p - I(x \le \theta_0) \] -
Condition 2: Lipschitz continuity and squareโintegrable envelope
The check function \(\rho_p(u)\) is Lipschitz continuous with constant bounded by \(\max(p, 1-p) \le 1\). Therefore, for any \(\theta_1, \theta_2\):
\[ |m_{\theta_1}(x) - m_{\theta_2}(x)| = |\rho_p(x-\theta_1) - \rho_p(x-\theta_2)| \le 1 \cdot |\theta_1 - \theta_2| \]We can choose the envelope \(\dot{m}(x) = 1\). The condition \(P\dot{m}^2 = E[1^2] = 1 < \infty\) is clearly satisfied.
-
Condition 3: Secondโorder Taylor expansion and nonโsingular \(V_{\theta_0}\)
The expected objective function is \(Pm_\theta = -E[\rho_p(X-\theta)]\). From part (a), its first derivative is:
\[ \frac{d}{d\theta} (Pm_\theta) = p - F(\theta) \]Differentiating again, we obtain the second derivative at \(\theta_0\):
\[ V_{\theta_0} = \frac{d}{d\theta} (p - F(\theta)) \Big|_{\theta=\theta_0} = -f(\theta_0) \]The problem assumes that \(F\) has a positive derivative at the population \(p\)-quantile \(F^{-1}(p)\), i.e., \(f(\theta_0) > 0\). Therefore, the scalar \(V_{\theta_0} = -f(\theta_0)\) is strictly negative, satisfying the "nonโsingular" condition.
-
Applying Theorem 5.23
We have already established consistency \(\hat{\theta}_n \xrightarrow{p} \theta_0\) in part (b). All conditions are met. Applying Theorem 5.23:
\[ \sqrt{n}(\hat{\theta}_n - \theta_0) = -V_{\theta_0}^{-1} \frac{1}{\sqrt{n}} \sum_{i=1}^n \dot{m}_{\theta_0}(X_i) + o_p(1) \]Substituting \(V_{\theta_0}\) and \(\dot{m}_{\theta_0}(X_i)\):
\[ \sqrt{n}(\hat{\theta}_n - \theta_0) = \frac{1}{f(\theta_0)} \frac{1}{\sqrt{n}} \sum_{i=1}^n (p - I(X_i \le \theta_0)) + o_p(1) \]Let \(Y_i = p - I(X_i \le \theta_0)\). The sequence \(\{Y_i\}_{i=1}^n\) is i.i.d. with: \(E[Y_i] = p - P(X_i \le \theta_0) = p - F(\theta_0) = p - p = 0\) \(Var(Y_i) = Var(I(X_i \le \theta_0)) = p(1-p)\)
By the classical CLT:
\[ \frac{1}{\sqrt{n}} \sum_{i=1}^n Y_i \xrightarrow{d} \mathcal{N}(0, p(1-p)) \]Finally, by Slutskyโs theorem, multiplying by the constant scaling \(1/f(\theta_0)\) gives the asymptotic normal distribution:
\[ \sqrt{n}(\hat{\theta}_n - \theta_0) \xrightarrow{d} \mathcal{N}\left(0, \frac{p(1-p)}{f(\theta_0)^2}\right) \]