Chapter 8: Moment Estimation and Generalized Method of Moments (ME & GMM)

In statistical inference, Maximum Likelihood Estimation (MLE) is the gold standard for parameter estimation. However, MLE strongly depends on a full parametric specification of the data distribution. When the exact form of the distribution is unknown, or only partial moment conditions are known, we need other estimation methods. This chapter introduces Moment Estimation (ME) and the Generalized Method of Moments (GMM), which is extremely important in econometrics and statistics.

1. Moment Estimator (ME)

Let \(X_1, ..., X_n \stackrel{i.i.d}{\sim} F_\theta\), where \(\theta_0\) is the true parameter. Given a set of known functions \(f_1, ..., f_k\).

The population moments are defined as:

\[ E_\theta f_j(X) = \int f_j(x) dF_\theta(x), \quad j=1, ..., k \]

The most classical choice is \(f_j(x) = x^j\). Let the vector \(f = (f_1, ..., f_k)^T\).

Definition 12.1 (Moment Estimator, ME)

Match the sample moment \(\frac{1}{n} \sum_{i=1}^n f(X_i)\) with its corresponding population moment:

\[ P_n f := \frac{1}{n} \sum_{i=1}^n f(X_i) \stackrel{\text{match}}{=} e(\theta) := E_\theta f(X) := P_\theta f \]

If \(e(\theta)\) is one-to-one, then the moment estimator can be expressed explicitly as \(\hat{\theta}_n = e^{-1}(P_n f)\).

If the inverse map \(e^{-1}\) is differentiable and the second moment \(E_{\theta_0} f(X)f^T(X) < \infty\) (which guarantees the asymptotic normality of \(P_n f\)), then \(\hat{\theta}_n\) is also asymptotically normal.

Theorem 12.1 (Central Limit Theorem for Moment Estimator )

If \(e(\theta) = P_\theta f := E_\theta f(X)\) is one-to-one on an open set \(\Theta \subset \mathbb{R}^k\), continuously differentiable at \(\theta_0\) with non-singular Jacobian matrix \(e'(\theta_0)\), and \(P_{\theta_0} \|f\|^2 < \infty\).

Then \(\hat{\theta}_n\) exists with probability approaching 1 (wpa 1) and satisfies asymptotic normality:

\[ \sqrt{n}(\hat{\theta}_n - \theta_0) \xrightarrow{d} N\left(0, [e'(\theta_0)]^{-1} (E_{\theta_0} ff^T - E_{\theta_0} f E_{\theta_0} f^T) ([e'(\theta_0)]^{-1})^T \right) \]

To prove this theorem, we need the Inverse Function Theorem:

Lemma 12.1 (The Inverse Function Theorem)

Let \(A\) be an open set in \(\mathbb{R}^n\); \(g: A \mapsto \mathbb{R}^n\) is continuously differentiable at \(a\), and differentiable in a neighborhood of \(a \in A\). If the Jacobian matrix \(Dg(x) := \partial g(x) / \partial x^T\) is non-singular at \(a\).

Then there exists a neighborhood \(U\) of \(a\) and an open set \(V\) in \(\mathbb{R}^n\) such that \(g: U \mapsto V\) is one-to-one; and the inverse function \(g^{-1}: V \mapsto U\) exists, is continuously differentiable, and satisfies:

\[ Dg^{-1}(y) := \partial g^{-1}(y) / \partial y^T = (Dg(x))^{-1} \]

Proof of Theorem 12.1 (click to expand)

Continuous differentiability at \(\theta_0\) implies differentiability in a neighborhood of \(\theta_0\) and continuity of \(\theta \mapsto e'_\theta\).
Non-singularity of \(e'_\theta\) implies it is also non-singular in some neighborhood of \(\theta_0\).
Therefore, by the Inverse Function Theorem, there exist an open neighborhood \(U\) of \(\theta_0\) and an open neighborhood \(V\) of \(P_{\theta_0}f\) such that \(e: U \mapsto V\) is a diffeomorphism (one-to-one) with differentiable inverse \(e^{-1}: V \mapsto U\).
By the Law of Large Numbers (LLN), \(\mathbb{P}_n f \equiv \frac{1}{n} \sum_{i=1}^n f(X_i) \xrightarrow{a.s.} e(\theta_0) \in V\). Hence, wpa 1 \(\mathbb{P}_n f \in V\), which implies \(\hat{\theta}_n = e^{-1}(\mathbb{P}_n f)\) exists wpa 1. Then by the continuous mapping theorem:

\[ \hat{\theta}_n := e^{-1}(\mathbb{P}_n f) \xrightarrow{a.s.} e^{-1}(e(\theta_0)) = \theta_0 \]

By the Central Limit Theorem (CLT), the sequence \(\sqrt{n}(\mathbb{P}_n f - P_{\theta_0} f)\) satisfies asymptotic normality.
Finally, applying the Delta Method completes the proof: multiplying by the Jacobian matrix \((e^{-1}(x_0))' = (e'(\theta_0))^{-1}\) yields the final asymptotic variance.

2. Examples of Moment Estimation and Exponential Family

2.1 Example: Moment Estimation of the Beta Distribution

Let \(X_1, X_2, ..., X_n\) be a random sample from a Beta distribution with density \(f(x; \alpha, \beta) = \frac{1}{B(\alpha, \beta)} x^{\alpha-1} (1-x)^{\beta-1}\), where \(x \in (0,1)\), parameters \(\alpha > 0, \beta > 0\).

The population moments are:

\[ \mathbb{E}X^k = \frac{1}{B(\alpha, \beta)} \int x^{\alpha+k-1} (1-x)^{\beta-1} dx = \frac{B(\alpha+k, \beta)}{B(\alpha, \beta)} = \prod_{r=0}^{k-1} \frac{\alpha+r}{\alpha+\beta+r} \]

Matching the first two moments:

\[ \overline{X} = \frac{\alpha}{\alpha+\beta}, \quad \overline{X^2} = \frac{\alpha(\alpha+1)}{(\alpha+\beta)(\alpha+\beta+1)} \]

Solving gives the moment estimators:

\[ \hat{\alpha} = \overline{X} \left[ \frac{\overline{X}(1-\overline{X})}{\overline{X^2} - \overline{X}^2} - 1 \right], \quad \hat{\beta} = (1-\overline{X}) \left[ \frac{\overline{X}(1-\overline{X})}{\overline{X^2} - \overline{X}^2} - 1 \right] \]

The asymptotic distribution can be obtained by computing the Jacobian matrix and the covariance matrix in the theorem, yielding \(\sqrt{n}(\hat{\theta}_n - \theta) \Rightarrow N(0, \Sigma)\).

2.2 Exponential Family

Assume \(X_1, ..., X_n \sim F_\theta\), and the density belongs to an exponential family:

\[ p_\theta(x) = c(\theta)h(x)\exp\{\theta^T t(x)\} \]

The log-likelihood is \(l_\theta(x) = \log c(\theta) + \log h(x) + \theta^T t(x)\). The score function is \(\dot{l}_\theta(x) = \frac{\dot{c}(\theta)}{c(\theta)} + t(x) = t(x) - E_\theta t(X)\) (since \(E \dot{l}_\theta(X) = 0\) implies \(\dot{c}(\theta)/c(\theta) = -E_\theta t(X)\)).

Therefore, the Maximum Likelihood Estimator (MLE) is essentially a Moment Estimator (ME):

\[ \frac{1}{n} \sum_{i=1}^n t(X_i) = E_\theta t(X) = e(\theta) \]

If \(e(\theta)\) is continuously differentiable and \(E_\theta \|t(X)\|^2 < \infty\), then \(\hat{\theta}_{mle} = \hat{\theta}_{me}\), and:

\[ \sqrt{n}(\hat{\theta}_{me} - \theta) \rightarrow N\left(0, [(e'(\theta))^{-1}] var_\theta(t(X)) [(e'(\theta))^{-1}]^T \right) = N(0, I_\theta^{-1}) \]

where \(I_\theta\) is the Fisher information matrix \(I_\theta = var(\dot{l}_\theta(X))\).

Limitations of Moment Estimation: 1. The statistical efficiency of ME is usually lower than that of MLE. 2. ME can be semiparametric, while MLE must be fully parametric. 3. The choice of "moments" to match is somewhat ad-hoc.

3. Generalized Method of Moments (GMM)

MLE requires a full distributional specification of the model (e.g., joint density, conditional density). GMM, proposed by Hansen (1982), generalizes moment estimation and does not require specifying the distribution density, only moment constraints. This makes GMM very attractive for models with infinite-dimensional nuisance parameters (semiparametric).

3.1 Setup and Examples

Let \(\{w_i\}_{i=1}^n\) be i.i.d. random variables in \(\mathbb{R}^m\), \(g(w_i; \theta) \in \mathbb{R}^r\) be a known \(r\)-dimensional function, and parameter \(\theta \in \Theta \subset \mathbb{R}^p\). Assume there exists a unique true parameter \(\theta_0 \in \Theta\) such that:

\[ E\{g(w_i; \theta_0)\} = 0 \]

Just-identified: When \(r = p\) (number of moment conditions equals number of parameters), statisticians usually directly solve the equation \(\frac{1}{n}\sum_{i=1}^n g(w_i, \theta) = 0\).
Over-identified: When \(r > p\), the system of equations cannot be solved exactly; this is common in econometrics and requires GMM.

Example 1: Parametric Regression Data \(\{w_i = (Y_i, X_i)\}_{i=1}^n\). Model \(Y_i = m(X_i, \theta) + \epsilon_i\), with \(E(\epsilon_i | X_i) = 0\). The least squares score function is \(g(w_i, \theta) = \frac{\partial m(X_i, \theta)}{\partial \theta}(Y_i - m(X_i, \theta))\). Here \(r = p\), can be solved directly.

Example 2: Poisson Regression Suppose count data \(Y_i\) satisfy \(E(Y_i | X_i) = \mu(X_i, \theta)\) and \(Var(Y_i | X_i) = \mu(X_i, \theta)\). We can combine information from both the mean and variance to construct an over-identified score function:

\[ g(w_i, \theta) = \begin{pmatrix} g_1(w_i, \theta) \\ g_2(w_i, \theta) \end{pmatrix} = \begin{pmatrix} \frac{\partial \mu(X_i, \theta)}{\partial \theta}(Y_i - \mu(X_i, \theta)) \\ a(X_i, \theta) \{[Y_i - \mu(X_i, \theta)]^2 - \mu(X_i, \theta)\} \end{pmatrix} \]

This uses more information and is more efficient than using only the mean.

3.2 Definition of GMM Estimator

Definition 12.2 (GMM Estimator)

Given \(g(w_i; \theta)\) satisfying \(E g(w_i; \theta_0) = 0\). The GMM estimator \(\hat{\theta}_{GMM}\) is defined as minimizing the following quadratic form:

\[ \hat{\theta}_n = \arg\min_{\theta \in \Theta} \left( \frac{1}{n}\sum_{i=1}^n g(w_i, \theta) \right)^T \hat{W}_n \left( \frac{1}{n}\sum_{i=1}^n g(w_i, \theta) \right) \]

where \(\hat{W}_n\) is an \(r \times r\) nonnegative definite weight matrix satisfying \(\hat{W}_n \xrightarrow{p} W_0 > 0\), with \(W_0\) deterministic and possibly depending on \(\theta_0\).

For identification, we assume \(\theta_0\) is the unique parameter such that \(E g(w_i; \theta) = 0\). Because \(W_0 > 0\), \(\theta_0\) is also the unique minimizer of the objective function \(E\{g^T(w_i; \theta)\} W_0 E\{g(w_i; \theta)\}\).

4. Asymptotic Normality of GMM

Assumptions: (i) \(g(w, \cdot)\) is continuously differentiable at the interior point \(\theta \in Int(\Theta)\). (ii) \(G_0 = E\left( \frac{\partial g(w, \theta_0)}{\partial \theta^T} \right)_{r \times p}\) exists and has full column rank (rank \(p\)).

Theorem 12.2 (Asymptotic Normality of GMM)

Under conditions ensuring \(\hat{\theta}_n \rightarrow \theta_0\), we have:

\[ \sqrt{n}(\hat{\theta}_n - \theta_0) \xrightarrow{d} N\left(0, (G_0^T W_0 G_0)^{-1} G_0^T W_0 \Lambda_0 W_0 G_0 (G_0^T W_0 G_0)^{-1} \right) \]

where by the Central Limit Theorem, \(\frac{1}{\sqrt{n}} \sum_{i=1}^n g(w_i, \theta_0) \rightarrow N(0, \Lambda_0)\), with \(\Lambda_0 = Var\{g(w_i; \theta_0)\}\).

Proof of Asymptotic Normality (click to expand)

From the definition that \(\hat{\theta}_n\) minimizes the quadratic form, the first-order condition (FOC) is:

\[ \frac{1}{n} \sum_{i=1}^n \frac{\partial g^T(w_i, \hat{\theta}_n)}{\partial \theta} \hat{W}_n \left( \frac{1}{n} \sum_{i=1}^n g(w_i, \hat{\theta}_n) \right) = 0 \quad \cdots (1) \]

(Note the matrix calculus rule: \(\frac{\partial g^T W g}{\partial \theta_i} = 2 \frac{\partial g^T}{\partial \theta_i} W g\).)

Perform a Taylor expansion of \(g(w_i, \hat{\theta}_n)\) in (1) around the true parameter \(\theta_0\):

\[ \frac{1}{n} \sum_{i=1}^n \frac{\partial g^T(w_i, \hat{\theta}_n)}{\partial \theta} \hat{W}_n \left\{ \frac{1}{n} \sum_{i=1}^n g(w_i, \theta_0) + \frac{1}{n} \sum_{i=1}^n \frac{\partial g(w_i, \hat{\theta}_n^*)}{\partial \theta^T} (\hat{\theta}_n - \theta_0) \right\} = 0 \quad \cdots (2) \]

where \(\hat{\theta}_n^*\) lies between \(\theta_0\) and \(\hat{\theta}_n\). Since \(\hat{\theta}_n \rightarrow \theta_0\), by the Law of Large Numbers the gradient matrices converge in probability:

\[ \frac{1}{n} \sum_{i=1}^n \frac{\partial g^T(w_i, \hat{\theta}_n)}{\partial \theta} \xrightarrow{p} G_0^T, \quad \frac{1}{n} \sum_{i=1}^n \frac{\partial g^T(w_i, \hat{\theta}_n^*)}{\partial \theta} \xrightarrow{p} G_0^T \]

Since \(\hat{W}_n \rightarrow W_0\), from (2) we obtain:

\[ (G_0^T W_0 G_0)(\hat{\theta}_n - \theta_0) = -G_0^T W_0 \frac{1}{n} \sum_{i=1}^n g(w_i, \theta_0) \{1 + o_p(1)\} \]

Let \(A_0 = G_0^T W_0 G_0\), multiply both sides by \(\sqrt{n}\):

\[ \sqrt{n}(\hat{\theta}_n - \theta_0) = -A_0^{-1} G_0^T W_0 \frac{1}{\sqrt{n}} \sum_{i=1}^n g(w_i, \theta_0) + o_p(1) \]

Since \(\frac{1}{\sqrt{n}} \sum_{i=1}^n g(w_i, \theta_0) \xrightarrow{d} N(0, \Lambda_0)\), we finally obtain:

\[ \sqrt{n}(\hat{\theta}_n - \theta_0) \xrightarrow{d} N(0, A_0^{-1} G_0^T W_0 \Lambda_0 W_0 G_0 A_0^{-1}) \]

The proof is complete.

5. Optimal Weight Matrix and Inference

5.1 Optimal Weight Matrix

The efficiency of the GMM estimator depends on the choice of the weight matrix \(\hat{W}_n\) (converging to \(W_0\)). Which is best?

If we choose \(W_0 = \Lambda_0^{-1}\), the middle term in the asymptotic variance becomes \(G_0^T \Lambda_0^{-1} \Lambda_0 \Lambda_0^{-1} G_0 = G_0^T \Lambda_0^{-1} G_0\), and the asymptotic variance simplifies to:

\[ Avar(\hat{\theta}_{GMM}) = A_0^{-1} = (G_0^T \Lambda_0^{-1} G_0)^{-1} \]

Hansen (1982) showed that for any \(W_0 > 0\):

\[ (G_0^T W_0^{-1} G_0)^{-1} (G_0^T W_0 \Lambda_0 W_0 G_0) (G_0^T W_0^{-1} G_0)^{-1} \ge (G_0^T \Lambda_0^{-1} G_0)^{-1} \]

Therefore, \(W_0^* = \Lambda_0^{-1}\) is the optimal weight matrix.

5.2 Two-step GMM Estimation

To obtain the optimal GMM estimator, a two-step procedure is commonly used:

Step 1: Construct an initial estimator \(\tilde{\theta}\) using any positive definite weight matrix (e.g., \(\hat{W}_n = I_r\)).
Step 2: Estimate the covariance matrix using the initial estimator to obtain the optimal weight matrix:

\[ \hat{W}_n^* = \hat{\Lambda}^{-1} = \left( \frac{1}{n} \sum_{i=1}^n g(w_i; \tilde{\theta}) g^T(w_i; \tilde{\theta}) \right)^{-1} \]

Re-estimate using \(\hat{W}_n^*\) to get the efficient GMM estimator \(\hat{\theta}_{GMM}^*\).

6. Hypothesis Testing and the Sargan-Hansen Test

6.1 Sargan-Hansen Test (Overidentification Test)

For over-identified models (\(r > p\)), we need to test whether the moment conditions themselves are valid, i.e., test \(H_0: E g(w_i; \theta_0) = 0\). Using the optimal GMM estimator \(\hat{\theta} = \hat{\theta}_{GMM}^*\), the objective function satisfies:

\[ T_n(\hat{\theta}) = n^{-1/2} \sum_{i=1}^n g^T(w_i; \hat{\theta}) \hat{W}_n^* n^{-1/2} \sum_{i=1}^n g(w_i; \hat{\theta}) \xrightarrow{d} \chi^2_{r-p} \]

Rejection rule: If \(T_n(\hat{\theta}) > \chi^2_{r-p, 1-\alpha}\), reject the null hypothesis.
In this context, \(\hat{\theta}_{GMM}\) is also called the minimum chi-square estimator.

6.2 Tests of Parameter Hypotheses

For testing \(H_0: c(\theta_0) = 0\), where \(c(\theta) \in \mathbb{R}^Q\) and \(Q \le p\):

Wald test:

\[ c^T(\hat{\theta}) (\hat{V}(\hat{\theta}))^{-1} c(\hat{\theta}) \xrightarrow{d} \chi^2_Q \]

where \(\hat{V}(\hat{\theta}) = var(c(\hat{\theta}))\).
Distance Discrepancy Test:

\[ \tilde{T}_n = \frac{1}{n} \left[ \sum_{i=1}^n g^T(w_i; \tilde{\theta}_n) \hat{W}_n^* \sum_{i=1}^n g(w_i; \tilde{\theta}_n) - \sum_{i=1}^n g^T(w_i; \hat{\theta}_n) \hat{W}_n^* \sum_{i=1}^n g(w_i; \hat{\theta}_n) \right] \xrightarrow{d} \chi^2_Q \]

where \(\tilde{\theta}_n\) is the restricted GMM estimator under the constraint \(c(\theta) = 0\).

7. An Important Feature of GMM: More Moment Conditions Increase Efficiency

If we have \(r\) restrictions, does adding more moment conditions improve efficiency? Consider \(g(w, \theta) = (g_{(r-1)}^T(w, \theta), g_r(w, \theta))^T\).

The asymptotic variance based on the \(r\)-dimensional condition is: \(V_r = [E\left(\frac{\partial g^T}{\partial \theta}\right) E^{-1}(gg^T) E\left(\frac{\partial g}{\partial \theta}\right)]^{-1}\). The asymptotic variance based on the \((r-1)\)-dimensional condition is: \(V_{r-1} = [E\left(\frac{\partial g_{(r-1)}^T}{\partial \theta}\right) E^{-1}(g_{(r-1)} g_{(r-1)}^T) E\left(\frac{\partial g_{(r-1)}}{\partial \theta}\right)]^{-1}\).

Important Feature

\[ V_r \le V_{r-1} \]

Hence, GMM with \(r\) restrictions is at least as efficient as GMM with \(r-1\) restrictions.

Proof of the Feature (click to expand)

At \(\theta = \theta_0\), partition the covariance matrix:

\[ E(gg^T) = \begin{pmatrix} E(g_{(r-1)} g_{(r-1)}^T) & E(g_{(r-1)} g_r^T) \\ E(g_{(r-1)}^T g_r) & E(g_r^2) \end{pmatrix} := \begin{pmatrix} A & B \\ B^T & C \end{pmatrix} \]

Using the partitioned matrix inversion formula, let the Schur complement be \(D = C - B^T A^{-1} B\):

\[ \begin{pmatrix} A & B \\ B^T & C \end{pmatrix}^{-1} = \begin{pmatrix} A^{-1} & 0 \\ 0 & 0 \end{pmatrix} + \begin{pmatrix} A^{-1}B \\ -I \end{pmatrix} D^{-1} (B^T A^{-1}, -I) \]

This can be written compactly as:

\[ (E(gg^T))^{-1} = \begin{pmatrix} (E(g_{(r-1)} g_{(r-1)}^T))^{-1} & 0 \\ 0 & 0 \end{pmatrix} + H \]

where \(H \ge 0\) (positive semidefinite). Substitute this into the quadratic form of the information matrix:

\[ V_r^{-1} = \left( E\left(\frac{\partial g_{(r-1)}^T}{\partial \theta}\right), E\left(\frac{\partial g_r^T}{\partial \theta}\right) \right) \left\{ \begin{pmatrix} (E(g_{(r-1)} g_{(r-1)}^T))^{-1} & 0 \\ 0 & 0 \end{pmatrix} + H \right\} \begin{pmatrix} E\left(\frac{\partial g_{(r-1)}}{\partial \theta}\right) \\ E\left(\frac{\partial g_r}{\partial \theta}\right) \end{pmatrix} \]

Expanding gives:

\[ V_r^{-1} = E\left(\frac{\partial g_{(r-1)}^T}{\partial \theta}\right) (E(g_{(r-1)} g_{(r-1)}^T))^{-1} E\left(\frac{\partial g_{(r-1)}}{\partial \theta}\right) + \tilde{H} = V_{r-1}^{-1} + \tilde{H} \]

Because \(\tilde{H} \ge 0\), we have \(V_r^{-1} \ge V_{r-1}^{-1}\), i.e., \(V_r \le V_{r-1}\). This is the appeal of GMM: by adding valid moment conditions in certain directions, the asymptotic variance can be reduced.