Chapter 1: Probability Spaces and Random Variables

This chapter reviews the basic language of probability theory. We first start from sample spaces, event fields, and probability measures to define probability spaces; then introduce independence of events and product probability spaces; finally give the definitions and basic properties of random variables, induced distributions, and distribution functions.

The course will subsequently further discuss numerical characteristics of random variables (expectation, variance, and characteristic functions) and classical limit theorems.

1. Probability Spaces

1.1 Sample Space and Events

Definition 1.1 (Sample Space)

The set of all possible outcomes of a random experiment is called the sample space, denoted by \(\Omega\). An element \(\omega\in\Omega\) in the sample space is called a sample point.

A subset of the sample space is called an event. In order to assign probabilities to events, one must first specify a family of events that are allowed to be discussed, which leads to the \(\sigma\)-algebra.

1.2 \(\sigma\)-Algebra

Definition 1.2 (\(\sigma\)-Algebra)

Let \(\mathcal{A}\) be a family of subsets of \(\Omega\). If it satisfies:

(i) \(\varnothing,\Omega\in\mathcal{A}\);

(ii) If \(A\in\mathcal{A}\), then \(A^c\in\mathcal{A}\);

(iii) If \(A_1,A_2,\ldots\in\mathcal{A}\), then

\[ \bigcup_{n=1}^{\infty}A_n\in\mathcal{A}. \]

then \(\mathcal{A}\) is called a \(\sigma\)-algebra on \(\Omega\), and its elements are called events.

From the closure under complements and countable unions, by De Morgan's formulas we also obtain: if \(A_1,A_2,\ldots\in\mathcal{A}\), then

\[ \bigcap_{n=1}^{\infty}A_n\in\mathcal{A}. \]

Definition 1.3 (Generated \(\sigma\)-Algebra)

Let \(\mathcal{C}\) be a family of subsets of \(\Omega\). The smallest \(\sigma\)-algebra containing \(\mathcal{C}\) is called the \(\sigma\)-algebra generated by \(\mathcal{C}\), denoted by

\[ \sigma(\mathcal{C}). \]

In particular, the \(\sigma\)-algebra generated by events \(A_1,\ldots,A_n\) is denoted by

\[ \sigma(A_1,\ldots,A_n). \]

1.3 Probability Measure

Definition 1.4 (Probability Measure)

Let \(\mathcal{A}\) be a \(\sigma\)-algebra on \(\Omega\). A mapping

\[ P:\mathcal{A}\longrightarrow[0,1] \]

if it satisfies:

(i) \(P(\Omega)=1\);

(ii) For any pairwise disjoint events \(A_1,A_2,\ldots\in\mathcal{A}\),

\[ P\left(\bigcup_{n=1}^{\infty}A_n\right) =\sum_{n=1}^{\infty}P(A_n). \]

then \(P\) is called a probability measure on \((\Omega,\mathcal{A})\).

The triple

\[ (\Omega,\mathcal{A},P) \]

is called a probability space.

Example 1.5 (Classical Probability Model)

Let the sample space be finite:

\[ \Omega=\{\omega_1,\ldots,\omega_N\}, \]

and let each sample point be equally likely, that is,

\[ P(\{\omega_i\})=\frac{1}{N}, \qquad i=1,\ldots,N. \]

Then for any event \(A\subseteq\Omega\),

\[ P(A)=\frac{\# A}{N}, \]

where \(\#A\) denotes the number of sample points in event \(A\).

2. Independence of Events

2.1 Independence of Two Events

Definition 2.1 (Independence of Two Events)

Events \(A,B\in\mathcal{A}\) are called mutually independent if

\[ P(A\cap B)=P(A)P(B). \]

When \(P(B)>0\), the above condition is equivalent to

\[ P(A\mid B)=P(A). \]

That is, knowing that \(B\) occurs does not change the probability that event \(A\) occurs.

2.2 Mutual Independence of Three Events

Definition 2.2 (Mutual Independence of Three Events)

Events \(A,B,C\in\mathcal{A}\) are called mutually independent if

\[ P(A\cap B)=P(A)P(B), \]

\[ P(A\cap C)=P(A)P(C), \]

\[ P(B\cap C)=P(B)P(C), \]

and

\[ P(A\cap B\cap C)=P(A)P(B)P(C). \]

Mutual Independence and Pairwise Independence

The first three equalities only say that \(A,B,C\) are pairwise independent. Pairwise independence generally cannot imply

\[ P(A\cap B\cap C)=P(A)P(B)P(C), \]

and therefore cannot imply that the three events are mutually independent.

3. Product Probability Spaces

Let

\[ (\Omega_1,\mathcal{A}_1,P_1) \quad\text{and}\quad (\Omega_2,\mathcal{A}_2,P_2) \]

be two probability spaces.

3.1 Product Sample Space

The Cartesian product of the two sample spaces is

\[ \Omega_1\times\Omega_2 =\{(\omega_1,\omega_2):\omega_1\in\Omega_1,\ \omega_2\in\Omega_2\}. \]

3.2 Product \(\sigma\)-Algebra

Definition 3.1 (Product \(\sigma\)-Algebra)

The \(\sigma\)-algebra generated by all measurable rectangles \(A_1\times A_2\) is called the product \(\sigma\)-algebra, denoted by

\[ \mathcal{A}_1\otimes\mathcal{A}_2 =\sigma\bigl(\{A_1\times A_2:A_1\in\mathcal{A}_1,\ A_2\in\mathcal{A}_2\}\bigr). \]

3.3 Product Probability Measure

Definition 3.2 (Product Probability Measure)

The product probability measure \(P_1\otimes P_2\) is the probability measure defined on \(\mathcal{A}_1\otimes\mathcal{A}_2\) and satisfying, for every measurable rectangle,

\[ (P_1\otimes P_2)(A_1\times A_2) =P_1(A_1)P_2(A_2). \]

Therefore, the product of two probability spaces can be written as

\[ (\Omega_1\times\Omega_2, \mathcal{A}_1\otimes\mathcal{A}_2, P_1\otimes P_2). \]

4. Random Variables and Induced Distributions

4.1 Borel \(\sigma\)-Algebra

Let \(\mathcal{B}(\mathbb{R})\) denote the Borel \(\sigma\)-algebra on the real line, that is, the \(\sigma\)-algebra generated by all open sets. It can also be generated by half-lines of the form \((-\infty,x]\).

For example, an open half-line can be represented as

\[ (-\infty,x) =\bigcup_{n=1}^{\infty} \left(-\infty,x-\frac{1}{n}\right]. \]

4.2 Random Variables

Definition 4.1 (Random Variable)

Let \((\Omega,\mathcal{A},P)\) be a probability space. A mapping

\[ X:\Omega\longrightarrow\mathbb{R} \]

if for every \(B\in\mathcal{B}(\mathbb{R})\) it satisfies

\[ X^{-1}(B) =\{\omega\in\Omega:X(\omega)\in B\} \in\mathcal{A}, \]

then \(X\) is called a random variable, and is also called a measurable mapping from \((\Omega,\mathcal{A})\) to \((\mathbb{R},\mathcal{B}(\mathbb{R}))\).

A random variable is not the naive concept of a “variable taking random values”, but a measurable function that maps abstract sample points to real numbers.

4.3 Induced Distribution of a Random Variable

Definition 4.2 (Induced Distribution)

The probability measure induced by a random variable \(X\) on \(\mathbb{R}\) is defined as

\[ P_X(B) =(P\circ X^{-1})(B) =P\bigl(X^{-1}(B)\bigr), \qquad B\in\mathcal{B}(\mathbb{R}). \]

\(P_X\) is called the distribution or probability law of \(X\).

5. Distribution Functions

5.1 Definition

In the definition of the induced distribution, take

\[ B=(-\infty,x], \]

to obtain the distribution function of the random variable.

Definition 5.1 (Distribution Function)

The cumulative distribution function of a random variable \(X\) is defined as

\[ F_X(x) =P_X((-\infty,x]) =P\{\omega:X(\omega)\leq x\} =P(X\leq x), \qquad x\in\mathbb{R}. \]

Hence

\[ F_X:\mathbb{R}\longrightarrow[0,1]. \]

5.2 Basic Properties of Distribution Functions

Proposition 5.2 (Properties of Distribution Functions)

Any distribution function \(F_X\) satisfies:

(i) Nondecreasing: If \(x_1<x_2\), then

\[ F_X(x_1)\leq F_X(x_2). \]

(ii) Right-continuous: For every \(x\in\mathbb{R}\),

\[ \lim_{h\downarrow0}F_X(x+h)=F_X(x). \]

(iii) Limits at the two ends:

\[ \lim_{x\to-\infty}F_X(x)=0, \qquad \lim_{x\to+\infty}F_X(x)=1. \]

Explanation of Proposition 5.2 (click to expand)

If \(x_1<x_2\), then

\[ \{X\leq x_1\}\subseteq\{X\leq x_2\}, \]

so by the monotonicity of a probability measure,

\[ F_X(x_1)\leq F_X(x_2). \]

For any sequence satisfying \(x_n\downarrow x\),

\[ \{X\leq x_n\}\downarrow\{X\leq x\}. \]

By the continuity of a probability measure for decreasing sequences of events,

\[ F_X(x_n)\longrightarrow F_X(x), \]

that is, \(F_X\) is right-continuous.

Finally, from

\[ \{X\leq x\}\downarrow\varnothing \quad (x\to-\infty) \]

and

\[ \{X\leq x\}\uparrow\Omega \quad (x\to+\infty), \]

one obtains the limits at the two ends, respectively. \(\square\)

6. Summary of This Section

This section establishes the basic framework of probability theory:

A probability space consists of a sample space, a \(\sigma\)-algebra, and a probability measure;
Independence can be characterized by the probability multiplication formula for joint events;
Product probability spaces are used to describe a joint experiment composed of multiple random experiments;
A random variable is a measurable mapping from the sample space to the real line;
A random variable induces a distribution on the real line through \(P\circ X^{-1}\), and its distribution function is \(F_X(x)=P(X\leq x)\).