10 Tips and Tricks for Statistical Proofs

I’ve been taking probability theory this year and I noticed that a lot of proofs will assume that the reader already knows some commonly used “tricks.” If you aren’t familiar with them, it can be hard to follow the proofs in the textbook,¹ let alone prove it yourself. I felt like this was happening to me a lot, so in an effort to better familiarize myself, I’ve written down some useful tips and tricks, along with explanations and/or examples.²

1. Bounding the Union with a Sum

• You can always bound the probability of a union by the sum of the probabilities: $$P\left({\cup }_n{A}_n\right)\le \sum _{n}P\left(A_n\right)$$ This is a very simple trick that uses the inclusion-exclusion principle. When in doubt, try this! It’s not a very strong upper bound, but it may be sufficient for your needs.

2. Cauchy-Schwarz Inequality

• The Cauchy-Schwarz Inequality is pretty much unavoidable in math. Here are some canonical forms: $$\begin{array}{cc}|⟨u,v⟩|& \le \parallel u\parallel \parallel v\parallel \\ {\left|\sum _i{u}_i{v}_i\right|}^2& \le \sum _j|u_j{|}^2\sum _k|v_k{|}^2\end{array}$$ A form you might encounter in statistics is the following: $$|E\left(XY\right){|}^2\le E\left(X^2\right)E\left(Y^2\right)$$

• Example: If $X\sim Poisson\left(\lambda \right)$ and $Y\sim Poisson\left(2\lambda \right)$, find a bound on $P(X\le Y)$ of the form $A\exp (-c\lambda )$. First observe that $P(X\ge Y)=P(e^{t(X-Y)}\ge 1)\le E\left(e^{t(X-Y)}\right)$ by Chebyshev’s inequality. Then $E\left(e^{t(X-Y)}\right)=E\left(e^{tX}{e}^{-tY}\right)\le \sqrt{E\left(e^{2tX}\right)E\left(e^{-2tY}\right)}$ by Cauchy-Schwarz. The last equation gives us $e^{\left(\lambda \right(e^{2t}-1)+2\lambda (e^{-\left(2t\right)}-1\left)\right)/2}$ with moment-generating functions. Take the derivative with respect to $t$ and plug back in to get the bound.

3. Continuous Mapping Theorem

• If $X_n\Rightarrow X_{\infty }$ (converges in distribution) and $g$ is a measurable function that is continuous except on a set of measure 0, then $g\left(X_n\right)\Rightarrow g\left(X_{\infty }\right)$. If $g$ is furthermore bounded, then $Eg\left(X_n\right)\to Eg\left(X_{\infty }\right)$. The result is often something you hope is true and you just need to check that the function you’re applying is in fact continuous.

4. Convergence Theorems

• A “last step” in many proofs is to take the limit inside the integral:

  $$\underset{n\to \infty }{lim}\int X_{n}d\mu =\int \underset{n\to \infty }{lim}{X}_{n}d\mu$$

  (An equivalent formulation is ${lim}_{n\to \infty }E\left(X_n\right)=E\left({lim}_{n\to \infty }{X}_n\right)$ or $E\left(X_n\right)\to E\left(X\right)$, where $X$ is the limit of $X_n$.)

• This is true under any of these three scenarios:

  1. If $X_n\ge 0$ and $X_n\le X_{n+1}$ for all $n$ (monotone convergence)
  2. If $\left|X_n\right|\le M$ for some constant $M$ (bounded convergence)
  3. If $\left|X_n\right|\le Y$ for an integrable random variable $Y$, i.e. $E\left|Y\right|<\infty$ (dominated convergence)

• Example (dominated convergence): Consider the probability space $(\Omega ,F,\mu )$. Let $X\ge 0$ be an integrable random variable and define $v\left(A\right)={\int }_{A}Xd\mu$ for $A\in F$. To show that $v$ is a measure, a necessary condition is that $v\left({\cup }_{i=1}^{\infty }{A}_i\right)={\sum }_{i=1}^{\infty }v\left(A_i\right)$ for disjoint $A_i$. This is true since

  $$\begin{array}{cc}\sum _{i=1}^{\infty }v\left(A_i\right)& =\underset{n\to \infty }{lim}\sum _{i=1}^{n}v\left(A_i\right)\\ & =\underset{n\to \infty }{lim}\sum _{i=1}^n{\int }_{A_i}Xd\mu \\ & =\underset{n\to \infty }{lim}\sum _{i=1}^n\int X{1}_{A_i}d\mu \\ & =\underset{n\to \infty }{lim}\int \sum _{i=1}^{n}X{1}_{A_i}d\mu \text{since}{A}_i\text{disjoint}\\ & =\int \underset{n\to \infty }{lim}\sum _{i=1}^{n}X{1}_{A_i}d\mu \text{since}\sum _{i=1}^{n}X{1}_{A_i}\le X\text{(dominated convergence)}\\ & =\int \sum _{i=1}^{\infty }X{1}_{A_i}d\mu \\ & ={\int }_{{\cup }_{i=1}^{\infty }{A}_i}Xd\mu \\ & =v\left({\cup }_{i=1}^{\infty }{A}_i\right)\end{array}$$

5. Expectation of Positive Random Variables

• If $X$ is a nonnegative random variable, then $E\left(X\right)={\sum }_{n=0}^{\infty }P(X\ge n)$ or $E\left(X\right)=\int P(X>x)dx$. A proof is available on Wikipedia. This is a really useful property and is often used for stopping times. More generally, $E\left(X^p\right)=\int p{y}^{p-1}P(X>x)dx$ for $X\ge 0$ and $p>0$.

• Example: If $\tau$ is a stopping time wrt $F_n$, s.t. $P(\tau \le n+3|F_n)>0.1$ for every $n$, then $E\left(\tau \right)<\infty$.

  This can be proved by first showing that $P(\tau >3k)\le (1-0.1{)}^k$, which can be done using induction. Then since $E\left(\tau \right)={\sum }_{n=0}^{\infty }P(\tau \ge n)$, it follows that $E\left(\tau \right)<3\ast {\sum }_{k=0}^{\infty }P(\tau >3k)<\infty$.

6. Jensen’s Inequality

• Jensen’s inequality is something you should be able to recognize and apply right away. Intuitively, it says that if a function $\varphi$ is convex, then the average of the function values is greater than the function applied to the average.³ $$E\left[\varphi \right(X\left)\right]\ge \varphi \left(E\right[X\left]\right)$$ One of the most common applications is with Euler’s number: $$E\left[e^X\right]\ge e^{E\left(X\right)}$$

• Example: If $X_n$ is a martingale, then $(X_n+c{)}^2$ is a submartingale. Let $g\left(x\right)=(x+c{)}^2$, which is convex. Then by Jensen’s inequality, $E\left(g\right(X_{n+1}\left)|F_n\right)\ge g\left(E\right(X_{n+1}|F_n\left)\right)$. Since $X_n$ is a martingale, $g\left(E\right(X_{n+1}|F_n\left)\right)=g\left(X_n\right)=(X_n+c{)}^2$.

7. Markov’s (Chebyshev’s) Inequality

• Markov’s inequality is often used whenever you need to bound the probability. It states that $$a^{2}P(|X|\ge a)\le E(X^2;|X|\ge a)\le E\left(X^2\right)$$ More generally, for $\varphi :R\to R$, $\varphi \ge 0$, $A\in B$, and $i_A=inf\left\{\varphi \right(y):y\in A\}$, $$i_{A}P(X\in A)\le E\left(\varphi \right(X);x\in A)\le E\left(\varphi \right(X\left)\right)$$

• Example: See the example for Cauchy-Schwarz (#2).

8. Slutsky’s Theorem

• This theorem states that if $X_n\Rightarrow X$ (in distribution) and $Y_n\to c$ in probability, then $X_n+Y_n\Rightarrow X+c$, $X_n{Y}_n\Rightarrow cX$, and $X_n/Y_n\Rightarrow Y/c$. It is not always easy to immediately see that you will need to use this theorem. However, you might be able to prove something close to the answer and realize that if you then apply Slutsky’s, the proof will be completed.

• Example: Let $X_1,X_2,\dots$ are iid with $E{X}_i=0$ and $E{X}_i^2={\sigma }^2<\infty$, $N_n$ b a sequence of nonnegative integer-valued random variables, and $a_n$ a sequence of integers with $a_n\to \infty$ and $N_n/a_n\to 1$ in probability. If $S_n={\sum }_{i=1}^n{X}_i$, show that $Y_n=S_{N_n}/\sigma \sqrt{a_n}\Rightarrow N(0,1)$.

  One way to prove this is to first show that $Z_n=S_{a_n}/\sigma \sqrt{a_n}\Rightarrow N(0,1)$, which is easy to do. Then if you can show that $Y_n-Z_n\to 0$ in probability, which is harder to do, you can conclude with Slutsky’s Theorem to say that $Y_n=(Y_n-Z_n)+Z_n\Rightarrow N(0,1)$.

9. Taylor Expansion & Euler’s Number

• The Taylor series is often used as an approximation and bound. In particular, the Taylor series for Euler’s number $e$ is very important: $$e^x=1+x+\frac{x^2}{2!}+\frac{x^3}{3!}+\cdots$$

• Another useful property of $e$ is that if $a_n\to a$, then $$e^a=\underset{n\to \infty }{lim}{(1+\frac{a_n}{n})}^n$$

• Example: See the example for characteristic functions (Bonus #2).

10. Truncating an Infinite Sum

• To get bounds on an infinite sum, you will generally want to truncate it $$\sum _{n=1}^{\infty }=\sum _{n=1}^N+\sum _{n>N}$$ The idea is that you can bound the two sums on the right individually. First, you can bound ${\sum }_{n=1}^N$ because the individual summands are bounded by some $ϵ$, so the whole sum is bounded by $ϵN$. Second, you can bound the rest of the sum ${\sum }_{n>N}$ as $N\to \infty$.

• Example: Let ${\mu }_n,\mu$ be probability measures on $N$. Show that if ${\mu }_n\left(x\right)\to \mu \left(x\right)$ for each $x\in N$, then the total variation distance $d({\mu }_n,\mu )=\frac{1}{2}{\sum }_{x=1}^{\infty }|{\mu }_n\left(x\right)-\mu \left(x\right)|\to 0$.

  If we just tried to bound the infinite sum directly using the assumption, we would get a bound of $ϵ\ast \infty$. Therefore, we need to split the infinite sum. $$\begin{array}{cc}\frac{1}{2}\sum _{x=1}^{\infty }|{\mu }_n\left(x\right)-\mu \left(x\right)|& \le \frac{1}{2}\sum _{x=1}^N|{\mu }_n\left(x\right)-\mu \left(x\right)|+\frac{1}{2}\sum _{x=N}^{\infty }|{\mu }_n\left(x\right)-\mu \left(x\right)|\\ & =A_n+B_n\end{array}$$ Now we can bound the finite sum “$A_n$” and the rest of the sum “$B_n$” separately.

  1. Since ${\mu }_n\left(x\right)\to \mu \left(x\right)$, bound $|{\mu }_n\left(x\right)-\mu \left(x\right)|<{ϵ}_1$ for $n\ge N_1$. Thus, $A_n<{ϵ}_{1}N$.
  2. Since ${\mu }_n,\mu$ are probability measures, then both $P(|X_n|>N)$ and $P(|X|>N)\to 0$ as $N\to \infty$, where $X_n$ has distribution ${\mu }_n$ and $X$ has distribution $\mu$. In other words, ${\sum }_{z=N}^{\infty }{\mu }_n\left(z\right)<{ϵ}_2$ and ${\sum }_{z=N}^{\infty }\mu \left(z\right)<{ϵ}_2$ for $N$.⁴ Then ${\sum }_{z=N_2}^{\infty }|{\mu }_n\left(x\right)-\mu \left(x\right)|<2{ϵ}_2$. Thus, $B_n<{ϵ}_2$.

  To conclude, we have shown that $A_n+B_n<{ϵ}_{1}N+{ϵ}_2\to 0$. Since $N$ is finite and we can make ${ϵ}_1,{ϵ}_2$ arbitrarily small, then $A_n+B_n\to 0$.