The three theorems of complete statistics

The Lehmann-Scheffe theorem

In the last article, we discussed the Rao-Blackwell theorem, which allows us to improve any estimator by averaging on a sufficient statistic (i.e. transforming it into a function of a sufficient statistic). Furthermore, the maximum improvement is attained by averaging on a minimal sufficient statistic, and the resulting estimator cannot be improved by further Rao-Blackwellization -- it is the best estimator "of that class".

One may wonder if averaging an unbiased estimator on a particular MSS could lead to the best estimator overall. 

Let's call such an MSS a complete sufficient statistic, or CSS. What we really require, then, is the uniqueness of the Rao-Blackwellization on a CSS -- then no other estimator could possibly be better, because you can't be better than your Rao-Blackwellization. 

We can then "reverse mathematics" out the definition of a complete statistic: $T$ is called a complete statistic if there is at most one unique unbiased estimator of $\theta$ that can be written as a function of $T$. Equivalently, there is a unique unbiased estimator of 0 that can be written as a function of $T$.

It is in fact sufficient for the estimator to be unique up to disagreement on a set of measure zero -- i.e. the probability of two estimators disagreeing is zero.

The statement that an unbiased CSS is the best estimator overall is known as the Lehmann-Scheffe theorem.

---

Basu's theorem

Often the data provides information on stuff other than the parameter of interest. We cannot quite ask for a sufficient statistic that provides information only on $\theta$ (by the law of multiple explanations, this is impossible unless the statistic determines $\theta$ with certainty), but there are various ways we can ask that irrelevant information be "minimal". 

One is in the definition of an MSS, as in the last article. The other has to do with something known as ancillary statistics.

An ancillary statistic is a statistic that provides no information on $\theta$ -- e.g. $X-\bar{X}$ does not provide information on $\mu$ in a normal model. So we might want to ask for a sufficient statistic such that no function of it is ancillary. Well, this is implied by completeness but is still not as strong as completeness -- it is possible to have statistics which aren't independent of $\theta$ but still have constant mean.

In fact, a stronger result, known as Basu's theorem, is implied by completeness: not only is no function of a CSS ancillary, but a CSS is independent of any ancillary statistic $V$. This follows from the fact that $P(V|T)-P(V)$ is itself a function of $T$, and thus its mean being 0 implies its triviality.

---

Bahadur's theorem

In the above theorems, CSSs seems like everything we wanted from MSSs. In fact, every CSS is an MSS.

Sufficient statistics and the Rao-Blackwell theorem

The motivation for the definition of a sufficient statistic is very obvious from a Bayesian standpoint -- we call a statistic $T(X)$ sufficient for $\theta$ if it carries all the information that $X$ has to offer on $\theta$, i.e. if

$$P(\theta|T\land X)=P(\theta|T)$$

(Or equivalently, since $T$ is a function of $X$, $P(\theta|T)=P(\theta|X)$), i.e. $\theta$ and $X$ are conditionally independent knowing $T$. An equivalent, more common formulation that $P(X|T)$ is independent of $\theta$:

$$P(X|T\land\theta)=P(X|T)$$

This captures the intuitive idea that $\theta$ is causally linked to $X$ only through $T$ -- from this perspective, the two formulations are precisely symmetric. 

One can find if a statistic is sufficient simply by looking at the distribution of $X$. We can write $P(X|\theta)=P(X|T(X))P(T(X)|\theta)$ for any statistic $T(X)$ -- thus for $T(X)$ to be sufficient, i.e. for $P(X|T(X))$ to be independent of $\theta$, we only require that such a factorization exists:

$$f(X|\theta)=h(X)g(T(X),\theta)$$

I.e. the distribution function can be written as the product of a function that doesn't depend on $\theta$ and a function that depends on $\theta$ and $T(X)$ but not $X$ directly. This is called the factorization criterion.

The interpretation of this theorem is that when doing Bayesian inference, two sets of data yielding the same value of $T(X)$ should yield the same inference about $\theta$ -- for this to be possible, the likelihoods' dependence on $\theta$ should only be in conjunction with $T(X)$ so that the direct dependence of the posterior on $X$ cancels out through the normalization factor.

We are further interested in sufficient statistics that are "minimal" in the sense that they don't carry any more superfluous information than other sufficient statistics -- "necessary and sufficient statistics", if you will. 

Can we define "necessary and sufficient statistics" as statistics that give no information other than that about $\theta$, i.e. such that knowing $T$ doesn't give us any additional information on $X$ than $\theta$? Consider if basic examples of minimal sufficient statistics have this property, and provide a simple counter-example.

We thus define a minimal sufficient statistic as a sufficient statistic that can be written as a function of any other sufficient statistic. So, for example, the sample mean may be a minimal sufficient statistic for the mean, but the entire sample itself is not, because it cannot be obtained from the sample mean alone.

A particular MSS of importance is the equivalence class of likelihood-ratios independent of $\theta$: one can partition the space of $X$s by the equivalence relation of $P(X|\theta)/P(Y|\theta)$ being independent of $\theta$ -- the resulting quotient function (that sends $X$ to its equivalence class) is then a minimal sufficient statistic for $\theta$.

As all MSSs can be recovered from each other, this is often used as a characterization of MSSs: $T$ is an MSS iff: 

$$T(X)=T(Y)\iff \frac{P(X|\theta)}{P(Y|\theta)}\text{ independent of }\theta$$

E.g. if two samples have the same mean, then the difference in their likelihoods comes from a factor other than the distribution mean.

---

Rao-Blackwell theorem

If an estimator $\hat{\theta}$ of $\theta$ takes different values for different $X$ with the same value of sufficient statistic $T(X)$, then it stands to reason that this estimator is sub-optimal, in that irrelevant features of the data contribute to the variance in $\hat{\theta}$. So for any sample $X$, we might want to take the average of $\hat{\theta}$ among all samples with the same value of $T(X)$, which would be a new estimator $\tilde{\theta}$.

This is known as the Rao-Blackwell theorem. Specifically, the Rao-Blackwell Theorem says that the new estimator $\tilde{\theta}=E(\hat{\theta}|T(X))$ has the same bias, and less or equal variance than $\hat{\theta}$ (with equality if $\hat{\theta}$ was already a function of $T$). The proof follows straightforwardly from the conditional breakdown of variance (ANOVA).

$$\mathrm{Var}(\hat{\theta})=\mathrm{E}(\mathrm{var}(\hat{\theta}|T))+\mathrm{var}(\mathrm{E}(\hat{\theta}|T))$$

Continuing this line of reasoning, we would like to average over the minimal sufficient statistic to be eliminate as much superfluous information as possible. Indeed, it is easy to show that if $T_2=h(T_1)$ are both sufficient statistics, then averaging over $T_2$ gives a lower variance estimator than averaging over $T_1$.