Dutch book argument for probability theory

aka "how to profit off people's irrationalities".

The philosophical question of what it means to be rational is answered by the popular slogan "Rationality is systematized winning" -- so, in particular, a rational belief is one that is expected to deliver you a profit.

This is closely linked to the "practical" definition of probability -- assigning probability $p$ to observation $X$ means valuing at $p$ a unit wager on $X$ (an asset with payoff 1 if $X$ happens, an 0 otherwise) -- you are willing to buy the wager for any price less than $p$, and sell it for any price higher. 

(We use money to make things simple, but more fundamentally we have a cardinal utility function whose expectation we seek to maximize -- so this is a valid abstract definition and does not depend on a specific currency or monetary system.)

However, if you accept that probability is definitionally subjective (even if updated by evidence via Bayes's theorem, your priors are still subjective), then assigning a probability $p$ to a particular event cannot itself be rational or irrational. You can't quite calculate whether someone who holds that belief will make a profit or loss, because doing so would require assigning an underlying "objective" probability to $X$. 

You can only say that a person is irrational if you can prove that their beliefs always lead to them losing, regardless of the results of the observation. Such beliefs are said to be incoherent, a notion that is identical to the financial concept of arbitrage. Irrationality in this sense is about an inconsistency in beliefs about different, related observations.

Consider observations $X_1,\dots X_n$ (in the language of probability theory, these are elements of our sigma algebra). Suppose you assigned to them the probabilities $p_1,\dots p_n$. 

Your beliefs are irrational iff I can come up with a list of unit wagers (called a Dutch book) to trade with you that always deliver me a profit regardless of the outcome of each observation.

For example, if $X_1\subseteq X_2$ and you assign probabilities $p_1=0.5,p_2=0.1$, then I can buy from you a unit wager on $X_2$ and sell you a unit wager on $X_1$, and always make a profit.

The axioms of probability theory can then be shown to be equivalent to forbidding a Dutch book. If the values of all bets are positive between 0 and 1, the value of the bet on the entire sample space is 1 and bets are additive, then it becomes impossible to construct a Dutch book. Conversely, a violation of any of these axioms allows a bet on something equivalent to the sample space to be traded for less than or greater than 1, allowing for a Dutch book.

Similar Dutch book arguments can be extended to conditionalization and Bayes's theorme.

Frequentism as a bureacuratic restriction on speech

Although it is traditional to claim that Bayesian and frequentist statistics are two "schools" of probability and statistics, there are actually two distinct ideas that are labeled "frequentist" (and analogously "Bayesian"):

• Frequentist probability: An interpretation of probability, an alternative to the subjective interpretation of probability normally associated with Bayesianism. Under the frequentist interpretation, probability is understood as having to do with frequencies of observations, e.g. a probability of 1/2 means that you can expect 50 coin tosses out of 100 to come up heads. In reality, this merely shifts the problem to interpreting expectation instead of probability.
• Frequentist statistics: A formulation of statistical inference, in which we don't really discuss probabilities of theories at all, but restrict ourselves to talking about tautological statements about the probabilities of observations given theories. The supposed advantage of this is that we don't need to follow unproven beliefs about priors.

In the Bayesian view, the frequentist interpretation of probability appears as follows: given a random variable $X$, we sample a sequence of IID variables $X_i$ -- the "limiting" distribution of these $X_i$s is then seen as the probability distribution of $X$.

(From a "positivist"/"scientific" viewpoint, it is rather absurd to define the probability of heads on a coin-flip in terms of a large number of other coin flips. After all, it is only an approximate assumption that all these coin flips are distributed identically. The purest way to talk about IID variables would be to consider an infinite number of universes, each in which a coin flip occurs. Of course, if the universes were identical in every way, the coin flips would be the same in every universe and the probability of heads will be 0 or 1. Probability arises from the fact that we have limited information about the state of the universe -- thus the average is not being taken over all identical universes, but over all universes which match the limited information we have. This "purification" of frequentism is simply Bayesian probability, as the "set of universes" is just the sample space, and the distribution of the unknown information is a Bayesian prior.)

The reason that frequentist statistics is typically associated with frequentist probability is that frequentist probabilists see it as odd to talk about the frequency of a theory being true -- to them, the same theory is true in each experiment conducted, since frequentist probability does not consider the situations in "alternate universes". Of course, this distinction between theory and observation is completely arbitrary -- on a logical level, theories and observations are both just logical statements, and analogously parameters and observed quantities are both just random variables.

To "forbid" talking about the probability of a theory is simply a bureaucratic restriction on what one can say, not something that affects your decisions in any way. When it comes to making decisions, one has to compute expected utilities, which requires a Bayesian approach (because frequentism does not consider "multiple universes"). Even if you don't say the probability of a theory is 80%, and only say that the probability of some observation would be 95% given that theory, the fact that your discipline "accepts" a p-value of 0.05 itself implicitly means that you believe the theory to be true with sufficient certainty.

Lossless compression, invertible change of variables and entropy

In the last article, we introduced the notion of information, and defined a bit as a unit of information that halved the probability mass on the sample space. We used that to more generally define the information associated with an observation, in intuitive terms of a "number of bits" it possesses.

$$I(X=x)=\log_{1/2}(P(X=x))$$
All this makes perfect sense of course, when talking about a sample space of IID uniform binary sequences (e.g. fair coin tosses), then discovering each term in the sequence (i.e. a bit) literally gives you one bit of information.

But e.g. consider the sample space $\{HH,HT,TH,TT\}$ and consider receiving the information that the result is either $HH$ or $TT$. You haven't actually received a physical bit, but you have halved your sample space, so you have received an information-theoretic bit. 

That's uncomfortable. Maybe we should start looking for the source of our discomfort.

Well, out of all the words in italic sequence above, the only phrase I'm uncomfortable is "physical bit". That's a term that makes sense when we're talking about physical computers, but what does it really mean in an abstract, mathematical, information-theoretic sense? What exactly does it mean to receive the "physical bit" $H$, and how does it differ from receiving information like "the two tosses are the same"?

And then you realize, it doesn't differ. "Receiving the physical bit $H$" just means receiving the information that the first bit is an $H$. That's all the bit means. There is no such thing as a physical bit. (I mean, there is -- we'll use "physical bit" interchangeably with "character" from now on, but the notion of a physical bit particularly having to be something like "$H$" or "$T$", rather than some arbitrary encoding, is something arbitrary.)

Instead of encoding our sample space in terms of "(what was the first coin toss?)(what was the second coin toss?)", we could encode our sample space in terms of "(were the two coin tosses the same?)(what was the first coin toss?)". There is absolutely nothing fundamental about either encoding. And the notion of a physical bit doesn't really exist (think of unfair coins), all we have is the abstract idea of a bit.

This is of course completely expected -- most of our data (not on computers) isn't even in the form of binary sequences.

What we have is an invertible change of variables between two encodings.

---

Often what determines our choice of encoding are cost considerations.

What do I mean? Suppose we have two coin tosses, as before, but instead of being fair, the probability of a heads is 90%. Then HH will appear 81% of the time, HT and TH will both appear 9% of the time, and TT will appear 1% of the time.

Then consider the following two encodings:

• (first toss is an H?)(second toss is an H?)
• (result is HH?)(else: result is HT?)(else: result is TH?)

In the first encoding, the data will always occupy 2 characters. 

In the second encoding, the data occupies 1 character 81% of the time, 2 character 9% of the time and 3 character 10% of the time, i.e. an expected value of 1.29 characters.

This sort of knowledge about probability distributions is used for data compression all the time. For example, images are very likely to have long, continued strings of the same colour, so rather than writing an extra character for each pixel, you just store information like "5 reds, 6 greens..." (this is the idea behind run-length encoding, by the way).

This use of change of encoding is known as data compression -- and the basic idea is to get more common strings to have shorter encodings (the probability of a string's occurrence captures the probabilities of each individual character and the correlations between the characters). 

(and we can have a non-invertible change of variables too, of course, which leads to lossy compression, where we project our data by discarding dimensions we don't care about.)

---

The natural question to ask is what the optimal data encoding should be. This is a very fundamental question -- about how much information is actually present in an object, how many Yes/No questions are actually needed, on average, to specify this particular object in the sample space.

(And if you don't understand why these are the same question: you should re-read the first article in this course: Information and bits.)

Well, the formulation of the problem basically spells out "binary tree". We need to organize the elements of our sample space into a binary tree so that the expected number of yes/no questions we need to ask/bits we need to provide are minimized, so these physical bits are actual bits of information. We organize our data space into a binary tree, with each split (bit) dividing the data into two halves of the probability mass -- this way each physical bit (character) represents an actual bit of information.

Such an encoding -- an optimal binary tree -- is known as entropy encoding; here, the optimal length of code for an object is precisely equal to its information, $\log_{1/2}(P(X))$. As a result, the expected number of bits necessary to specify an object is $\sum P(X)\log_{1/2}(P(X))$, which is known as the entropy of the system.

Simple entropy encodings include the Huffman coding and the Arithmetic coding.

---

In machine learning, we are often more interested in lossy compression -- the problem of determining the optimal representation for a given expected value of "distortion" (calculated via a loss function). Often practically, we have a given size (rate) of representation and calculate the minimum expected loss (distortion) that can be achieved through a compression algorithm.

In general, the distortion is represented by a loss function $d(x,\hat{x})$ -- this loss function is not an information theoretic construct, but derives from the practical motivation to reduce distortion, i.e. the preferences/utility of the decision-making agent choosing the compression algorithm. Typical loss functions include the Hamming (discrete/Dirac) distortion $\delta(x\ne\hat{x})$, appropriate for symbols that are essentially categorical, the mean-squared-error, appropriate for continuous variables, the cross-entropy, appropriate for probability distributions etc.

One may imagine that in the encoder-decoder setting as above, $Y$ should be a lossless encoding  for $\hat{X}$ and thus its length should be the entropy of $\hat{X}$. However, in general, we are only concerned with the information that $\hat{X}$ provides on $X$, i.e. the mutual entropy $I(X;\hat{X})$. Thus the rate-distortion function -- representing the minimal rate for a given distortion -- ought to be given by $R(D)=\inf\limits_{\hat{X}\mid E(d(\hat{X},X)) < D}I(X;\hat{X})$, where the minimization is over probability distributions $p(\hat{X}\mid X)$.

Information and bits

When we deal with questions of statistical inference, we often talk in terms of information -- we say that an observation provides information, that interferes through the Bayes mechanism with our prior distribution to form the posterior.

But what exactly does "information" mean? Surely there must be a way to quantify it, to be able to compare two sources of information based on the "amount" of information they provide.

Well, consider our sample space, say for an experiment that involved tossing three coins:

$$\{HHH, HHT, HTH, HTT, THH, THT, TTH, TTT\}$$
"Information" would be something like the statement "the first toss was a heads", or "the first two tosses yielded different results", or "the tosses were not all heads". This is information, because this narrows down our sample space. So we can think of information as being quantified by the fraction of hypotheses eliminated, or more precisely: the fraction of probability mass eliminated.

"But statistical inference isn't just about eliminating hypotheses!" It kind of is, actually. Observations are about restricting your sample space, i.e. completely eliminating specific possible observations. I.e. even when applying Bayes's theorem $P(Y|X=x)=P(X=x|Y)P(Y)/P(X=x)$, the statement $X=x$ is a certain fact, and all inferences made through Bayes's theorem are a recalculation of probabilities on this restricted sample space.

It's generally common (for reasons that will become obvious later in the article on data compression) to represent the amount of information in terms of bits.

Each bit of information eliminates half the probability mass upon being found to be true. 

Intuitively, one may imagine that we're attempting to identify some data in the form of a binary sequence 11001????????... where each sequence has an equal probability. Discovering one new bit halves the space of possible sequences. Of course, our data may not be in this form, something we will get to in the data compression article.

Or in general, the quantity of information carried by an observation $X=x$ is defined as:

$$I(X=x)=\log_{1/2}(P(X=x))$$
So this notion of information captures our intuitive idea that "the unlikelier a fact seems, the more information it will provide us if it turns out to be true/the more it will change our beliefs".

The Dirichlet (also Beta) distribution

Here's a category of distributions we may often want: a distribution on the simplex.

I.e. a multivariate distribution on $n$ nonnegative numbers that add up to 1. This is something that we can definitely see using as prior distributions on parameters that can be interpreted as "probabilities" of something. One can see that this is important for the Categorical and Multinomial distributions, for instance: and in the case of two numbers (i.e. a univariate distribution, since it's on a line segment), for the Bernoulli and Binomial distributions.

Here's one such distribution family that may come to your mind: for $x_i$ in the simplex $\sum_i x_i = 1$,

$$f(x_1,\dots x_n\mid\theta_1,\dots\theta_n)\propto \prod_{i}{x_i}^{\theta_i}$$
By adjusting the values of the $\alpha$s, one can get suitable priors that represent our beliefs correctly. This is known as the Dirichlet distribution, and its univariate case $f(x|\theta_1,\theta_2)\propto x^{\theta_1}(1-x)^{\theta_2}$ is known as the Beta distribution.

In fact the parameters of said distributions are usually provided a bit differently, with $\alpha-1=\theta$.

Exercise: Prove that:

• The normalization constant is given by $\frac{\Gamma\left(\sum_i\alpha_i\right)}{\prod_i \Gamma(\alpha_i)}$
• The mean is given by $E(X_i)=\frac{\alpha_i}{\sum_i\alpha_i}$
• The Dirichlet distribution is the conjugate prior to the categorical/multinomial distribution. This is the key fact that makes the Beta/Dirichlet distribution important.

Poisson processes: from geometric to Gamma distributions

The notion of a Poisson process is rather beautiful, and connects a number of distributions together.

To start, consider a Bernoulli process: a discrete time series with each value IID Bernoulli. Then we can study some properties of this:

• The waiting time for the 1st event is distributed geometrically. 
• The waiting time for the nth event is distributed as the sum of geometric distributions, which is negative binomial.
• The number of events in a given period of time is distributed binomially.

(You should be able to derive these distributions easily.)

(Note that this is not specific to a time series -- one has the same results for a spatial lattice or for some general abstract set of points.) 

How would one generalize this to continuous time? Well, with continuous time you really can't talk about the result for each point in time being "Bernoulli", or about them being IID. But let's do it anyway. Suppose an event has an $\mu\; dt$ chance of occurring in the timespan $dt$. Then the chance that the first event occurs at time $t$ is (by geometric distribution) ${(1 - \mu \;dt)^{t/dt}}\mu \;dt$, or: $\mu {e^{ - \mu t}}\; dt$, i.e. a probability density $\mu e^{-\mu t}$. This is called the exponential distribution.

The waiting time till the nth event is analogously just the sum of exponential random variables and its distribution can be computed through the standard MGF route. It is left as an exercise to the reader to show that the sum of exponential random variables with parameters $\mu_1,\dots\mu_\alpha$ is given by the Gamma distribution $\Gamma(\alpha,\sum\mu_i)$:

$$\Gamma(\alpha,\beta)\sim \frac{1}{(\alpha - 1)!}\beta^\alpha t^{\alpha-1}e^{-\beta t}$$
What does our notion of independence translate to? The idea that there being an event at some time should not depend on whether there was at any other time. Well, the natural way to write independence in a way that makes sense for continuous distributions is to consider the waiting time for the first event

\[P(T>t+s|T>t) = P(T>s)\]
This is known as memorylessness. Indeed, one can check that the only memoryless discrete distribution is geometric, and the only memoryless continuous distribution is exponential.

OK -- what about the number of events in the continuous case? Well, in some interval of size $T$, the probability of the number of events equaling some $n$ is (by binomial):

\[\left( {\begin{array}{*{20}{c}}
  {T/dt} \\
  n
\end{array}} \right){(\mu \,dt)^n}{(1 - \mu \,dt)^{T/dt}}\]
Which it is easy to see that equals:

\[\frac{{{{(\mu T)}^n}{e^{ - \mu T}}}}{{n!}}\]
Which is the Poisson distribution with rate parameter $\mu T$.

Here's a table of analogies between memoryless discrete and continuous processes:

	Discrete time	Continuous time
Overall phenomenon	Bernoulli process	Poisson process
Single-event	Bernoulli distribution	-
Waiting time (1st event)	Geometric distribution	Exponential distribution
Waiting time (nth event)	Negative Binomial distribution	Gamma distribution
Number of events	Binomial distribution	Poisson distribution

But isn't the continuous analog of the binomial distribution (and many other distributions) the normal distribution? Do not conflate discrete time with discrete number. Both the binomial and Poisson distributions above are discrete distributions: the Poisson is just the relevant one for the continuous-time process. These are completely unrelated notions.

Probabilistic inequalities

Consider a random variable with mean 0 and variance 1 (this is like the "natural units" of second-moment probability). Now this variance puts a value on how dispersed the PDF of the random variable can be: one couldn't, for example, have two Dirac-delta poles really far from the origin, because you can calculate the variance of that, and it's too high.

Which raises the question: for some $k$, what's the maximum fraction of the distribution that can be outside $(-k,k)$?

Ok, one thing is clear: at this maximum, there should be nothing outside $[-k,k]$, because then you could bring inwards without changing the fraction but reducing the variance.

Also, for any value in $(-k,k)$, if you moved it over to the mean (0), the variance would go down. So the distribution must be comprised of just three poles at 0, $-k$ and $+k$, and is necessarily symmetric so that the mean is at 0. Letting $p/2$ be the height of each pole at $k$, the variance in terms of $p$ is $pk^2$. So $pk^2=1$, and $p=1/k^2$. I.e.

$$P\left(\left|X\right|>k\right) \le 1/k^2$$
Or for general mean and variance:

$$P\left(\left|\frac{X-\mu}{\sigma}\right|>k\right) \le 1/k^2$$
This is Chebyshev's inequality, and gives you a limit on how much of the distribution can be some given distance $k$ from the mean. Note how it only becomes interesting for large $k$ ($k>1$).

Well, clearly this approach seems to open up a whole world of similar inequalities. Another is the Markov inequality, which states that (for a nonnegative random variable) no more than $1/k$ of a population can have value more than $k$ times the mean, i.e.

$$P\left(X\ge k\right)\le\frac{\mu}{k}$$
(Justify this with similar reasoning as Chebyshev's.)

In fact, Chebyshev's inequality can be derived as a special case of Markov's (do it).

The point of these inequalities is that means and variances are generally easy to track, even when probability distributions are unknown. Providing bounds on probability fractions based on these is very useful for proving convergence in probability -- for example, the weak law of large numbers becomes elementary with Chebyshev's inequality.

Probabilistic convergence

The law of large numbers is something that we know, that in our heads is almost the definition of probability (it's not): "as sampling increases, the average of a variable approaches its expected value". I.e. for $X_i$ IID:

$$\lim_{n\to\infty}\frac1n \sum{X_i}=\mu$$
Let's think about what this statement really says: when you take more and more readings of $X$, the average will go closer and closer to $\mu$. But the values of these readings are inherently probabilistic: this is not an actual sequence of real numbers you can take the limit of. Rather, you are talking saying that of all the possible realizations (which are real number sequences), almost all of them (probabilistically) converge to the thing. I.e.

$$\mathrm{Pr}\left[\lim_{n\to\infty}X_n=X\right]=1$$
This is known as almost sure convergence.

In general, the thing on the right could've been a random variable, rather than a real number. And here's where some probability theory (read the article) comes in, because the random variables $X_n$ and $X$ need to be defined on the same sample space for this to make sense (i.e. it's not just about the distribution).

But with this, the definition as above still works: as an example, consider the sample space $[0,1]$ and consider a sequence of random variables $X_n$ that is respectively 1 on some corresponding sequence of sub-intervals approaching $[0,1/2]$. Then this approaches the random variable that is 1 on $[0,1/2]$ almost surely.

And yes, this is entirely due to the correlations between these things.

In any case, almost sure convergence isn't really the best way to express random variables converging to each other, as you can see. E.g. the central limit theorem -- like $\frac{1}{\sqrt{n}}\sum\frac{X_n-\mu}{\sigma}\sim N(0,1)$, cannot be phrased in terms of almost sure convergence, because $N(0,1)$ is a distribution, not a random variable.

Indeed, you may have figured that the problem of a random sequence converging to a random variable is somewhat similar to the notion of "functions converging to a function" -- indeed, one may think of the distributions of the random variables in the sequence and discuss their convergence. I.e.

$$F_n(x)\to F(x)$$
This is called convergence in distribution.

While convergence in distribution does not imply almost sure convergence in general as we've seen, we would expect that it does imply it in the case where the limiting random variable is constant (because then issue of correlations disappears).

But you may realize that this is not really so: a sequence may look increasingly like something without actually limiting to it. For example, think about a sequence like 1, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1... with an infinite number of 1s, but decreasing in frequency. This doesn't limit to zero. If this were a deterministic sequence, this would never be expected to limit to 0 as the positions of the 1s would be hardcoded into the generation of the sequence. However, the sequence can also be realized as a realization of a sequence of random variables $X_n$ that have probability $1/n$ of being 1. Then the $X_n$ converge in distribution to 0, but their realizations almost never (thus in particular don't almost surely) converge to 0.

So it seems that asking for realizations to almost surely converge to the right thing is a bit too strong for a lot of purposes. A weaker notion of convergence than almost sure convergence can be constructed by considering probabilities of each $X_n$ separately rather than as a sequence: $X_n$ converges to $X$ if each $X_n$ is in the limit almost surely arbitrarily close to $X$. Or more precisely:

$$\lim_{n\to\infty}\mathrm{Pr}\left(\left|X_n-X\right|<\varepsilon\right)=1$$
This is known as convergence in probability. Indeed:

1. Almost sure convergence implies convergence in probability (obviously).
2. Convergence in probability implies convergence in distribution (because they are both topological notions of convergence and the map from a random variable to its distribution is continuous).
3. When the limit random variable is constant, convergence in distribution implies convergence in probability.

In fact, the law of large numbers that we stated above (in terms of almost-sure convergence) is the strong law of large numbers, while the weak law of large numbers only states convergence in probability.

---

Exercise:

Prove Slutsky's lemma: given $X_n, Y_n$ converge to $X,y$ in probability and $y$ is a constant random variable:

1. $X_n+Y_n$ converges to $X+y$ in probability.
2. $X_nY_n$ converges to $Xy$ in probability.
3. $X_n/Y_n$ converges to $X/y$ in probability.

Why is it necessary that $y$ be a constant?

Two-envelopes problem: beyond the Bayes explanation

The two-envelopes problem is phrased as follows:

There are two envelopes, one with twice the amount of money as the latter -- you pick one of them at random and open it to find $x$ dollars. You don't know if this was the larger or smaller sum.

Should you switch?

Though symmetry would tell you that it doesn't matter, it seems that by switching you get $x$ if the other envelope is bigger, but only lose $x/2$ if the other envelope is smaller. So you should switch. Right?

The standard explanation to this problem is that we need to pick a prior to make any actual inference, and the probability of the other envelope being bigger is usually less than the probability of it being smaller, in the sense that probability distributions approach zero towards infinity.

But this explanation doesn't really make the problem go away. Consider this prior:

$$
f(x) = \left\{ {\begin{array}{*{20}{c}}{\frac{1}{4}{{(3/4)}^n}}&{{\rm{if }}\ x = {2^n},\,\,n \in {\mathbb{Z}_{ \ge 0}}}\\0&{{\rm{else}}}\end{array}} \right.
$$
This is a perfectly valid prior distribution (the improper prior assumed in the original "paradox" would have worked as well, but I don't want to distract you with questions about distribution propriety) for which exactly the same paradox applies -- one may show (and I encourage you to work this out) that whatever value $x$ I find in the envelope, the expected value of the other envelope is $8/7\ x$.

Let's try to remember why this is weird -- what exactly is so paradoxical about it?

What's weird is that before opening your envelope, the situation is completely symmetrical. Yet you know that when whatever happens once you open the envelope, you'll want to switch.

Well, that's weird, but the reason it's paradoxical is this: since each expectation $E(X_2\mid X_1=x_1)$ (for varying possible $x_1$) is greater than $x_1$, the expectation of $X_2$ should itself be greater than the expectation of $X_1$, right? It's a tree diagram, you're just summing over each possible value of $X_1$.

How can $E(X_2)=8/7\cdot E(X_1)$ but at the same time $X_1$ and $X_2$ are completely symmetric?

Hopefully some light bulbs are turning on in your head -- indeed:

$$E(X_1)=\sum_{n\in\mathbb{Z}_{\ge0}}\frac14\left(\frac34\right)^n2^n=\infty$$
Yep. That's why the situation is symmetric. Because eight sevenths of an infinity is still an infinity.

Covariance matrix and Mahalanobis distance

In the article Random variables as vectors, we discussed that random variables were vectors, and their covariance was their dot product.

The basic motivation for coming up with this idea was from contrasting $\mathrm{Var}(X+X)=4\mathrm{Var}(X)$ to the formula for variables with zero covariance $\mathrm{Var}(X+Y)=\mathrm{Var}(X)+\mathrm{Var}(Y)$. These correspond to the geometric cases of adding two parallel and perpendicular vectors -- a more general addition is expressed through the cosine rule. What's the "cosine rule for random variables"?

Well, it's $\mathrm{Var}(X+Y)=\mathrm{Var}(X)+\mathrm{Var}(Y)+2\mathrm{Cov}(X,Y)$. To me, this -- like the dot product form of the cosine rule -- is highly suggestive of a bilinear form, specifically the Gram matrix, called the covariance matrix, of the random vector $\mathbf{X}=\left[ {\begin{array}{*{20}{c}}X\\Y\end{array}} \right]$ (which is really to be seen as a "matrix", because the random variables are to be understood as row vectors).

$$\Sigma ({X_1}, \ldots {X_n}) = \left[ {{\rm{Cov}}({X_i},{X_j})} \right]$$
One may compare this Gram matrix interpretation -- $\Sigma=\mathbf{X}\mathbf{X}^T$ (note: not $\mathbf{X}^T\mathbf{X}$, the way we've defined $X$ -- this is important!) -- to the variance formula $\sigma^2=XX^T$, and realise that the covariance matrix is the "right" measure of variance of a random vector (note how if we made random variables column vectors, this would all become $X^TX$, etc.).

(yeah, yeah, you need to subtract the mean, etc.)

Analogously, one may define a cross-covariance matrix $K_{\mathbf{X}\mathbf{Y}}=\mathrm{E}((\mathbf{X}-\mu_{\mathbf{X}})(\mathbf{Y}-\mu_{\mathbf{Y}})^T)$ measuring the covariance between two random vectors.

---

It is rather natural to see this, being a bilinear form, as related to some notion of distance -- the standard deviation, after all, can be seen as a "natural distance unit" in one dimension (in the sense that the "unlikeliness" of a data point depends on its distance from the mean in units of standard deviation).

Suppose we wish to find the variance across some direction, i.e. the variance of some random variable $u_1X+u_2Y=\mathbf{X}\hat{u}$ with $|\hat{u}|=1$ -- this is clearly just $\hat{u}^T\Sigma\hat{u}$. So this defines a natural distance scale in the direction of $\hat{u}$, so that the norm of a vector $\vec{v}$ is defined as:

$$\|\vec{v}\|=\frac{\vec{v}^T\vec{v}}{\hat{v}^T\Sigma\hat{v}}$$
It is not too hard to show -- from the bilinearity of the expression -- that this is equivalent to:

$$\|\vec{v}\|=\vec{v}^T\Sigma^{-1}\vec{v}$$
Another way to interpret is this $\Sigma^{-1}$ maps the distribution into a spherical one (one with identity covariance matrix), and this norm is just the norm of the data point in this spherical distribution, which is adjusted for variances and covariances. This measure of distance is called the Mahalanobis distance.

Introduction to Bayesian inference

When the Higgs boson was discovered by the LHC, you heard comments e.g. here that "there is no way to definitively talk about the probability of the existence of a Higgs boson" -- more specifically, the quoted physicist in the linked article claimed "there is no way eliminate the conditional".

This is just wrong. There is a way to eliminate the conditional -- it's just Bayes's theorem!

Imagine setting up a tree diagram for this experiment. (The LHC made a certain observation which is highly correlated with the existence of the Higgs boson, which is what we're expressing below.)

As a graphical interpretation of Bayes's theorem:

$$\mathrm{Pr}(\mathrm{Theory}\mid\mathrm{Data})=\frac{\mathrm{Pr}(\mathrm{Data}\mid\mathrm{Theory})}{\mathrm{Pr}(\mathrm{Data})}\mathrm{Pr}(\mathrm{Theory})$$

What this needs, however, is a prior distribution $\mathrm{Pr}(\mathrm{Theory})$ on the probability of the Higgs boson existing -- before any data is supplied.

And this distribution obviously exists. If you've ever heard a physicist say "we were already pretty sure it existed, this experiment just increased our confidence in it", then you know what I'm talking about. This "pretty sure"-ness in its existence is a prior distribution.

OK -- but how does this prior distribution exist? Why were physicists pretty sure that the Higgs boson existed?

Well, there's a reason they postulated a Higgs boson in the first place -- to explain the mass of certain particles, etc. And they had already observed that those particles have a mass -- i.e. they had already collected some data.

This prior distribution is the posterior distribution of a previous experiment.

---

But... you're saying that the prior distribution is inferred as the posterior distribution of a previous experiment, which itself depends on the prior distribution of that experiment... isn't it just turtles all the way down?

Well, as you might have guessed, we need a true prior distribution on all physical theories about the universe, before collecting any data. This prior is essentially arbitrary, which might be philosophically troubling for you, but that's just how things are. There is no right prior distribution, it's fundamentally subjective.

But I'd say most people would adopt a prior distribution based on Occam's razor, giving higher priors to simpler theories -- e.g. even though all existing data agrees with both "string theory" and "string theory until New Year 2020, and then the world ends", we give higher priority to the first one, simply because it takes "fewer lines of code" to write. If you're interested in this stuff, I recommend reading about:

• Occam's razor
• Kolmogorov complexity
• Solomonoff's theory of inductive inference

I suppose Solmonoff's is not the only way to set priors -- a principle that I have heard of but do not really understand (and don't bother with, because it seems really boring) is the principle of indifference.

---

This perspective on inference, though, requires the following key fact about Bayesian inference, known as the stability of Bayesian inference: feeding some data, then feeding some other independent data, is equivalent to feeding both data together (the independence is important, because e.g. if you fed in the same piece of data twice, that should not affect your distribution). We can prove this fairly easily:

Given a prior distribution $\mathrm{B}_0(\theta)$ on theory-space values $\theta$, the posterior distribution upon observing some data $\delta_1$ is:

$$\mathrm{B}_1(\theta)=\frac{\mathrm{P}(\delta_1\mid\theta)}{\sum_\phi \mathrm{P}(\delta_1\mid\phi)\mathrm{B}_0(\phi)}\mathrm{B}_0(\theta)$$
Which then becomes the prior distribution for a subsequent observation of $\delta_2$:

$$\mathrm{B}_2(\theta)=\frac{\mathrm{P}(\delta_2\mid\theta)}{\sum_\phi\mathrm{P}(\delta_2\mid\phi)\mathrm{B}_1(\phi)}\mathrm{B}_1(\theta)$$
Substituting in $\mathrm{B}_1(\theta)$ and simplifying:

$$\mathrm{B_2}(\theta)=\frac{\mathrm{P}(\delta_1\land\delta_2\mid\theta)}{\sum_\phi\mathrm{P}(\delta_1\land\delta_2\mid\phi)\mathrm{B}_0(\phi)}\mathrm{B}_0(\theta)$$
Which is precisely what we wanted.

Apparently there are systems of probability theory called noncommutative probability in which this is not possible, and statistical inference is not possible -- see (Redei 1992) (isn't it weird how everything in probability and statistics is so recent?). Obviously, this is not relevant to the physical applications of probability.

---

If you want to see Bayesian inference in action, have a look at this interactive RShiny applet I wrote that demonstrates Bayesian inference from a continuous stream of data (relying on this stability of Bayesian inference). Here's a snapshot from the applet, to pique your interest, of the evolution of this belief distribution while tossing a coin that you gradually learn is pretty unfair:

Sigma fields are Venn diagrams

The starting point for probability theory will be to note the difference between outcomes and events.

An outcome of an experiment is a fundamentally non-empirical notion, about our theoretical understanding of what states a system may be in -- it is, in a sense, analogous to the "microstates" of statistical physics. The set of all outcomes $x$ is called the sample space $X$, and is the fundamental space to which we will give a probabilistic structure (we will see what this means).

Our actual observations, the events, need not be so precise -- for example, our measurement device may not actually measure the exact sequence of heads and tails as the result of an experiment, but only the total number of heads, or something -- analogous to a "macrostate". But these measurements are statements about what microstates we know are possible for our system to be in -- i.e. they correspond to sets of outcomes. These sets of outcomes that we can "talk about" are called events $E$, and the set of all possible events is called a field $\mathcal{F}\subseteq 2^X$.

For instance: if our sample space is $\{1,2,3,4,5,6\}$ and our measurement apparatus is a guy who looks at the reading and tells us if it's even or odd, then the field is $\{\varnothing, \{1,3,5\},\{2,4,6\},X\}$. We simply cannot talk about sets like $\{1,3\}$ or $\{1\}$. Our information just doesn't tell us anything about sets like that -- when we're told "odd", we're never hinted if the outcome was 1 or 3 or 5, so we can't even have prior probabilities -- we can't even give probabilities to whether a measurement was a 1 or a 3.

Well, what kind of properties characterise a field? There's actually a bit of ambiguity in this -- it's clear that a field should be closed under negation and finite unions (and finite intersections follow via de Morgan) -- if you can talk about whether $P_1$ and $P_2$ are true, you can check each of them to decide if $P_1\lor P_2$ is true (and since a proposition $P$ corresponds to a set $S$ in the sense that $P$ says "one of the outcomes in $S$ is true", $\lor$ translates to $\cup$). But if you have an infinite number of $P_i$'s, can you really check each one of them so that you can say without a doubt that a field is closed under arbitrary union?

Well, this is (at this point) really a matter of convention, but we tend to choose the convention where the field is closed under negation and countable unions. Such a field is called a sigma-field. We will actually see where this convention comes from (and why it is actually important) when we define probability -- in fact, it is required for the idea that one may have a uniform probability distribution on a compact set in $\mathbb{R}^n$.

---

A beautiful way to understand fields and sigma fields is in terms of venn diagrams -- in fact, as you will see, fields are precisely a formalisation of Venn diagrams. I was pretty amazed when I discovered this (rather simple) connection for myself, and you should be too.

Suppose your experiment is to toss three coins, and make "partial measurements" on the results through three "measurement devices":

• A: Lights up iff the number of heads was at least 2.
• B: Lights up iff the first two coins landed heads.
• C: Lights up iff the third coin landed heads.

What this means is that $A$ gives you the set $\{HHT, HTH, THH, HHH\}$, $B$ gives you the set $\{HHH, HHT\}$, $C$ gives you the set $\{HHH, HTH, THH, TTH\}$. Based on precisely which devices light up, you can decide the truth values of $\lnot$'s and $\lor$'s of these statements, i.e. complements and unions of these sets -- this is the point of fields, of course.

Or we could visualise things.

Well, the Venn diagram produces a partition of $X$ corresponding to the equivalence relation of "indistinguishability", i.e. "every event containing one outcome contains the other"? The field consists precisely of any set one can "mark" on the Venn diagram -- i.e. unions of the elements of the partition.

A consequence of this becomes immediately obvious:

Given a field $\mathcal{F}$ corresponding to the partition $\sim$, the following bijection holds: $\mathcal{F}\leftrightarrow 2^{X/\sim}$.

Consequences of this include: the cardinalities of finite sigma fields are precisely the powers of two; there is no countably infinite sigma field.

---

Often, one may want to some raw data from an experiment to obtain some processed data. For example, let $X=\{HH,HT,TH,TT\}$ and the initial measurement is of the number of heads:

$$\begin{align}
\mathcal{F}=&\{\varnothing, \{TT\}, \{HT, TH\}, \{HH\},\\
& \{TT, HT, TH\}, \{TT, HH\}, \{HT, TH, HH\}, X \}
\end{align}$$
What kind of properties of the outcome can we talk about with certainty given the number of heads? For example, we can talk about the question "was there at least one heads?"

$$\mathcal{G}=\{\varnothing, \{TT\}, \{HT, TH, HH\}, X\}$$
There are two ways to understand this "processing" or "re-measuring". One is as a function $f:\frac{X}{\sim_\mathcal{F}}\to \frac{X}{\sim_\mathcal{G}}$. Recall that:

$$\begin{align}
\frac{X}{\sim_\mathcal{F}}&=\{\{TT\},\{HT,TH\},\{HH\}\}\\
\frac{X}{\sim_\mathcal{G}}&=\{\{TT\},\{HT,TH,HH\}\}
\end{align}$$
Any such $f$ is a permissible "measurable function", as long as $\sim_\mathcal{G}$ is at least as coarse a partition as $\sim_\mathcal{F}$. In other words, a function from $X/\sim_1$ to $(X/\sim_1)/\sim_2$ is always measurable.

But there's another, more "natural", less weird and mathematical way to think about a re-measurement -- as a function $f:X\to Y$, where in this case $Y=\{0,1\}$ where an outcome maps to 1 if it has at least one heads, and 0 if it does not.

But there's a catch: knowing that an event $E_Y$ in $Y$ occurred is equivalent to knowing that an outcome in $X$ mapping to $E_Y$ occurred -- i.e. that the event $\{x\in X\mid f(x)\in Y\}$ occurred. Such an event must be in the field on $X$, i.e.

$$\forall y\in\mathcal{F}_Y,f^{-1}(y)\in\mathcal{F}_X$$
This is the condition for a measurable function, also known as a random variable.

---

One may observe certain analogies between the measurable spaces outlined above, and topology -- in the case of countable sample spaces, there actually is a correspondence. The similarity between a Venn diagram and casual drawings of a topological space is not completely superficial.

The key idea behind fields is mathematically a notion of "distinguishability" -- if all we can measure is the number of heads, $HHTTH$ and $TTHHH$ are identical to us. For all practical purposes, we can view the sample space as the partition by this equivalence relation. They are basically the "same point".

It's this notion that a measurable function seeks to encapsulate -- it is, in a sense, a generalisation of a function from set theory. A function cannot distinguish indistinguishable points -- in set theory, "indistinguishability" is just equality, the discrete partition; a measurable function cannot distinguish indistinguishable points -- but in measurable spaces, "indistinguishability" is given by some equivalence relation.

Let's see this more precisely.

Given sets with equivalence relations $(X,\sim)$, $(Y,\sim)$, we want to ensure that some function $f:X\to Y$ "lifts" to a function $f:\frac{X}{\sim}\to\frac{Y}{\sim}$ such that $f([x])=[f(y)]$.

(Exercise: Show that this (i.e. this "definition" being well-defined) is equivalent to the condition $\forall E\in\mathcal{F}_Y, f^{-1}(E)\in \mathcal{F}_X$. It may help to draw out some examples.)

Well, this expression of the condition -- as $f([x])=[f(y)]$ -- even if technically misleading (the two $f$'s aren't really the same thing) give us the interpretation that a measurable function is one that commutes with the partition or preserves the partition.

While homomorphisms in other settings than measurable spaces do not precisely follow the "cannot distinguish related points" notion, they do follow a generalisation where equivalence relations are replaced with other relations, operations, etc. -- in topology, a continuous function preserves limits; in group theory, a group homomorphism preserves the group operation; in linear algebra, a linear transformation preserves linear combinations; in order theory, an increasing function preserves order, etc. In any case, a homomorphism is a function that does not "break" relationships by creating a "finer" relationship on the target space.

Projection operators, generalised Born's rule, position basis, wavefunction

At the end of the last article, I asked you to investigate Born's rule for continuous variables like position and momentum.

Well, the problem is that if $x$ is continuously distributed (i.e. we have an operator $X$ whose eigenvalues form a continuous spectrum $\Sigma_X$), typically $P(x=\lambda)=0$ -- and this gives us very little information about the actual probability distribution. What we're really interested in is $P(x\in B)$ for $B$ some subset of $\Sigma_X$.

Technically, we need $B$ to be a "Borel subset", or "measurable subset". We will be omitting several such technicalities in the article, such as the need for the spectral theorem to define a "projection-valued measure" or "spectral measure" on an operator with a continuous spectrum -- this is something that will be covered in the MAO1103: Linear Algebra course.

First, let's think about $P(x\in B)$ in the countable case. One can write $B=\{\lambda_1,\ldots\lambda_n\}$, and then simply say that

$$P(x\in B)=\sum |\langle\psi|\phi_k\rangle|^2$$
But the term on the right is a Pythagorean sum -- specifically, it is the length-squared of the vector formed by summing all the projections of $|\psi\rangle$ onto the eigenstates $|\phi_1\rangle\ldots|\phi_k\rangle$. But this is the same as the length of the projection of $|\psi\rangle$ onto the span of these eigenstates.

(Note on notations: From here onwards, we will use the notation $|\lambda\rangle$ to refer to the eigenvector corresponding to the eigenvalue $\lambda$ (if the eigenspace has dimension more than 1, we'll figure something out). We will use the notation $\{|B\rangle\}$ to refer to the span of the eigenvectors corresponding to the eigenvalues in $B$.)

So we could just define a Hermitian projection operator $L_X(B)$ for any subset $B$ of the spectrum of $X$ -- it is an easy exercise to write down an explicit form for $L_X(B)$ in terms of the eigenvectors of $X$.

Then the probability $P(x\in B)$ is simply $|L_X(B)|\psi\rangle|^2$. Recalling that a Hermitian projection operator satisfies $L^*=L=L^2$, we can write the generalised Born's rule as:

$$\begin{align}P(x\in B) &= |L_X(B)|\psi\rangle|^2\\ &= \langle\psi|L_X(B)|\psi\rangle\end{align}$$
Well, this is interesting! In the last article, you proved that the expected value of an observable $X$ given a state $|\psi\rangle$ is given by $\langle\psi|X|\psi\rangle$. But here we have a probability given by the same expression. So we want to interpret our projection operators as some sort of "observable" -- we can omit the "Hermitian", since all observables are Hermitian.

There's another place you might've seen something like this, and that is with indicator variables in probability and statistics -- the expected value of an indicator variable for an event is the probability that the event occurs.

Try to interpret these projection operators as observables that are analogous to "indicators" in some sense. If you think a little about it, you might see exactly what these observables represent: the eigenvalues of $L_X(B)$ are all 1 and 0 -- if the value "1" is realised, the state has been projected into the $\{|B\rangle\}$ -- and if the value "0" is realised, it hasn't.

So projection operators are a special type of observable, measuring the answer to "Yes/No questions" -- if the answer to "is the system in one of the states $\{|B\rangle\}$?" is yes, the observable $L_X(B)$ takes the value 1 -- if the answer is no, then it takes the value 0. So it is precisely an "indicator variable" for $\{|B\rangle\}$.

We have seen such projection operators, of course, in the context of polarisation -- where the operator represented whether or not the photon has passed through. Indeed, one may formulate quantum mechanics entirely in terms of projection operators, as any question can be formulated with some number of Yes/No questions (the key reason why this can be done, as we will see -- is that these "yes/no questions" all commute, i.e. the corresponding projection operators share an eigenbasis). Let's not.

---

Well, this can be generalised in the straightforward way to an operator with a continuous spectrum, resulting in the same expression. We can also calculate probability densities using this result. Let $X$ be an operator with continuous spectrum $\Sigma_X$ -- then we can write the state $|\psi\rangle$ in the eigenbasis of $X$:

$$ |\psi\rangle = \int_{\Sigma_X} |x\rangle\, \Psi(x)\, dx $$
Where $\Psi(x)\, dx=\langle\psi|x\rangle$ are the coefficients of the state in the eigenbasis, i.e. the probability amplitudes -- we call $\Psi(x)$ the wavefunction, and it represents probability amplitude densities. Then for some set $M\subseteq \Sigma_X$ of eigenvalues $L_X(B)|\psi\rangle$ is the projection:

$$ L(M)|\psi\rangle = \int_B |x\rangle\, \Psi(x)\, dx $$
And one may calculate the dot product, noting that complex dot products require taking the complex conjugate:

$$ \langle \psi | L_X(B) | \psi \rangle = \int_B \Psi^*(x)\, \Psi(x)\, dx $$
Which gives us an expression for the probability density function on $\Sigma_X$ as:

$$\begin{align}\rho(x) &=\Psi^*(x)\,\Psi(x) \\
&=|\Psi(x)|^2\end{align}$$
And this applies to any operator with a continuous spectrum, like position and momentum.

---

Some texts define the eigenvectors $|x\rangle$ of a continuous-spectrum observable differently from us -- it is often conventional to let $|x\rangle$ be infinitely large so that $\langle x_1|x_2\rangle = \delta(x_1-x_2)$. This is so that the amplitudes $\langle\psi|x\rangle$ are not infinitesimal, but instead $\langle\psi|x\rangle=\Psi(x)$ (without multiplication by $dx$). For consistency with discrete spectra, we do not use this convention.

From polarisation to quantum mechanics: states, observables, Born's law

Like most texts on the theory, I will motivate the mathematics of quantum mechanics from the example of polarisation -- mostly because it's a very accessible example of stuff being wavelike. From this example, we will be able to motivate: the state vector (generalising the polarisation), state vector collapse (the event of polarisation), observables and their eigenvalues (stuff like energy, number of photons, etc.), eigenstates and their orthogonality (polarisation basis), noncommuting operators and uncertainty (the noncommuting of lenses).

The key feature of quantum mechanics -- the fundamentally probabilistic nature -- comes from the following two facts, confirmed by experiments (the famous experiments here are the double-slit experiment and photoelectric effect respectively):

• Everything is a wave -- objects behave as waves, following the superposition principle and the waves represent densities of observations at large scales.
• Everything is a particle -- which manifests itself in the form of some stuff, like energy and momentum, coming in little quanta.

This is the principle of wave-particle duality. You may realise how this implies a probabilistic description, but the following example should make it quite clear: consider a wave of light, with energy $hf$ (so it's a single photon) polarised at angle $\theta$ to the horizontal -- and it passes through a horizontal polarising filter. Well, then the wave that passes through would be a horizontally polarised wave with energy $hf\cos^2\theta$, right?

But this is impossible, since energy levels in quantum mechanics are quantised -- you can't have $\cos^2\theta$ of a photon, you can only have integer multiples of a photon. But the fact that energy drops as $E\cos^2\theta$ is something that you can verify at your home, using sunglasses -- what the heck?

The key point is that the empirical verification of the $\cos^2\theta$ business that you can do at home is on a macroscopic level, when you have a large number of photons $E=Nhf$. So something occurs with the photons on a microscopic level such that when you try it with a large number of photons, $\cos^2\theta$ of the photons pass through.

Well, this is essentially the "definition" of probability! A single photon passes through the filter with a probability of $\cos^2\theta$ so that for a large number of photons, $\cos^2\theta$ of the photons pass through. This is a non-trivial result -- wave-particle duality makes no mentions of probability as such, it just tells us that stuff is both a particle and a wave, but this simple condition in itself implies a probabilistic, non-deterministic reality.

---

Similar thought experiments can illustrate the probabilistic nature of other things (the "things" in question here will soon be called "eigenstates"): position is easy -- consider a standing wave photon in a box (this can easily be constructed). This is uniformly distributed throughout the box -- so how much of the energy is in some chunk of the box?

Momentum is trickier, but shouldn't be too hard if you're familiar with Fourier transforms -- what's the analog of a "box" in momentum-space? Well, consider a concentrated pulse of light -- this can be written, via a Fourier transform, as the sum of several light waves of different momenta (i.e. frequencies), each wave with some lower energy. Taking "some chunk" of this "box" amounts to filtering some specific frequencies of the light. This can be done easily, e.g. with a colour filter -- so how much of the energy is contained in the waves with these specific momenta?

In both cases, the key point is that you can't have a fraction of the energy of the photon at these positions/momenta, so you must have a probability of measuring the photon to be in a specific range of positions or a specific range of positions -- to be in a specific region or in a specific region of momentum-space.

---

The fundamental point here can be made for any quantity $X$: if you can filter out the "part" of a collection of particles that has $X$ in a certain subset of its range, then on a microscopic level, is probabilistic. The act of "filtering out the parts with a certain $X$", applied to a single particle, is just the act of checking if a particle is in a certain $X$-interval, and is called measurement. Any quantity that you can measure is called an observable. 

Something like polarisation is really a form of measurement -- you're finding out whether or not the photon is in a certain polarisation $|\phi_{\parallel}\rangle$. You may have another observable, corresponding to a different polarisation -- even one that is orthogonal to the first polarisation -- $|\phi_{\perp}\rangle$ and still get that the photon is in $|\phi_\perp\rangle$. There is nothing wrong with this, as we just know beforehand that the photon is in $|\phi_\parallel\rangle$ or $|\phi_\perp\rangle$. If you perform the polarisation with $|\phi_\perp\rangle$ after the polarisation with $|\phi_\parallel\rangle$, you will find that the photon doesn't pass through, as you know for sure that the photon is not in both $|\phi_\parallel\rangle$ and $|\phi_{\perp}\rangle$.

Now, you may have certain psychological issues with this, as have many in history -- however, you might want to note that the aim of quantum mechanics is not to fix your psychological problems but to explain nature. You need to accept logical positivism and learn to shut up and calculate to be comfortable with quantum mechanics.

So whatever calculus we invent to describe these probabilistic phenomena, it is going to apply to all observables.

In our first example, the polarisation of the photon can be represented by a unit vector which we will denote as $|\psi\rangle$. The polarising filter has two special axes, represented by unit vectors $|\phi_{\parallel}\rangle$ and $|\phi_\perp\rangle$ -- these are special in the sense that an incoming photon polarised as $|\phi_{\parallel}\rangle$ or $|\phi_\perp\rangle$ will simply be scaled, by factors of 1 and 0 respectively -- so these form an eigenbasis for a certain operator.

Well, we said that the photon passes through (with polarisation $|\phi_{\parallel}\rangle$) with probability $\cos^2\theta$ -- this arises simply from considering the amplitude of $|\psi\rangle$ in the direction of $|\phi_\parallel\rangle$. So we can write the probability that the photon ends up in a state $|\phi\rangle$ as $|\langle\psi|\phi\rangle|^2$ where $\langle\psi|\phi\rangle$ is called the corresponding "probability amplitude".

This expression, $P(x=\lambda)=|\langle\psi|\phi_\lambda\rangle|^2$ is called Born's rule.

Let's get back to the eigenbasis -- what exactly is this an eigenbasis of? We said that the corresponding eigenvalues are 1 and 0, so this gives us a complete description of the operator. Note that this operator depends only on the observable (namely "number of photons in the $|\phi_{\parallel}\rangle$ direction), not on the state or any other feature of the observation. So we decide to call this operator/matrix the "observable", and its eigenvalues are the values of the observable that can be measured.

To find properties of these observables, the natural way is to note that the only feature we've really required of them is Born's rule, i.e. the probabilistic interpretation -- so we can apply the axioms of probability and see what they apply in the context of these observables.

• $P(E)\ge 0$ -- imply that the observables are over either the reals or complexes, so that $|\langle\psi|\phi\rangle|^2\in \mathbb{R}$ in the first place. The nonnegativity then follows.
• $P(\Omega)=1$ and $P\left(\bigcup_i E_i\right) = \sum_i P(E_i)$ for disjoint $E_i$ -- this, along with the second axiom, implies that $\sum |\langle\phi|\psi\rangle|^2 = 1=|\langle\psi|\psi\rangle|^2$ where the sum is taken over all eigenstates $|\phi\rangle$ of the operator. As this must be true for all states $|\psi\rangle$, the thing on the left must be a Pythagorean sum, so the $|\phi\rangle$s must form an orthogonal basis. This implies that all observables are normal operators.

The latter fact is very important, and can also be seen in the following way -- if you a system is in one eigenstate, it cannot possibly collapse onto another eigenstate (the probabilistic interpretation is: if you know for sure the value of the symbol is a thing, it's that thing) -- so we must have $|\langle \phi_1|\phi_2\rangle|^2=0$ for all eigenstates $|\phi_1\rangle$ and $|\phi_2\rangle$.

Another restriction we add is that the observables be not only normal, but Hermitian operators in particular, so they have real eigenvalues. This may seem an odd choice, but it makes sense, as any normal operator may be uniquely written as $X_H+iX_{AH}$ where $X_H$ and $X_{AH}$ are Hermitian, and $X_H$ and $X_{AH}$ commute, so any complex observation can be done unambiguously as two real observations. So we stick to real eigenvalues.

This also makes it essential that we allow complex operators rather than just real ones (the two choices were given to us from the first probability axiom), so that this decomposition is possible. Later, we will see concrete examples of this with commutators $[X,Y]$, which must be multiplied by $i$ to turn Hermitian. We will also see more fundamental reasons to choose complex numbers in QM.

---

Exercise: Show that the expected value of an observable $X$ given a state $\psi$ can be given as $\langle \psi|X|\psi \rangle$ (i.e. $\psi^*X\psi$ in conventional notation).

Exercise: Explain Born's rule with other observables, like position and momentum. Explain why it holds in general.

Probability of immortality for a transhuman being

Some species in nature, including the Turritopsis dohrnii "the immortal jellyfish" are biologically immortal. This means that they do not die due to biological reasons -- however, they obviously may die due to other physical reasons, like getting smashed with a hammer. If I asked you to calculate what the probability of a biologically immortal species being truly immortal -- i.e. of it never dying (ever) -- would be, what'd you answer?

Well, obviously the probability is zero. Provided there is any chance at all of the jellyfish getting squashed by a hammer this year, with a sufficient amount of time you can be as certain as you want -- the probability can be as close to 1 as you want -- that the jellyfish will get squashed by a hammer.

But what if the probability of getting smashed by a hammer in that year was decreasing with time? Perhaps this is not the case with jellyfish, but it certainly would be true for, e.g. a transhuman society where technological innovation continually decreases the probability of dying (to be precise, the probability density of being dead in the next interval of time $\Delta t$ given that you haven't already died).

Let $p(t)\Delta t$ be the probability of our transhuman dying between $t$ and $t+\Delta t$. Then the probability of the transhuman never dying any time from 0 to infinity is:

\[\begin{gathered}
  P = \left( {1 - p(0)\Delta t} \right)\left( {1 - p(\Delta t)\Delta t} \right)\left( {1 - p(2\Delta t)\Delta t} \right)... \\
   = \coprod\limits_{t = 0}^\infty  {\left( {1 - p(t)dt} \right)}  \\
\end{gathered} \]
Of course, we need to take the limit as $\Delta t \to 0$.

The reason this problem is so interesting is because it introduces the idea of multiplicative calculus. If the product had been a sum, the solution would've been utterly, ridiculously straightforward. But since it's not, it's only really ridiculously staightforward. The natural way (no pun intended) to convert a product (we use the symbol \(\coprod {} \) to refer to the multiplicative integral) into a sum (or rather an integral) is to take the logarithm:

\[\begin{gathered}
\ln P = \ln \left( {1 - p(0)\Delta t} \right) + \ln \left( {1 - p(\Delta t)\Delta t} \right) + \ln \left( {1 - p(2\Delta t)\Delta t} \right)... \\
= \int_0^\infty {\ln \left( {1 - p(t)dt} \right)} \\
\end{gathered} \]
This may look awkward to you -- and indeed, the standard form of the multiplicative integral typically has the $dt$ differential as the exponent of the integrand so as to obtain after taking the logarithm the additive integral in its standard form.

But you might remember that

\[\ln (1 - x) =  - x - \frac{{{x^2}}}{2} - \frac{{{x^3}}}{3} - ...\]
Or to first-order in $x$ (since the "x" here, $p(t)dt$ approaches 0), $\ln (1 - x) \approx  - x$. Thus:

\[\ln P =  - \int_0^\infty  {p(t)dt} \]
Or:

\[P = {e^{ - \int_0^\infty  {p(t)dt} }}\]
Which is pretty neat! Interestingly, this means that if the integral of $p(t)$ diverges (e.g. if $p(t)\sim1/t$), you will almost certainly die. So this gives us a manual of how fast technological progress on this issue needs to be in the transhuman age to guarantee immortality. Internalise it in your demand, fellow robot!

---

Here, we've calculated the probability of immortality. The probability of eventual mortality is of course 1 minus this, but could also be calculated from the get go -- try this out. You'll get $P'=1-\int_0^\infty p(t) e^{-\int_0^t p(\tau) d\tau}dt$, which you can then simplify with a variable substitution. Perhaps this gives you some insight into variable substitutions in integrals of this sort.