Dutch book argument for probability theory

aka "how to profit off people's irrationalities".

The philosophical question of what it means to be rational is answered by the popular slogan "Rationality is systematized winning" -- so, in particular, a rational belief is one that is expected to deliver you a profit.

This is closely linked to the "practical" definition of probability -- assigning probability $p$ to observation $X$ means valuing at $p$ a unit wager on $X$ (an asset with payoff 1 if $X$ happens, an 0 otherwise) -- you are willing to buy the wager for any price less than $p$, and sell it for any price higher. 

(We use money to make things simple, but more fundamentally we have a cardinal utility function whose expectation we seek to maximize -- so this is a valid abstract definition and does not depend on a specific currency or monetary system.)

However, if you accept that probability is definitionally subjective (even if updated by evidence via Bayes's theorem, your priors are still subjective), then assigning a probability $p$ to a particular event cannot itself be rational or irrational. You can't quite calculate whether someone who holds that belief will make a profit or loss, because doing so would require assigning an underlying "objective" probability to $X$. 

You can only say that a person is irrational if you can prove that their beliefs always lead to them losing, regardless of the results of the observation. Such beliefs are said to be incoherent, a notion that is identical to the financial concept of arbitrage. Irrationality in this sense is about an inconsistency in beliefs about different, related observations.

Consider observations $X_1,\dots X_n$ (in the language of probability theory, these are elements of our sigma algebra). Suppose you assigned to them the probabilities $p_1,\dots p_n$. 

Your beliefs are irrational iff I can come up with a list of unit wagers (called a Dutch book) to trade with you that always deliver me a profit regardless of the outcome of each observation.

For example, if $X_1\subseteq X_2$ and you assign probabilities $p_1=0.5,p_2=0.1$, then I can buy from you a unit wager on $X_2$ and sell you a unit wager on $X_1$, and always make a profit.

The axioms of probability theory can then be shown to be equivalent to forbidding a Dutch book. If the values of all bets are positive between 0 and 1, the value of the bet on the entire sample space is 1 and bets are additive, then it becomes impossible to construct a Dutch book. Conversely, a violation of any of these axioms allows a bet on something equivalent to the sample space to be traded for less than or greater than 1, allowing for a Dutch book.

Similar Dutch book arguments can be extended to conditionalization and Bayes's theorme.

You can't hack Bayes's theorem

Aka the difference between rationalization and rationality.

Eliezer Yudkowsky calls this the dilemma of the clever arguer: A propagandist (clever arguer) is hired to sell you a box that may or may not contain a diamond -- he tells you that the box has a blue stamp on it (which you know occurs more on boxes containing diamonds). If you could handle the box yourself, you could rationally evaluate all the characteristics of the box and compile their influences on your probability estimate Are you then forced, for each argument the clever arguer provides, to helplessly update your probabilities as the clever arguer wishes, even though you know the propagandist has omitted the evidence he doesn't want you to know of?

There are various equivalent formulations of the problem including:

• p-hacking
• Filtered evidence: an experimenter flips a coin 10 times and tells you that the 4th, 5th and 7th tosses came up heads, without telling you anything about the other tosses. 

The key matter to realize is that the information you get is not simply the 4th, 5th and 7th tosses came up heads, but instead: the propagandist tells me that the 4th, 5th and 7th tosses came up heads. The statistical process we are studying is no longer the "natural" (IID Bernoulli) process we're used to, but a different process, which depends on the inner mechanism used by the propagandist. For example:

• If the propagandist always tells you only the 4th, 5th and 7th tosses, you update your beliefs from this evidence as normal.
• If the propagandist only tells you the coin tosses that came up heads, then you now know that the other seven tosses come up tails, and you update your beliefs accordingly.
• If the propagandist chooses any three heads that came up to tell you about, then the probability of 4, 5 and 7 specifically being chosen is only slightly greater with a biased coin (you can calculate this, but the key point is that the process we're observing is not about which coins come up heads, but which coins are chosen by the propagandist). 

We could have a probability distribution on the various possible mechanisms underlying the propagandist, and then the information we get from our observations is actually split between information on the propagandist's mechanism and information on the coin's mechanism.

Here's a model that makes it easier to perform inference on: we have $n$ features $X_i$ distributed as $N(\mu, 1)$ with prior $N(0,1)$ on $\mu$, and the propagandist chooses to tell us the value $x$ of $X_j$ that has the greatest value among all $X_i$s. The posterior density on $\mu$ can then be calculated:

\[\frac{{{\Phi _\mu }{{(x)}^{n - 1}}{\phi _\mu }(x){\phi _0}(\mu )}}{{\int_\mu  {{\Phi _\mu }{{(x)}^{n - 1}}{\phi _\mu }(x){\phi _0}(\mu )\,d\mu } }}\]

Where $\phi_\mu$, $\Phi_\mu$ are the normal PDF and CDF respectively. Here's an interactive visualization of this density:

Link to interactive version.(Note that in this visualization, $\mu$ is $x$ and $x$ is $z$).

For example for $n=10$, $x=1$, the distribution actually shifts leftwards, because surely if $\mu$ were 0, the propagandist could have found a feature with a better value than 1.

The moral of the story is that you can't hack Bayes's theorem; you can't fool a rational agent. If you have a parameter whose value you know, you can't systematically produce misinformation about the parameter. This is a result of the conservation of expected evidence: the expectation of the posterior probability of each value is its prior probability.

So when we talk about scientific standards -- about scientists revealing all relevant information and not p-hacking, etc. -- these are not requirements for Bayesian inference, but they're simply a way to ensure that scientific research is maximally informative. If you don't know the underlying process that scientists use to report their data, or if you know that they use a "biased" process, then your estimator of the relevant parameter will be less informative than it could have otherwise been.

The method they teach you in primary school to combat filtered evidence -- listing "arguments and counter-arguments", "pros and cons", etc. -- is far inferior to the standard of scientific ethics. Listing arguments and counter-arguments replaces a one-sided rationalization with a two-sided rationalization, but it doesn't truly approach rationality -- you just have two propagandists instead of one. 

Bayes theorem, estimators and the theory of Amazon ratings

If you've ever bought something on Amazon, or looked at player ratings on an online game, you've often had to weigh the average rating against the number of ratings. A higher average rating is surely a good thing, but would you really prefer one 5.0 rating to a hundred 4.5 ratings?

The way to think about a question like this is to ask: What are we trying to maximise, here? What is it that we're trying to achieve?

We're trying to maximise what our own rating would be -- or rather, the expected value of what our own rating would be.

So we should think of the rating as a random variable with some distribution, whose true mean (and not the sample mean) is what we seek to estimate. It was not immediately obvious that the problem would be a statistical one, but it is now.

So, why don't we just use the sample mean to estimate the true mean, and trust the "unreliable" 5.0 rating over the "reliable" 5.0 rating? Three reasons:

• Items usually don't have a true 5.0 rating, and just one rating is unlikely to change our opinion of this.
• We're not really trying to maximise the expected value, e.g. we may be concerned about risk (i.e. a greater variance in the distribution of the true value of the rating is itself a negative, because a true 1.0 rating may mean the application is dangerous or contains viruses)
• Having fewer ratings may itself be an indicator of lower quality, because it's less popular.

In any case: the keyword is "belief distribution". The first point points out that we have a prior belief distribution and each rating updates this belief distribution. The second point points out that we need an idea of what the entire posterior distribution looks like, rather than just a point estimate.

Well, obviously, the solution is Bayesian statistics (link to the main Bayes's theorem article). If you have a prior distribution on the true mean, and a likelihood function for the observations, you can compute the posterior distribution via Bayes's theorem.

Exercise: Compute, in the case of just two possible ratings (upvote and downvote/like and dislike) the "correct" rating (say, mode of posterior) given a uniform prior. Derive the so-called Laplace's rule of succession. 

Machine learning as function approximation; statistical and cognitive motivations

The most general formulation of machine learning is that it is the attempt to approximate (or "learn") some mathematical object. For example:

• Regression attempts to approximate a function.
• Classification attempts to approximate an equivalence relation.
• Dimensionality reduction attempts to approximate a parameterization.
• Generative neural networks attempt to approximate a distribution (i.e. a sampler).
• Representation attempts to approximate a compression algorithm.
• Game-theoretic learning (e.g. reinforcement learning, no-regret learning) attempts to approximate a distribution of strategies. 
• Neural networks in NLP attempt to approximate a vector space that effectively represents desired relationships between words.

The Data Science course addresses several "toy algorithms" that discover low-order or simplified approximations. There are two -- different -- explanations of what more "AI-esque" machine learning is, which also lead to two different motivations for machine learning, and two different ways to understand each neural network architecture and each of some ideas in machine learning.

1. The statistical motivation -- machine learning is a non-linear generalisation of "linear" statistical techniques (data mining) like linear regression, PCA and linear decision boundaries. One can always do these linear techniques with some transformation of the data that makes relationships linear (while making sure the transformation is not absurd), but you need a way to "train" what the right such function is. In this sense, machine learning acts as a function approximator.
2. The cognitive science motivation -- a computer should be able to do whatever a brain can, but how exactly does a brain do the stuff it does? To take a simple example, the brain can recognise digits -- well, whatever the brain does, it takes an image as input and outputs a digit, i.e. it's a function. So once again, we need a function approximator.

Great. So machine learning is about making function (or object) approximators. The basic idea is that we're looking for a function that minimises the overall error for a population of data -- it's basically a calculus of variations problem, isn't it? Well, except it isn't, because we don't have access to the entire population, so we need to avoid overfitting (i.e. we need to consider a Bayesian prior). This is also what we meant by "making sure the transformation is not absurd" as we mentioned.

As a general rule, I'd say that if a feature of human brains are present at birth, we should expect to have to hard-code it, while if a feature is learned by humans, we should definitely get our AI to learn it, too (this is the "converse" of a general rule I have when trying to organize knowledge about biology in my head, which is we pretend that evolution is not an algorithm but a hard-coding, simply because actually trying to simulate an evolutionary algorithm in a highly complicated environment is hard). So for example, the transformations that a spatial transformer network finds acceptable are hard-coded, because we didn't actually need to stand on our head or squish our eyes with a truncheon to learn how to read squished-up text. 

Anyway, we want a universal function approximator -- a system that can generate a function arbitrarily close to any given function given sufficiently many parameters. A basic example of such a system is polynomial regression, but for most applications this has the wrong Bayesian prior (it gives zero prior probabilities to high-order polynomials, but most machine learning applications require functions with heavy non-local effects).

Another function approximator is a neural network. That a neural network (even of single layer) is a universal approximator is called the universal approximation theorem, i.e. functions can be written as linear combinations of some scaled and translated sigmoid functions.

(Exercise: explain why the universal approximation theorem is true for the sigmoid function. What other kinds of functions is it true for? It's actually not that hard at all. If you do get stuck, check out the visuals in Michael Nielson's e-book. A rigorous proof can be found here.)

In fact, the universal approximation theorem is not actually particularly important at all to the success of neural networks -- like we said, plenty of systems are universal approximators, but they don't have the right Bayesian prior (and this matters when you have limited data). The single-layer neural network actually has a bad implied prior for many tasks, which is why we usually study "deep" neural networks, which have a surprisingly good prior.

I have seen many explanations as to why this is so: deep learning means doing things in steps, deep learning corresponds to decomposing "hierarchy" or "structure" in the world and our world is "inherently hierarchial", etc. But honestly, these all seem like terrible rationalizations -- I don't even see how these claims are testable. You would need to create a system that isn't "inherently hierarchial" and demonstrate that deep learning doesn't do very well on it (which, as far as I can see, makes no sense). The right way to "explain why deep learning works so well" is to compute the implied prior and find out how it scales with depth.

Someone do this.

Uniform priors and Maximum Likelihood Estimation

In the last article, we saw that Bayes's theorem allows us to infer theories from data -- or rather infer from data a distribution on the theory-space. This is the general way to make inferences and use these inferences to make predictions. We don't really need to bother about questions of whether we should choose the mean or median or mode as the theory we "choose", because we don't have to choose -- we make predictions with the entire distribution.

For example in the most general setting, we're trying to calculate our optimal action out of a set of choices based on a certain unknown utility function that depends on the right theory of physics (to calculate the consequences of the action). We don't need to "choose" the right theory of physics from the probability distribution we have on it -- we just make predictions for each possible theory in the support and integrate over the distribution to calculate the distribution on possible consequences. It's only at this point where we need to care about whether we want the action with the maximum mean, median, mode or whatever of the utility function.

Can you show that Bayesian inference can always converge? I.e. that there is always some data collection mechanism that allows your distribution to converge to a Dirac delta at some point? Well, there is, and it's called "sampling". What if we can't sample directly but collect some other indirect data? Comment on the philosophical implications of such a theory-space (hint: think about a probabilistic generalisation of falsifiability (a criterion for a theory to be scientific)).

But some special cases of Bayesian inference -- with special choices of prior -- exist and are often discussed/used in the literature for various reasons, and in fact, they often come with mechanisms to choose specific estimates based on the posterior distribution, as well as specific projections of our data to use. We will discuss some of these examples below.

---

Perhaps the most straightforward example is maximum likelihood estimation. Here, the prior distribution is the uniform distribution, i.e.

$$\mathrm{Pr}(\theta\mid x)=\frac{\mathrm{Pr}( x\mid\theta)}{\mathrm{Pr}(x)}$$
And we always take the mode of this (the value of theta that maximises $\mathrm{Pr}(\theta\mid x)$) to be the estimate of the parameter. This is just the value of theta that maximises $\mathrm{Pr}( x\mid\theta)$ (for observed data values $x$), which is called the likelihood function.

I.e. we're saying we have no preconceptions about the theory, and the right theory is just the one that gives the observed data the highest probability (or probability density).

Note that even in maximum likelihood estimation, you kinda do have non-uniform prior, in the sense that the prior is only uniform on its support, and you give zero values to things outside the domain of the parameter, or to theories other than the family being tested, etc. So if you were just taught maximum likelihood estimation, it would be easy for you to come up with the idea of Bayesian inference.

---

Suppose you have a prior distribution given by $\rho(\theta)$ and some specific value of $\theta_0$, which, regardless of the actual value of the prior distribution, you are "decisively" biased towards for some reason (e.g. you're looking at a suspect's guilt and you'll act on the assumption of innocence as long as the probability of his innocence is at least 1%). You may also be happy to consider a small neighbourhood around $\theta_0$ as "effectively" the same as $\theta_0$ for any practical purposes you may have (e.g. no coin is perfectly fair, but most coins are pretty fair).

Two-envelopes problem: beyond the Bayes explanation

The two-envelopes problem is phrased as follows:

There are two envelopes, one with twice the amount of money as the latter -- you pick one of them at random and open it to find $x$ dollars. You don't know if this was the larger or smaller sum.

Should you switch?

Though symmetry would tell you that it doesn't matter, it seems that by switching you get $x$ if the other envelope is bigger, but only lose $x/2$ if the other envelope is smaller. So you should switch. Right?

The standard explanation to this problem is that we need to pick a prior to make any actual inference, and the probability of the other envelope being bigger is usually less than the probability of it being smaller, in the sense that probability distributions approach zero towards infinity.

But this explanation doesn't really make the problem go away. Consider this prior:

$$
f(x) = \left\{ {\begin{array}{*{20}{c}}{\frac{1}{4}{{(3/4)}^n}}&{{\rm{if }}\ x = {2^n},\,\,n \in {\mathbb{Z}_{ \ge 0}}}\\0&{{\rm{else}}}\end{array}} \right.
$$
This is a perfectly valid prior distribution (the improper prior assumed in the original "paradox" would have worked as well, but I don't want to distract you with questions about distribution propriety) for which exactly the same paradox applies -- one may show (and I encourage you to work this out) that whatever value $x$ I find in the envelope, the expected value of the other envelope is $8/7\ x$.

Let's try to remember why this is weird -- what exactly is so paradoxical about it?

What's weird is that before opening your envelope, the situation is completely symmetrical. Yet you know that when whatever happens once you open the envelope, you'll want to switch.

Well, that's weird, but the reason it's paradoxical is this: since each expectation $E(X_2\mid X_1=x_1)$ (for varying possible $x_1$) is greater than $x_1$, the expectation of $X_2$ should itself be greater than the expectation of $X_1$, right? It's a tree diagram, you're just summing over each possible value of $X_1$.

How can $E(X_2)=8/7\cdot E(X_1)$ but at the same time $X_1$ and $X_2$ are completely symmetric?

Hopefully some light bulbs are turning on in your head -- indeed:

$$E(X_1)=\sum_{n\in\mathbb{Z}_{\ge0}}\frac14\left(\frac34\right)^n2^n=\infty$$
Yep. That's why the situation is symmetric. Because eight sevenths of an infinity is still an infinity.

Introduction to Bayesian inference

When the Higgs boson was discovered by the LHC, you heard comments e.g. here that "there is no way to definitively talk about the probability of the existence of a Higgs boson" -- more specifically, the quoted physicist in the linked article claimed "there is no way eliminate the conditional".

This is just wrong. There is a way to eliminate the conditional -- it's just Bayes's theorem!

Imagine setting up a tree diagram for this experiment. (The LHC made a certain observation which is highly correlated with the existence of the Higgs boson, which is what we're expressing below.)

As a graphical interpretation of Bayes's theorem:

$$\mathrm{Pr}(\mathrm{Theory}\mid\mathrm{Data})=\frac{\mathrm{Pr}(\mathrm{Data}\mid\mathrm{Theory})}{\mathrm{Pr}(\mathrm{Data})}\mathrm{Pr}(\mathrm{Theory})$$

What this needs, however, is a prior distribution $\mathrm{Pr}(\mathrm{Theory})$ on the probability of the Higgs boson existing -- before any data is supplied.

And this distribution obviously exists. If you've ever heard a physicist say "we were already pretty sure it existed, this experiment just increased our confidence in it", then you know what I'm talking about. This "pretty sure"-ness in its existence is a prior distribution.

OK -- but how does this prior distribution exist? Why were physicists pretty sure that the Higgs boson existed?

Well, there's a reason they postulated a Higgs boson in the first place -- to explain the mass of certain particles, etc. And they had already observed that those particles have a mass -- i.e. they had already collected some data.

This prior distribution is the posterior distribution of a previous experiment.

---

But... you're saying that the prior distribution is inferred as the posterior distribution of a previous experiment, which itself depends on the prior distribution of that experiment... isn't it just turtles all the way down?

Well, as you might have guessed, we need a true prior distribution on all physical theories about the universe, before collecting any data. This prior is essentially arbitrary, which might be philosophically troubling for you, but that's just how things are. There is no right prior distribution, it's fundamentally subjective.

But I'd say most people would adopt a prior distribution based on Occam's razor, giving higher priors to simpler theories -- e.g. even though all existing data agrees with both "string theory" and "string theory until New Year 2020, and then the world ends", we give higher priority to the first one, simply because it takes "fewer lines of code" to write. If you're interested in this stuff, I recommend reading about:

• Occam's razor
• Kolmogorov complexity
• Solomonoff's theory of inductive inference

I suppose Solmonoff's is not the only way to set priors -- a principle that I have heard of but do not really understand (and don't bother with, because it seems really boring) is the principle of indifference.

---

This perspective on inference, though, requires the following key fact about Bayesian inference, known as the stability of Bayesian inference: feeding some data, then feeding some other independent data, is equivalent to feeding both data together (the independence is important, because e.g. if you fed in the same piece of data twice, that should not affect your distribution). We can prove this fairly easily:

Given a prior distribution $\mathrm{B}_0(\theta)$ on theory-space values $\theta$, the posterior distribution upon observing some data $\delta_1$ is:

$$\mathrm{B}_1(\theta)=\frac{\mathrm{P}(\delta_1\mid\theta)}{\sum_\phi \mathrm{P}(\delta_1\mid\phi)\mathrm{B}_0(\phi)}\mathrm{B}_0(\theta)$$
Which then becomes the prior distribution for a subsequent observation of $\delta_2$:

$$\mathrm{B}_2(\theta)=\frac{\mathrm{P}(\delta_2\mid\theta)}{\sum_\phi\mathrm{P}(\delta_2\mid\phi)\mathrm{B}_1(\phi)}\mathrm{B}_1(\theta)$$
Substituting in $\mathrm{B}_1(\theta)$ and simplifying:

$$\mathrm{B_2}(\theta)=\frac{\mathrm{P}(\delta_1\land\delta_2\mid\theta)}{\sum_\phi\mathrm{P}(\delta_1\land\delta_2\mid\phi)\mathrm{B}_0(\phi)}\mathrm{B}_0(\theta)$$
Which is precisely what we wanted.

Apparently there are systems of probability theory called noncommutative probability in which this is not possible, and statistical inference is not possible -- see (Redei 1992) (isn't it weird how everything in probability and statistics is so recent?). Obviously, this is not relevant to the physical applications of probability.

---

If you want to see Bayesian inference in action, have a look at this interactive RShiny applet I wrote that demonstrates Bayesian inference from a continuous stream of data (relying on this stability of Bayesian inference). Here's a snapshot from the applet, to pique your interest, of the evolution of this belief distribution while tossing a coin that you gradually learn is pretty unfair: