**Aka the difference between rationalization and rationality.**

Eliezer Yudkowsky calls this the dilemma of the clever arguer: A propagandist (clever arguer) is hired to sell you a box that may or may not contain a diamond -- he tells you that the box has a blue stamp on it (which you know occurs more on boxes containing diamonds). If you could handle the box yourself, you could rationally evaluate all the characteristics of the box and compile their influences on your probability estimate Are you then forced, for each argument the clever arguer provides, to helplessly update your probabilities as the clever arguer wishes, even though you know the propagandist has omitted the evidence he doesn't want you to know of?

There are various equivalent formulations of the problem including:

- p-hacking
- Filtered evidence: an experimenter flips a coin 10 times and tells you that the 4th, 5th and 7th tosses came up heads, without telling you anything about the other tosses.

*the 4th, 5th and 7th tosses came up heads*, but instead:

*the propagandist tells me that the 4th, 5th and 7th tosses came up heads*. The statistical process we are studying is no longer the "natural" (IID Bernoulli) process we're used to, but a

*different process*, which depends on the inner mechanism used by the propagandist. For example:

- If the propagandist always tells you only the 4th, 5th and 7th tosses, you update your beliefs from this evidence as normal.
- If the propagandist only tells you the coin tosses that came up heads, then you now know that the other seven tosses come up tails, and you update your beliefs accordingly.
- If the propagandist chooses any three heads that came up to tell you about, then the probability of 4, 5 and 7 specifically being chosen is only slightly greater with a biased coin (you can calculate this, but the key point is that the process we're observing is not about which coins come up heads, but which coins are chosen by the propagandist).

*split*between information on the propagandist's mechanism and information on the coin's mechanism.

*surely*if $\mu$ were 0, the propagandist could have found a feature with a better value than 1.

**conservation of expected evidence**: the expectation of the posterior probability of each value is its prior probability.

*requirements*for Bayesian inference, but they're simply a way to ensure that scientific research is maximally informative. If you don't know the underlying process that scientists use to report their data, or if you know that they use a "biased" process, then your estimator of the relevant parameter will be less informative than it could have otherwise been.

## No comments:

## Post a Comment