Two-envelopes problem: beyond the Bayes explanation

The two-envelopes problem is phrased as follows:

There are two envelopes, one with twice the amount of money as the latter -- you pick one of them at random and open it to find $x$ dollars. You don't know if this was the larger or smaller sum.

Should you switch?

Though symmetry would tell you that it doesn't matter, it seems that by switching you get $x$ if the other envelope is bigger, but only lose $x/2$ if the other envelope is smaller. So you should switch. Right?

The standard explanation to this problem is that we need to pick a prior to make any actual inference, and the probability of the other envelope being bigger is usually less than the probability of it being smaller, in the sense that probability distributions approach zero towards infinity.

But this explanation doesn't really make the problem go away. Consider this prior:

$$
f(x) = \left\{ {\begin{array}{*{20}{c}}{\frac{1}{4}{{(3/4)}^n}}&{{\rm{if }}\ x = {2^n},\,\,n \in {\mathbb{Z}_{ \ge 0}}}\\0&{{\rm{else}}}\end{array}} \right.
$$
This is a perfectly valid prior distribution (the improper prior assumed in the original "paradox" would have worked as well, but I don't want to distract you with questions about distribution propriety) for which exactly the same paradox applies -- one may show (and I encourage you to work this out) that whatever value $x$ I find in the envelope, the expected value of the other envelope is $8/7\ x$.

Let's try to remember why this is weird -- what exactly is so paradoxical about it?

What's weird is that before opening your envelope, the situation is completely symmetrical. Yet you know that when whatever happens once you open the envelope, you'll want to switch.

Well, that's weird, but the reason it's paradoxical is this: since each expectation $E(X_2\mid X_1=x_1)$ (for varying possible $x_1$) is greater than $x_1$, the expectation of $X_2$ should itself be greater than the expectation of $X_1$, right? It's a tree diagram, you're just summing over each possible value of $X_1$.

How can $E(X_2)=8/7\cdot E(X_1)$ but at the same time $X_1$ and $X_2$ are completely symmetric?

Hopefully some light bulbs are turning on in your head -- indeed:

$$E(X_1)=\sum_{n\in\mathbb{Z}_{\ge0}}\frac14\left(\frac34\right)^n2^n=\infty$$
Yep. That's why the situation is symmetric. Because eight sevenths of an infinity is still an infinity.