Poisson processes: from geometric to Gamma distributions

The notion of a Poisson process is rather beautiful, and connects a number of distributions together.

To start, consider a Bernoulli process: a discrete time series with each value IID Bernoulli. Then we can study some properties of this:

• The waiting time for the 1st event is distributed geometrically. 
• The waiting time for the nth event is distributed as the sum of geometric distributions, which is negative binomial.
• The number of events in a given period of time is distributed binomially.

(You should be able to derive these distributions easily.)

(Note that this is not specific to a time series -- one has the same results for a spatial lattice or for some general abstract set of points.) 

How would one generalize this to continuous time? Well, with continuous time you really can't talk about the result for each point in time being "Bernoulli", or about them being IID. But let's do it anyway. Suppose an event has an $\mu\; dt$ chance of occurring in the timespan $dt$. Then the chance that the first event occurs at time $t$ is (by geometric distribution) ${(1 - \mu \;dt)^{t/dt}}\mu \;dt$, or: $\mu {e^{ - \mu t}}\; dt$, i.e. a probability density $\mu e^{-\mu t}$. This is called the exponential distribution.

The waiting time till the nth event is analogously just the sum of exponential random variables and its distribution can be computed through the standard MGF route. It is left as an exercise to the reader to show that the sum of exponential random variables with parameters $\mu_1,\dots\mu_\alpha$ is given by the Gamma distribution $\Gamma(\alpha,\sum\mu_i)$:

$$\Gamma(\alpha,\beta)\sim \frac{1}{(\alpha - 1)!}\beta^\alpha t^{\alpha-1}e^{-\beta t}$$
What does our notion of independence translate to? The idea that there being an event at some time should not depend on whether there was at any other time. Well, the natural way to write independence in a way that makes sense for continuous distributions is to consider the waiting time for the first event

\[P(T>t+s|T>t) = P(T>s)\]
This is known as memorylessness. Indeed, one can check that the only memoryless discrete distribution is geometric, and the only memoryless continuous distribution is exponential.

OK -- what about the number of events in the continuous case? Well, in some interval of size $T$, the probability of the number of events equaling some $n$ is (by binomial):

\[\left( {\begin{array}{*{20}{c}}
  {T/dt} \\
  n
\end{array}} \right){(\mu \,dt)^n}{(1 - \mu \,dt)^{T/dt}}\]
Which it is easy to see that equals:

\[\frac{{{{(\mu T)}^n}{e^{ - \mu T}}}}{{n!}}\]
Which is the Poisson distribution with rate parameter $\mu T$.

Here's a table of analogies between memoryless discrete and continuous processes:

	Discrete time	Continuous time
Overall phenomenon	Bernoulli process	Poisson process
Single-event	Bernoulli distribution	-
Waiting time (1st event)	Geometric distribution	Exponential distribution
Waiting time (nth event)	Negative Binomial distribution	Gamma distribution
Number of events	Binomial distribution	Poisson distribution

But isn't the continuous analog of the binomial distribution (and many other distributions) the normal distribution? Do not conflate discrete time with discrete number. Both the binomial and Poisson distributions above are discrete distributions: the Poisson is just the relevant one for the continuous-time process. These are completely unrelated notions.

Moments as tensors

We discussed the second multivariate moment a bit haphazardly in the last article. In general, we'd like a nice way of expressing the general moment (i.e. multivariate cross-moment).

Let $X=(X^1,\ldots X^n)$ be a vector of random variables, and consider their $p$th order moments ($p\le n$) -- these form a rank-$p$ tensor of dimension $n$, the moment tensor, given by:

$$Mp[X]^{j_1\ldots j_p}=\mathrm{E}(X^{j_1}\ldots X^{j_p})$$
(e.g. $p=1$ gives you the mean vector, $p=2$ gives you the badly-named auto"correlation" matrix) And the central moments form a similar tensor, the central moment tensor, given by:

$$mp[X]^{j_1\ldots j_p}=\mathrm{E}\left((X^{j_1}-EX^{j_1})\ldots (X^{j_p}-EX^{j_p})\right)$$
(e.g. $p=1$ gives you zero, annoyingly, but $p=2$ gives you the covariance matrix aka autocovariance matrix) But, well, each random variable $X^i$ can also be understood as a vector, remember? Let's write $X^i=(X^i_\alpha)$ for $\alpha$ a pseudo-index that represents the idea that $X^i$ is a vector (I guess this is really Penrose (abstract index) notation rather than Einstein notation).

Actually, let's also make the following extension to tensor notation: every Greek index is summed over, regardless of whether/how many times it's repeated and where -- and we take the expectation instead of the sum (which is like a normalized sum, or some sort of a trace). So we write:

$$Mp[X]^{j_1\ldots j_p}=X^{j_p}_\alpha\ldots X^{j_p}_\alpha$$$$mp[X]^{j_1\ldots j_p}=(X^{j_p}_\alpha-X^{j_p}_{\alpha_1})\ldots (X^{j_p}_\alpha-X^{j_p}_{\alpha_p})$$
Where we use different dummy indices $\alpha_1,\ldots\alpha_n$ to indicate that these are summed over earlier (since they're not repeated again in the expression). These changes to index notation are all an artifact of the fact that random variables are not really "fundamentally quadratic", but rather "fundamentally $p$-normed".

---

OK -- so that's the univariate cross-moment -- it can also be considered a multivariate moment, the moment of the random vector $X$ -- its mean is the mean vector, its variance is the covariance matrix, etc. What about cross moments between random vectors? And you can imagine that once we have that, we'll call it a moment of a random rank-2 tensor, and so on.

What we're really looking for is the moment of a random tensor. This is a rank $pq$ tensor where $p$ is the degree of the moment and $q$ is the rank of the random tensor. As an example, when $p=2$ and $q=2$, one gets a rank 4 tensor consisting of cross-covariance (and autocovariance) matrices.

Note that this is not at all some unnecessary generalisation -- measuring the correlation between random vectors is a thing with very significant practical implication.

For example, a time series is a random vector -- its covariance matrix represents its internal correlations (how well its current value predicts a future value), but often we're interested in looking at correlations between time series -- how does the price of gold correlate with the price of S&P 500, etc. Then this cross-covariance matrix will be a bivariate function of $(t_1,t_2)$, called the cross-correlation function.