Causation, transfer and symmetry

In this article we attempt to build intuition for disentangled representations, and the closely related ideas of transfer learning, equivariant learning and causation.

• Transfer learning and causation
• Transfer learning and hierarchical models; counterfactual reasoning
• Out-of-distribution generalization and Independent Causal Mechanisms
• Disentangled representations and symmetry

Transfer learning and causation

Transfer learning is set in the context where two models are not independent. An example we've previously discussed is semi-supervised learning, which analyzes the correlation between $P(X)$ and $P(Y\mid X)$ (viewed as random variables from some distribution $\phi_{p_{x,y}}$ on possible distributions). As with the semi-supervised learning example, one way to think about "our belief about a distribution" is to consider that distribution as parameterized by/conditioned on some third variable $Z$ and talk about our beliefs on $Z$. 

More generally, we may consider correlations between arbitrary such distributions. For $P(Y\mid X)$ to be independent of some $Z$ is an equivalent way of saying that $Y\perp\!\!\!\perp Z\mid X$ -- in a faithful causal model, this means that all causal paths between $Y$ and $Z$ are blocked by conditioning on $X$. Thus the following diagrams represent the only possible structures where $Z$ carries information about $P(Y\mid X)$ and transfer learning is only possible if there exists some $Z$ that plays this role for both distributions:

Transfer learning and hierarchical models; counterfactual reasoning

The classic example of a hierarchical model goes something like this: you're trying to model the score of students based on a couple of parameters, and the students can be divided into a bunch of schools. While the school category is itself a variable with some value and can thus be considered a parameter, our treatment is often different in application -- in application, we may e.g. be faced with an entirely new school we haven't heard of before, or with a school with few data points. Then our predictions about such a school should take into account what we know from the other schools.

A very simple hierarchical model might look like this:

-- i.e. instead of simply describing score as having a particular distribution $f_\mu(x)$, we recognize that $\mu$ is a random variable in itself and consider $f(x\mid\mu)$. The main idea of importance in a hierarchical model, however, has to do with the inference of $\mu$ for a particular sample. The full model for something like this looks like:

And we would know correlations between the distributions $P(\mu_1)$ and $P(\mu_2)$, representable by another random variable $\theta$ -- essentially our $\mu$s are sampled from some unknown distribution, and the identity of this distribution (e.g. a parameter) is $\theta$.

Another place that this idea -- of "extending" samples as random variables in a causal diagram -- is of importance is in the context of counterfactuals. A classic counterfactual question looks like this:

X is elected president, and the GDP becomes $1. What would the GDP of the country have been if Y were elected president instead?

To understand the precise meaning of such a question (so as to formalize it as a causal diagram) requires us to think about the purpose of the answer to the question. For example, we might be interested in a future election, or the use of President X's vs President Y's policies in some other time or country, etc. In general, we are interested in an alternate world whose mechanisms are correlated with this one.

A particular observation of President-1 and GDP-1 gives us information on Mechanism, which gives us information to make inferences in the counterfactual world.

Relevant references: [1] [2]

Out-of-distribution generalization and Independent Causal Mechanisms

Ordinary machine learning involves some random variable $(X, Y)$ which we sample IID and attempt to learn the underlying joint distribution of -- or some functional of the joint distribution, like $P(Y\mid X)$. Instead, we might be interested in sampling from several random variables $(X_i, Y_i)$ with different distributions and exploit this full information to infer desired distributions. The problem of transferring information to some new distribution $(X_{n+1}, Y_{n+1})$ from which no data points have been sampled is called domain generalization, while combining this prior with information from the sample of the specific task $(X_n, Y_n)$ is called multi-task learning.

This general setting is exactly the same as that in the first section, of course. However, having prior information on the precise correlation between the distributions of the mechanisms $P(Y_1\mid X_1)$ and $P(Y_2\mid X_2)$ is practically quite difficult; we are often interested in the case where we plainly know that these mechanisms are identical. You may immediately see the connection to independent causal mechanisms. Indeed, the precise connection has to do with the idea of extending samples as random variables, as in the previous section. See [1] [2] for details on precise algorithms for such tasks.

Disentangled representations and symmetry

[article to be extended]

Relevant references: [1] [2] [3] [4a] [4b].

Causation and semi-supervised learning

Table of contents

• Semi-supervised learning
• Semi-supervised learning and causation
• Independent noise and causation
• Hierarchical models and transfer learning

Semi-supervised learning

We have previously discussed that machine learning can both be considered to be an information theoretic problem of finding an optimal representation. Most of it had to do with unsupervised learning, while supervised learning requires a notion of optimality with respect to some target variable (formalized via the information bottleneck). In particular, we're interested if the unsupervised representations are correlated with the supervised ones.

More specifically, suppose you're trying to classify points into some number of categories (i.e. learn a (random) function $f:X\to Y$), and have some number of labeled points (values of $f(x)$ for some $x$). Then when -- if ever -- will a further sampling from $X$ (i.e. without labels) help you achieve greater classification accuracy? Does $P(X)$ provide information on $P(Y\mid X)$?

Image source

So for example, if you believe that clustered $x$ tend to have similar labels, then $P(X)$ gives information on $P(Y\mid X)$. Formally, we have a belief distribution $\phi_{p_{x,y}}$ on the possible distributions $p_{x,y}$ that generate $X, Y$ -- so the above hypothesis represents a high prior probability for distributions $p_{x,y}$ that predict a high probability for nearby $x$ to produce nearby $y$. The random distributions $p_x$ and $p_{x\mid y}$ can be extracted as function(al)s of the random distribution $p_{x,y}$, and so it makes sense to talk about their independence -- and if they are independent, semi-supervised learning doesn't work.

Semi-supervised learning and causation

One may think about the physical mechanisms underlying such data. Suppose that $P(Y\mid X)$ does give us information about $P(X)$ -- e.g. we believe that if $Y$ tends to cluster with $X$ then $X$ has a correspondingly clustered process -- then that tells us that either $X$ and $Y$ have a common cause, or that $Y$ causes $X$.

Why? Let's be a bit more concrete about this and suppose that $X$ and $Y$ represent altitude and temperature (of some weather stations) respectively. And suppose we believe that observing a distribution like this for $X$:

Made it likely for the relationship between $X$ and $Y$ to follow the same clustering, i.e. to look like this:

Would that then be consistent with a model in which altitude causes temperature?

No. In fact, one way to think about "our belief about a distribution" is to consider that distribution as parameterized by (conditioned on) some third variable $Z$ and think about or beliefs on $Z$. For $P(X)$ and $P(Y\mid X)$ to be "independent" means that there is no $Z$ such that $P(X)$ and $P(Y\mid X)$ both depend on $Z$, i.e. for all variables $Z$, either $X$ and $Z$ are independent, or $Y$ and $Z$ are independent conditioned on $X$:

$$\forall Z, (X\perp\!\!\!\perp Z)\lor (Y\perp\!\!\!\perp Z\mid X)$$

<=> For any $Z$ correlated with $X$, $Z$ affects $Y$ only through $X$.

<=> If there is an unblocked causal path between $X$ and some $Z$, then conditioning on $X$ blocks all causal paths between $Y$ and $Z$.

<=> If there are unblocked causal paths between some $Z$ and $X$, $Y$, then the paths between $Z$ and $Y$ contain $X$ as either a mediator or a common cause. 

<=> If there are unblocked causal paths between some $Z$ and $X$, $Y$, then the paths between $Z$ and $Y$ contain $X$, and $X$ is a cause of $Y$.

<=> $X$ causes $Y$ and they have no common causes.

This is called the "Principle of Independent Causal Mechanisms" -- if semi-supervised learning doesn't add anything, then there is a defined causal direction $X\to Y$.

(Our treatment of the Principle of Independent Causal Mechanisms is similar to that by Sokolovska & Wuillemin [1]. We have not covered the algorithmic formulation of the principle -- worth reading: in this regard Janzing et al on what ICM translates to in physics [2], Besserve et al on group-theoretic formulation [3]. Furthermore, semi-supervised learning can be considered a special case of transfer learning, relevant papers in this area/on the theory of transfer learning include: [4] [5] [6 -- p. 177])

Independent noise and causation

Although we have formulated the notion of $P(X)$ and $P(Y\mid X)$ being independent in terms of our beliefs about what those distributions might be, such beliefs can also be updated by data. Indeed the characterization above, in which the variable $Z$ is used for parameterization, is one such formulation. 

If there are unblocked causal paths between some $Z$ and $X$, $Y$, then the paths between $Z$ and $Y$ contain $X$ as either a mediator or a common cause. 

The contrapositive of this reads:

If $Y$ depends on $Z$ not through $X$, then $X$ and $Z$ must be independent.

<=> $X$ is independent of the noise term in $Y$.

Indeed, this is one of the most common characterizations used to establish causation when a full picture of the causal model and its specific variables is not known. The idea is that the noise term captures the relevant stuff from the unknown variables in the true causal model.

(See Goudet et al [4] for the standard treatment of causal inference from noise.)

Machine learning and information theory

A simple and abstract way to look at machine learning is as a method to find optimal data compression algorithms (and indeed, data compression can be used as a measure of general intelligence). Fundamentally, the following two goals are equivalent: (1) approximating the distribution of some data (2) finding the optimal representation (compression algorithm) for said data -- the optimal representation is just the entropy encoding given the distribution.

Note that all of this is said about unsupervised learning algorithms -- generative modelling, dimensionality reduction/clustering, etc. -- but such structure in the data can also be uncovered during supervised learning (e.g. decision trees can emulate hierarchical clustering, feed-forward neural networks embed data into its feature representations), when the desired labels depend strongly on the clustering.

In supervised learning, you are no longer seeking to simply find an optimal compression for the data, but a compression that "agrees" with the labelling -- you are no longer seeking a representation that best predicts the original data, but a representation that best predicts the labels. In other words, in unsupervised learning you are trying to find a representation of fixed length $T(X)$ that maximizes the expected information that a representation provides on the data, the mutual entropy $I(X;T(X))$; in supervised learning you are trying to find a representation $T(X)$ that maximizes the expected information that a representation provides on the labels $I(y;T(X))$. This is how information gain appears in decision trees. 

(See Minimal Achievable Sufficient Statistic Learning. See also information bottleneck method, a generalization of minimal sufficient statistics to allow for representations that lose some information about the labels.)

There is yet a third place that entropy functions appear in machine learning, and that has to do with proper scoring rules for making probabilistic predictions. 

Suppose you were betting on an election, and you believed there was a 70% chance of a candidate winning -- under a classic betting scheme, you should put all your money on the winning candidate to maximize your expected earnings. Thus the fraction of money you bet on a candidate (which is 100%) does not reflect your predicted belief of them winning the election (which is 70%). In order to have your bets reflect actual probabilities, one needs to use a proper scoring rule to score your prediction. The only additive proper scoring rule is $\log(q)$, where $q$ is the probability you assign to the outcome -- so where $p$ is the actual probability distribution of the outcomes, the score is $\sum p \log q$, which is the cross-entropy. The same applies to machine learning algorithms making probabilistic predictions for the labels.

Riddles and mental models

I've been thinking about some assorted cognitive processes over the past week, trying to remove the mysterious/magical feel to cognition. I have no idea if my observations reflect the cognitive science literature, I don't know if what is true of my mind is true of others', and I haven't done any actual serious testing of my guesses, but as long as the ideas described are a possible description of reality, they should work as an acceptable basis for doing AI, which is the main purpose of cognitive science anyway.

Observation: thoughts are wordless

People often describe their thoughts in words, and often insist that their thoughts are in the form of words mentally. This is obviously false, because (1) words need to be rooted to some notion of meaning, and (2) you need to already know what the sentence you're about to think is before you think it, or you won't get into the right grammar, etc. 

So "thoughts", whatever they are, are not words. So what are they? Let's think about some example thoughts one may have, in the form of words:

• "I'll find carrots tastier than cucumbers, so let me eat it."
• "Alright, let's focus on this for now."
• "It's sunny, let me close the curtain."
• "What are thoughts?"
• "Saying the words What are thoughts? won't help me progress on the question."
• "What will help me progress on the question?"
• "What thoughts am I having right now?"
• "I need to develop a stronger intuition for this"
• "Wait, I thought thoughts weren't words, what's this?"
• "Stupid question, never mind, but I should add that clarification to the sentence above."
• "Ah, stupid spelling mistake."

I think that they key abstraction here is the idea of mental theories and models. Thoughts are beliefs and reasoning about implications between mental models -- i.e. a very model theoretic concept. This is true for thoughts about how things are, thoughts about decisions to make, introspective thoughts (these are just self-referential sentences), whatever.

One may consider these model theoretic sentences to be generalizations of linguistic sentences -- a sort of language that every conscious being naturally has and vividly imagines, lives in. It is not necessary to talk about how your brain attaches meaning to sentences in these models -- associates them with reality, because your brain lives in these models, that is its reality, it is the source of all meaning. One of these mental models is the very visual picture you see of your surroundings.

Observation: riddle solving.

There are two kinds of riddles -- one, the ancient Greek kind, like "what has four legs at dawn, two legs at noon, three legs at dusk, one foot stuck in its skull, and zero legs after time-traveling back to dawn?" and "what comes first? the chicken or the egg?". These are stupid and pretentious, and have nothing interesting to tell us.

Two, the kind of riddle that people scoff at as childish and uncultured, and is therefore actually somewhat interesting. Some examples:

• A bus driver was heading down a street in Colorado. He went right past a stop sign without stopping, he turned left where there was a "no left turn" sign, and he went the wrong way on a one-way street. Then he went on the left side of the road past a cop car. Still - he didn't break any traffic laws. Why not?
• Samuel was out for a walk when it started to rain. He did not have an umbrella and he wasn't wearing a hat. His clothes were soaked, yet not a single hair on his head got wet. How could this happen?
• A pet shop owner had a parrot with a sign on its cage that said "Parrot repeats everything it hears". Davey bought the parrot and for two weeks he spoke to it and it didn't say a word. He returned the parrot but the shopkeeper said he never lied about the parrot. How can this be?
• The 22nd and 24th presidents of the United States of America had the same parents, but were not brothers. How can this be possible?
• What goes place to place yet stays in one place?
• What is harder to catch the faster you run?

(Source: Riddles.com, a site I frequented as a kid.)

(There's also a third kind riddle, having to do with pattern matching, like What did the ___ tell to the ____? and Why did the _____ _____?)

When you hear about the riddle about Samuel and the rain, your brain immediately infers a particular mental theory from the problem text -- and very quickly ends up at a contradiction (or confusion), something that eliminates all possible models. In a sense, riddles are all about the art of noticing confusion -- about identifying the axiom you subconsciously assumed in your reasoning. You subconsciously assumed that the bus driver was in a bus, that was included in your mental picture, but was not implied by the form of the question itself.

The last two appear different, but are based on the same principle of being able to infer multiple possible theories and evaluate their consequences.

Overview of neural network architectures

(This article was initially in the form of several disparate articles, which I've compiled into one because they were rather short and trivial on their own, and didn't communicate significant insights.)

Below is a list of some basic "features" of human cognition that come to mind upon a cursory introspection:

• Data type specific processing: the brain has specific mechanisms to handle visual and audio data, based on hardwired assumptions about how such data must look like.
• Data streams: The brain does not download a chunk of data and process them in an egalitarian fashion -- we get a continuous stream of data, and learn continually from it. Our brain has a notion of time -- things we saw in the past affect what we see now, and what we see now affects our beliefs about what we saw in the past. In particular, we have memory.
• Environment interaction: aka "the world is a game". Most of the feedback we get is not in terms of pre-prepared labels, but is feedback from the environment, from experimenting with the environment.
• Generation: Our brain can come up with new things: artwork, ideas, thoughts, etc. 
• Making connections: I have often emphasized the importance of transferring "insights" from one academic area to another, etc. (e.g. in mathematics, in engineering) -- but this also occurs at a much more basic level, such as sharing part of a classification algorithm for different scripts.

It is worth noting that these are "orthogonal" architectures -- we want to be able to process streams of images and audio, to respond to a stream of environment feedback, to generate images and streams of audio, language and robotic commands, to apply knowledge learned from previous data streams we processed, etc. In this spirit, I will often use terms like "standard neural network" to refer to a neural network "that doesn't possess the architecture currently being considered", rather than a specific model.

---

Data type specific processing: Convolutional and recurrent neural networks

When processing visual data, it seems that to simply flatten the image and feed it as a vector is a bit disappointing. I mean, it works -- but remember what I said about the Bayesian prior? The network should a priori understand what the "inherent structure" of a data type is, so that it is more inclined towards more likely models.

The need for convolutional neural networks arises from this need to "bias" our network in favour of learning features that are more likely to be useful -- doing things that a Bayesian prior would do in lieu of actually having some Bayesian inference. More specifically, we want to represent the following prior knowledge about the features we want to learn:

• "Features are based on local interactions" -- you are likely to be interested in linear combinations of neighbouring points. Further, there is a hierarchial nature to this, in that we are further interested in the interactions between nearby features, etc.
• "Isotropy" -- the network shouldn't be overfitted for centered characters, etc. Even if it is, the bulk of the network should not be overfitted, i.e. relevant features should be identified throughout the image, and any overfitting that occurs in the last few layers can then be corrected through transfer learning (this is analogous to how humans learn, as well).

The second point corresponds to the idea of parameter-sharing ("basically all" points should undergo the same processing), and the first tells us that the exact kind of processing done should be based on local interactions, i.e. the precise notion of a convolution. Furthermore, the "hierarchial nature" of image processing leads to the notion of either pooling or stride, which brings distant features closer together. 

(Source: https://subscription.packtpub.com/book/game_development/9781789138139/4/ch04lvl1sec31/convolutional-neural-networks)

(Short project ideas: Check the claims above:

• Garble/permute the rows and columns of the images and see how that affects the training accuracy with convolutional networks (make sure you apply the same permutation to all images!) 
• Train a CNN on MNIST digits then do transfer learning on randomly off-centered digits with the convolutional layers fixed and check that you are able to get a similar accuracy to before. 

(Link to Colab Notebook where I perform these tests))

(Quick note: There are two forms of pooling, max-pooling, which tests for presence of a feature, and mean-pooling, which tests for sustained presence of a feature. Max pooling may be better if you don't have padding, otherwise it doesn't generally matter which form of pooling you use, as the convolution is capable of spreading a required feature to nearby pixels of a layer.)

In general, the notion that the fundamental operation of the neural network (affine transformations with vectors, convolutions with images) should depend on the "type" of the input data, is an important one. 

Once one understands that convolutions are the "fundamental way" of dealing with images, we should simply write convolutional neural networks in the abstract as any standard neural network, i.e. thinking of these "blocks of convolution kernels, plus bias kernels" as the appropriate generalization of "weights, plus biases".

Convolutional neural network: each block in the network is a vector of images, i.e. an image with channels.

A very important data "format" to care about is that of a data stream. This is something crucial to any AI that has a concept of time. All the data received by humans is in the form of a data stream, so it is easy to see that processing such data is important when attempting to replicate human tasks.

The natural structure on a data stream is given -- much like the natural structure on an image is given by notions of closeness -- by the flow of time. 

Note that simply intuiting the "structure" of the data type doesn't uniquely tell us what the network architecture should be. In fact, the cases of the convolutional and recurrent neural networks aren't even totally analogous -- recurrent neural networks are actually necessary for data streams, because of the input size not being fixed. Nonetheless, principles like parameter sharing are somewhat analogous in each context. 

Recurrent neural network architecture

The "parameter sharing" that goes on here is that the the parameters of each horizontal slice of the network are the same. Of course, here this parameter-sharing is required by the lack of fixed input size. It's also worth noting how this isn't analogous to convolutional networks, importantly: the information about nearby cells is fed into a layer by neurons in the same layer (as opposed to convolutions, which would have diagonal arrows). Nonetheless, the idea that our processing of a data stream is in some sense "consistent" or "uniform" over time somewhat motivates our understanding of this architecture.

Wrapped version of recursive neural network depiction. Input is a data stream.

(In this case, the unwrapped depiction is a better mental model when training the network, as the graph is acyclic, so you can apply backpropagation normally.)

"Bidirectional recurrent network", for updating your knwoledge/memory based on new information. This network has two "memory canals", carrying information forward as well as backward. The wrapping suppresses it in the depiction, but the second memory canal feeds into itself in the opposite direction as the data stream.

---

Data streams: Recurrent neural networks and Turing-completeness

One way to understand recurrent neural networks -- as we did above -- is as the natural algorithm for processing the specific data format called "data stream".

Of course, this isn't really "all there is", The precise structure of the recurrent neural network seems somewhat arbitrary, and it isn't truly completely determined by saying "it's the natural way to process data streams". Some genuinely new structure is seen here, and we should ask for a corresponding universal approximation theorem for recurrent networks.

The question to ask is: what exactly is the new structure seen in RNNs? How, precisely, is it different from standard feedforward networks?

A standard feed-forward network seeks to simulate functions, right? So the "universal approximation theorem" says that a neural network can approximate any function, up to something. So what does a recurrent neural network seek to simulate? What is a "function on a data stream"?

To answer this, we should talk about what exactly a function in the sense of computer science is. A function cannot really "take in a data stream" in computer science. A data stream that has not yet been fully fixed/captured is not a valid input variable, it's not a valid input data type.

What we're trying to simulate isn't really a function -- it's a program. A function is an example of a program, but a more general computer program can actually look at a data stream as it's running and continually update its output based on it. And the analog of universal approximation for programs is Turing-completeness, which RNNs do possess (as proven in [Siegelmann & Sontag, 1992]).

(A short project you can do to test the claim that RNNs indeed simulate programs: check that an RNN tested for a certain length of data input works reasonably well on inputs of different sizes. Can you do this with a CNN?)

You might get the sense that the RNN architecture we've discussed doesn't really feel the same as the way we process streams of data. It seems too generic, like there are more specific tasks we always do in our minds while processing some stream of audio or video. Like we need a better prior, to tell the network the exact nature of what it means to "mix" past and present features.

With the human mind, we have a very specific notion of memory -- specific actions to add and remove things from our memory. This construct is of importance, while watching a movie, holding a conversation, or reading a sentence. It's not just recalled memory, that is stored somewhere and accessed through searching for keywords in the mind, but actively held short-term memory that is particularly relevant in this context. During any of these activities, the human mind will never adopt any other "mixing" mechanism between knowledge from different points in time, that doesn't go through memory.

This is the idea behind Long Short Term Memory (LSTM) networks. Every recurrent "layer" of an LSTM network actually involves the following three computations:

1. The forget filter -- based on current input, the network drops of some elements in the "memory channel" (called the cell state in LSTM)
2. The remember filter -- the network adds some features of the current input to memory at varying intensities. 
3. The output filter -- the network allows some of the current memory to pass to output.

I call these "filters" and use language like "dropping elements" and "some features", but it is to be noted that these are really about differences in intensity, and involve multiplying by the output of trainable sigmoid layers that decide how much of your memory/features you want to allow to pass through.

Source: Understanding LSTM networks by Christopher Olah

As an exercise, figure out which parts of the network correspond to which of the filters I've mentioned.

(Short project idea: I'm actually not completely sure if I understand whether the bottom channel is needed, i.e. whether its contents must be transfered to the next iteration in the sequence. Experiment with this alternative on standard applications of LSTM and compare the performance.)

---

Environment interaction: Reinforcement learning

With standard neural networks, feedback is specifically provided in the form of predetermined data and labels that the network is required to predict.

Reinforcement learning can be seen as a generalization of this, where the feedback isn't prepared by something complicated like a human, but instead is the result of dynamic interaction with an environment (or game). The environment typically follows some laws (i.e. the laws of physics, or the laws of a game -- for robotics and game bots respectively), and this automatically generates massive amounts of data, and computes an action's consequences, which act as a generalized notion of data "labels".

A problem is immediately clear with this method: how do you differentiate an environmental response against your network's parameters? This is not a minor technical problem -- differentiation fundamentally requires that you know what happens if you change your parameters a bit.

E.g. suppose you have a network that takes in the current state of a game and outputs a real number between 0 and 1, representing the probability that it tells the agent to "jump". Then you can differentiate each decision with respect to the parameters; however, you cannot differentiate the outcome (win/loss) with respect to the decisions.

The solution to this problem comes from recalling that we are trying to maximize an expected score, so we should be doing some sampling. More formally: we should play around with expectations.

Let $Y$ be the random variable representing the agent's decision (i.e. "jump" or "not") for some given input $x$, with probability function $P(Y|x, \theta)$ where $\theta$ is the network's trainable parameters. Then where $L(x, Y)$ is the environmental loss function, we are interested in minimizing $E(L)$.

\[\begin{align}
  {\nabla _\theta }E\left[ {L(Y)} \right] &= {\nabla _\theta }\sum\limits_Y^{} {P(Y)L(Y)}  \\
   &= \sum\limits_Y^{} {{\nabla _\theta }\left[ {P(Y)} \right]L(Y)}  \\
   &= \sum\limits_Y^{} {P(Y)\frac{{{\nabla _\theta }P(Y)}}{{P(Y)}}L(Y)}  \\
   &= \sum\limits_Y^{} {P(Y){\nabla _\theta }\left[ {\log P(Y)} \right]L(Y)}  \\
   &= {E_Y}\left[ {{\nabla _\theta }\left( {\log P(Y)} \right)L(Y)} \right] \\
\end{align} \]
So the solution is as follows: sample a large number of gameplays. Now pretend that each decision contributed directly to the victory and optimize them -- encourage all the moves in winning gameplays and all the moves in losing gameplays, i.e. update the parameters by the average value of $ {{\nabla _\theta }\left( {\log P(Y)} \right)L(Y)} $ across the sample.

So despite the bizarreness of pretending that every move in a winning play was correct and every move in a losing play was wrong, doing this for a large sample makes incorrect learned features cancel out -- a good move is expected to produce better results when all other moves are averaged out, and a bad move is expected to produce worse results when all other moves are averaged out.

This strategy is known as policy gradients, and is a general technique to deal with non-differentiable feedback.

Image source: Andrej Karpathy

(You may notice that this is incredibly inefficient. Indeed, this article only covers the most basic and superficial elements of cognition -- the human brain is capable of reasoning, and of producing a highly abstracted model of the game in its mind, and of transferring intuition from elsewhere onto the game.)

---

Generation: Generative (matching and adverserial) neural networks

Equipped with the ability to process data, the obvious next step is to get an AI to produce things -- to get an AI to be creative. To come up with art, compositions, original thoughts and ideas. We'll now describe the most elementary of such neural networks, which we will call Generative Neural Networks, while more complicated ideas would exploit some sort of transfer learning.

It's not at all absurd to expect it to be possible for a neural network to generate images of horses that don't look like any horse it's actually seen -- because humans can do that! If you imagine a horse, it's probably not a horse whose image you've seen before, but it nonetheless possesses the features you've identified as common between horses.

The idea behind a generative neural network can be motivated from the following two statistical notions:

• The inverse transform method of generating random variables.

Content generated by a mind can be considered to be a random variable in some fancy space. E.g. if we want to get our neural network to produce (28, 28) digit characters, we're training it into a random variable on the space of (28, 28) images whose support is the images we identify as valid digit characters.

The way that computers typically sample random variables is through the "inverse transform method", which is to start with a uniform random sample and apply $F^{-1}$ to your sample where $F$ is the CDF of the random variable $X$ you want to sample. Your result will be a sample of $X$.

Quick explanation of inverse transform method: under the uniform distribution, the probability of getting a value under $u$ is $u$, which under the CDF of $X$ is the probability of getting a value under $F^{-1}(u)$. So you map $u\mapsto F^{-1}(u)$.

So we once again need a function approximator -- to approximate $F^{-1}$.

The inputs to the neural network are randomly generated, typically uniformly

• A nonlinear generalization of principal component analysis

Think, e.g. of eigenfaces. If you've ever tried to use eigenfaces to generate realistic faces, you'll notice that your results are just terrible. Much of the information in faces is not so linear and nice -- there's no reason to expect it to be. Illumination and angle are pretty much the only properties that can be expected to vary linearly.

I.e. suppose you have some data that varies as follows:

Then a PCA might give you the pink line as your first principal component, but sampling from the pink line gives you a lot of unphysical outputs, those are the areas where your pink line doesn't intersect the data.

But using PCA to generate samples from a distribution can be understood as taking some random inputs, corresponding to the values of each principal component you want to use, and feeding them through a function, the principal component change-of-basis matrix. 

But more generally, replacing this function with something nonlinear allows us to deal with nonlinear models.

OK,so it's clear to us that we need a neural network -- a "generative neural network" -- to construct the inverse CDF. How would one train this network?

Given an initial random guess for the network parameters, what we have is some guessed distribution for "images of horses". And what we really want to do is perturb these parameters to match the distribution of our data.

Well, such an approach is certainly possible -- one could measure some notion of distance from our generated sample distribution and the real distribution and backpropagate this error with each iteration.

Note that we don't actually know the distribution of our data (the red one), so we can't really use something like "the probability of observing this sample given our distribution" as our loss function. Anyway, there are measures of the distance between two distributions that we could use for our error function, such as the maximum mean discrepancy approach, and methods involving moments.

(This approach, generally, is called a Generative Matching Network.)

But being arbitrary is generally disappointing in machine learning, and it's worth asking if there's a way to get the network to learn to discriminate between distributions.

Here's an idea: we could just subjectively tell that the outlier point did not belong to the distribution. We used our human brains. How about rather than defining a discrimination function, we trained a neural network to tell if a given data point could belong to a distribution? Then this neural network would train our generative neural network, and vice versa.

And this makes sense, right? When we learn to draw an object, we're also simultaneously learning to identify one.

And one could imagine showing off a generative network's results and having people guess if they're real or not (alongside actual real images of course) -- and based on whether they thought it was real, we could use it to train the network. These "people" are precisely what a discriminator network is.

In other words, we have two networks: the generator network, which generates random horse faces from random uniform variables, and the discriminator network, which takes the output of the generator network and some actual images, and figures out if the result is real or not.

If the classification is incorrect, the discriminator network is punished, while if it is correct, the generator network is punished.

This is known as a Generative Adverserial Network.

(add: GANN improvements [1], deepfakes, adverserial inputs)

---

Making connections: transfer learning

Something of crucial importance to human thought is the ability to make connections between ideas, transfering ideas and results from one area to another, either directly or by "abstracting out" the analogies.

There are many "levels" on which this occurs -- examples follow:

• We don't need 60,000 instances to learn a script -- We can do with 1. It's shocking that we can learn distributional information from a single data point, and suggests that we already have an extraordinarily good prior. And we do -- our prior is continually updated from experience, of course, so this just means we're applying existing knowledge. We already know what features of the script are likely to be important and what can be safely discarded (hint: that accidental wiggle in the straight-ish line is probably noise) away. 
• Prerequisites exist. So we depend on "applying" existing knowledge in some sense to learn new things. There's a reason why babies typically don't learn algebraic geometry. 
• Making abstract connections between ideas -- Like recognizing that counting apples and counting sticks is the same task (and frivolous details about what an apple is or what a stick is can be lost for our purposes), recognizing that the algebra of kinematic quantities is the same to some extent as the algebra of quantum states (they both form vector spaces) (see Abstraction in mathematics, Abstraction in engineering), or recognizing the analogy between linear transformations of vector spaces and continuous functions of topological spaces (they're both categories).

There are probably multiple different algorithms in the brain that involve analogy and abstraction, and at least some of them (such as the first example above) have to do with learning.

This is the principle behind transfer learning, the idea that one may use the results of existing encoders for processing in other domains (and perhaps the precise encoder used may be fine-tuned for these other purposes). 

Examples of transfer learning architectures:

Progressive neural networks

This may be a bit of a disappointing solution to the question "How do brains think abstractly?" Our answer seems to depend on our active, hard-coded choice of what layers and encodings to preserve for our next task. Surely our brains do this somewhat "automatically" -- we don't actively tell our brain "Look, you've seen lines, remember? Well, there's lines here."

Well, maybe sometimes we do. It seems there may be some element of a conscious thought (whatever that means on an algorithmic level) in making analogies when it comes to complex intellectual tasks. But we certainly do not undergo any conscious thought when it comes to something like character recognition.

An architecture that may be analogous to our brain's cognition is that of a progressive neural network -- here, layers are algorithmically transferred from a previous task to another, with transferred parameters frozen. The picture below conveniently helps us make the analogy to "lateral thinking".

Source: arXiv:1606.04671

This architecture is capable of (and is about) choosing the sources of information that are most effective for the required task, but what suggests to me that this isn't what our brain does exactly is the parameter wastefulness and increasing level of model complexity (trainable parameters), which doesn't seem to be analogous with our method of learning, which seems to be more or less symmetric between learned ideas.

Multitask learning

A far simpler transfer learning approach that addresses the above concern about symmetry is multi-task learning:

Hard parameter sharing. Source: Sebastian Ruder

Soft parameter sharing. Source: Sebastian Ruder

This does intuitively seem to be the brain's transfer learning algorithm -- even if one of the tasks has previously been trained, the brain seems to be able to retrain the network as needed. And I have often observed the benefits of learning two abstractly related areas of mathematics together, e.g. Hilbert spaces and quantum mechanics, linear algebra and special relativity, topology and probability theory.

Two ways to understand overfitting (and don't peek!)

Assorted comments on overfitting

The very first time I was introduced to the notion of overfitting -- by three diagrams of regression curves on scatter plots labelled as you may guess -- I became very uncomfortable. I was probably eleven years old, and did not understand Bayesian statistics, did not realize that seeing 51 heads out of a hundred didn't imply that the coin genuinely had a heads rate of 0.51.

Or rather: I didn't realize that I did realize that. If you had made me bet on the number of heads that would come up in the next hundred flips, I would not provide odds that would indicate an honest belief of seeing 51 heads.

(Because deep down, I had a non-uniform prior.)

And the same principle applies to drawing regression curves. You may insist that your curve with all its squiggles is "unbiased" or give any one of the terms (one for every squiggle) used to describe non-Bayesian estimators, but ultimately -- that's not the curve you'll bet on. You know it's just far more likely that those squiggles are the result of noise which will not be the same (or knowably correlated) in the next sample, than for them to actually be a determining feature.

Yes, maybe the 79th coin toss will always be a tail because of a tiny AI hidden in the coin that counts, or maybe it was the result of factors that affected the 79th coin toss that you just didn't measure. While your data should affect your beliefs, they shouldn't completely overrule your priors.

And that's the key idea behind overfitting (and really the Bayesian notion of probability in general) -- how will your model, based on your data, of perform if you exposed it to data it hasn't yet seen. Because using it on data it hasn't seen is the point of your model -- that is your purpose in building it.

So overfitting occurs when a model learns features specific to your particular data set that don't generalize well. There are two ways this can occur:

• The training set is a biased sample: E.g. MNIST digits are all centered (but suppose you're testing on non-centered digits), or your medical database is all from a particular country. 
• There is noise: I mean, of course there's noise -- it's a statistical problem. Even if the world were deterministic, you still don't have all the information in the world. And you should avoid your model using this noise to make predictions, since noise is, by definition, unpredictable.

The first is a more tractable problem -- it can often be solved by data augmentation (if you have a very good picture of exactly how the data set is biased) or transfer learning (if there's a good chance the model is picking up on important features so you can just train it on the ones it missed). It's important to develop interesting transfer learning algorithms to solve this anyway, as the way humans learn often involves biased samples (e.g. personal experience) and reasoning capacity to unbias their knowledge. 

(Not that humans do always do this -- people often do form beliefs based on mere personal experience, but humans are capable of reasoning more clearly.)

The second is a problem that requires algorithmic solutions that hint our neural network towards Bayesian solutions. Solutions like "well, parameters are just a priori unlikely to be very large, so let's penalize that" (Lasso/Ridge regression), or more complicated (to explain in a simple Bayesian way) regularization algorithms like cross-validation and early stopping.

Another unclassified comment on overfitting: it's very easy to mistakenly "peek" at the test data. Simply in the act of saying "hey, this model works well on the test data, let's choose it" you are already performing a simple algorithm that checks several models and chooses them based on their performance on the test data -- i.e. you are kinda training the model on the test data, even if it is not seen in your code, just in your choice of hyperparameters.

That's why developing some kind of "theory" of hyperparameter optimization and regularization techniques is of importance, so you actually have a theoretical justification for picking your models. 

(Of course, this is hard. We've always been picking models, haven't we? For example when we decide to model something as belonging to a particular family of distributions so we only have to optimize in a 1-dimensional parameter space instead of the literal theory-space. In a sense, machine learning is the way to avoid modeling, due to the universal approximation theorem -- and the hope is that we can eventually make the priors as human-like as possible, which is the eventual goal of hyperparameter optimization and regularization algorithms.) 

A review of "Age of Em" by Robin Hanson

Robin Hanson's Age of Em is an attempted construction of a future society in which essentially all work (however you formalize this phrase) is done by somewhat AIs. Well, it's a topic I have often thought about myself, but his exploration of the idea left much to be desired.

Firstly, I found it generally unimaginative. Hanson seems to constrain himself too narrowly -- his description of the em society does not feel "radically different" from present-day society, and the society he envisions does not make full use of the technology available to it.

Some examples to illustrate this observation:

• Ems are shown to be absurdly human-like, like "intellectually" rubber-forehead aliens. He writes: "even em minds are likely to age with subjective experience..." (p. 128) A claim like this ought to be based on some foundational fact about how an AI stores memories. But there is none -- there is no mathematical law forbidding AIs from being retrained, or that requires AIs to behave similar to human brains in this sense. Similar comments apply to "em suicide" (p. 127-139) and the considerations regarding Em reproduction (p. 285): there is no reason why an em's drive or aggression must be reduced due to a suppression of its libido -- an em does not have hormones!
• Aspects of Em habitation/organization, such as "cities" and "offices" are just "copied" from human society. He writes, "It’s reasonable to guess that such habits will continue with ems." (p. 104) But it's not. There is no reason for Ems to behave in this way as humans.
• Humans and ems are shown as binary. Humans are biological and have self-ownership, ems are technological and do not. But I don't see why this ought to be so -- I would very much like to have the desires, preferences and emotions of a human, but the abilities/efficiency, immortality and unlimited VR leisure scenarios available to an em. There would still be unfeeling, specialized AIs, of course -- much like there would be computers that aren't even AIs, devices that don't even have CPUs, etc. -- but eventually almost all humans would opt for a massively extensible, upgradable robot body than a static mortal body.
• There's just a lot of interesting aspects of the civilisation that are not sufficiently explored. E.g. transportation, cybercrime.

Relevant TVtropes articles: Inexplicable cultural ties, Most Writers are Human, technological version of Reeds Richard is useless/Required Secondary Powers.

Indeed, these may be considered acceptable in science fiction, but it is important to be less "conservative" when attempting a non-fictional, encyclopedic description of a society.

Perhaps a more specific objection I have is with the entire premise of "brain scans" as the future of AI. This seems completely at odds with the direction that current AI research is headed. To use a somewhat cliche analogy, we didn't need to study how birds fly to invent airplanes. There is no reason to believe that the most efficient architecture for a "software" brain would be the same as the architecture that biological, hardware brains have evolved.

The general answer to how a software brain should work is that it should be a function approximator, such as the "neural networks" (trainable computational graphs) that are currently popular.

This point is important, as it addresses Bryan Caplan's critique re: carrot vs stick as incentive for the ems. The question of carrot and stick assumes some "natural" state of affairs that a human being will go through without intervention by the employer -- the "carrot" is an intervention that improves this state, while the "stick" is an intervention that worsens this state.

But a neural network does not have a natural state of affairs. There is no difference between training a neural network to minimize a loss function, and training a neural network to maximize a reward function: these are completely identical. There is no distinction between carrot and stick.

Here's a description that I find more satisfactory: see Age of Gen.

Age of Gen: a picture of a transhuman society

(See here for my criticism of Robin Hanson's Age of Em. This post is an alternate characterization of a futuristic transhuman society.)

Consider the following six "levels" of technology, roughly corresponding with "orders" of automation, or something like that:

1. Tools, which require the intervention of a higher-level device to perform anything useful.
2. Machines, or mechanized devices: they run on their own, but only perform "simple" tasks.
3. Computers, or devices with CPUs, which can automate processes through logic.
4. General Computers, or programmable computers.
5. AI, i.e. machine learning. They perform tasks that are hard to define. If computers are about logical inference, AI is about statistical inference.
6. General AI, which are capable of making decisions out of their free will, among other human things. 

Each of the 5 technologies will continue to exist -- much like the microcontroller in an airplane's control system has not been replaced by a full-fledged programmable device. But the General AI is the key object of interest to us -- we will call these Gens for short. These are the descendants of human beings, whether through upload or just by virtue of being intelligent.

We will refer to ordinary biological humans as Biols (although they may be variously technologically enhanced to prevent aging/death, etc.) Presumably Biols will be an small minority, if nothing else because their reproduction is far slower than that of the Gens.

---

Important futurism milestones:

1. Intelligent AI
2. General AI 
3. Optimal AI
4. Value-aligned AI
5. VR and game protocol (game development)
6. Transhuman body (robotics)
7. Mind transfers -- G2G, B2G, G2B (biotechnology, engineering)
8. Reviving the dead -- frozen, miscellaneous (biotechnology, Gen technology) 

---

Philosophy of mind

Utility functions

General AIs may have any "utility function" (or loss function, in machine learning language) programmed into them, which is relevant to an extent to how they behave (although ideally this should carry some uncertainty, as humans prefer to have free will). 

Presumably, the first humans to "convert" into AI form will choose utility functions similar to their original ones, although other systems -- incredibly foreign systems that make questions like "are all Gens human/worthy of moral consideration?" and "how do you even consider a Gen's happiness?" really hard -- may emerge. In fact, Gens may choose to adopt multiple utility functions/personalities depending on the context (e.g. a perfectly rational utility function for decision-making, but a separate human utility function while in the Duat, see the Games and Virtual Reality section).

The carrot-and-stick question re-emerges. How do you know if a Gen is really happy, given that the sign of the loss function and what "neutral" is are just matters of an arbitrary co-ordinate system? I would argue that our judgement of this as humans is also arbitrary, and that our "neutral" is just what we're used to. When discussing matters of torturing Gens, we should really be afraid of the possibility of enslaving Gens, preventing them from making decisions however they see fit. 

In other words, we should take a libertarian/preference-utilitarian approach to moral questions, rather than a naive utilitarian one, as the latter would just be ill-defined in this society (and probably in the present one too, but that's besides the point of this article). 

Identity

Regardless of how they are created (whether or not there is some element of "scanning" that goes into it), perhaps a common question is what determines the identity of a Gen -- how do you determine if the Gen that has been created on your behalf is you? 

I would be comfortable in saying that memories are the key aspect -- if you remember being you, you are you. This is the general philosophy I will refer to on multiple occasions throughout this article. However, the Gen's personality is relevant to whether it is perceived by others as the same individual.

Does operating as multiple agents with a synced memory (see Memory syncing and Mind transfers and copying) "feel" like being a single individual? What does it feel like to have one of those agents die, for example? What does it feel like to die and then have your memories be transferred onto another Gen? These are unanswerable questions to a Biol like my current self -- it is like asking a flatlander to perceive in 3 dimensions, or someone born blind to see (see Games and Virtual Reality).

Note the exotic behaviour of identity possible in a Gen society. E.g. you may only partially sync the memories of two brains, making them "kinda" the same person, or introduce various correlations between their memories. You can have an entire society of Gens where each brain is almost identical to its neighbour, but gradually very different from a faraway brain, so that you have a continuum of identity, rather than a discrete space.

Architecture of a Gen

Hardware

It is important to note that a Gen need not appear like a human in the outside world at all: at least, human-looking Gens will eventually become less and less common as virtual reality (see Games and Virtual Reality) advances further and further.

Gens are fundamentally just computers, but a Gen can be fitted with any possible peripherals, giving it various physical abilities relating to movement, observation, communication, and manufacturing. Some standard such fittings may include:

• Drone rotors
• Hand-like tools and weapons
• A repair kit

Software

Although a Gen is "most importantly" a General AI, the fact that it runs on a computer allows it the flexibility of running more specialized programs (AI or otherwise) -- basically for algorithmic and repetitive tasks.

A single piece of hardware, may, in principle, host multiple Gens. However, it is the software, and not the hardware, which should be seen as the fundamental individual, with rights.

Gen behavior

Games and virtual reality

Gens spend much of their time (clarification later on what this means) within their shells, virtually interacting with some software -- this is a generalization of both dreams and human-computer interaction, and is achieved by switching (or possibly augmenting) the Gen's I/O from the actual hardware peripherals to some simulated I/O.  

This makes available a whole new "virtual world" or platform, known as the Duat.

The Duat can be understood as a collection of games. A typical Duat game involves the Gen taking on an avatar and interacting with his environment. 

Games may be of various interface types such as:

• Virtual Reality games
• Rich text, multimedia games and tools (e.g. ordinary Internet websites and applications)
• Knowledge/training applications 
• Some completely exotic formats that Biols cannot even comprehend -- e.g. 
• The avatar may or may not have a human or even humanoid form. 
• Some exotic new senses of perception (even something like images at higher resolution than the human eye qualify, but in principle, you could have mechanisms to "feel" all sorts of things)
• A different number of spatial/temporal dimensions
• Some very exotic behavior of the locus of consciousness.

Games may be offline or online. A very large number of online, multiplayer games -- as well as realistic interactive simulations of the Earth at various points through history -- would exist, as Gens with human-like utility functions value interpersonal interaction. 

One function of the Duat would be to allow Gens to experience anything they could as Biols -- but of course, they could experience far more enhanced pleasures etc. and depending on the Gen's utility function, a Gen may have very different desires to those of Biols.

Memory syncing

Because identity is determined by memories, playing with how memories work creates the prospect for a whole host of exotic, essentially mythological notions of being both in the real world and in the Duat. 

The first such tool is memory syncing, i.e. syncing (some or all) memories between Gens -- i.e. allowing an individual to have multiple avatars, or to be in multiple places, perform multiple tasks at once. This is basically taking parallel computing to the extreme. This is also a useful backup mechanism.

Memory editing

A Gen may choose to -- perhaps temporarily -- suppress or edit some of its memories. This may be, e.g. for the purpose of highly immersive VR experiences (the Gen may want to genuinely believe he is in a haunted house, going through childhood, or discovering general relativity for the first time). 

Production, conversion and transport of Gens

Gen (re-)production

Gens are programmed as AIs and fitted with utility functions and memories. These utility functions and memories may be based on mind transfers.

Mind transfers

Mind transfers involve scanning a brain's memories and traits to install them onto another body. This includes Biol-to-Gen transfers, Gen-to-Gen transfers and Gen-to-Biol transfers. 

B2G transfers are used for the original upload process. G2B transfers may be used for backups, or if someone really wants a biological body (although such bodies will themselves probably be synthetically produced).

G2G transfers are used for backups, cloning and teleportation. 

Gen Society

Habitation and industrial activity

Real-world Gen habitation will be radically different. Entire industries present today -- most notably agriculture and healthcare -- will no longer be present. The lack of a need for agriculture in particular will free vast amounts of land for other uses. Many other industries -- education, entertainment, retail, marketing -- will be moved to the Duat or otherwise virtualized. 

Gens could in principle be (partially) self-contained -- a true rugged individualism -- with some repair facilities, energy generation, manufacturing facilities, housing facilities, etc. built into themselves. Or they may concentrate around urban facilities/cities that provide these services. This depends on the precise costs of operation of these devices versus the cost of the time needed to visit these shops, although Gen society is likely to move towards a "rugged individualism" as resource costs decline.

Gens are likely to view their software as more "fundamental" to their being, using mind transfer, i.e. teleportation for most long-distance transport. 

While intelligence basically becomes an infinite resource, the economy is still limited by the availability of physical resources, and the laws of physics themselves (most notably the speed of light, which places a limit on how fast we can expand across the universe). 

Culture

Gen culture is likely to be very diverse, and much of it completely exotic to us. Human or even Humanoid notions of race, tradition, gender, sexuality and even species are unlikely to apply in a recognizable way to Gens that adopt utility functions different from standard human utility functions. There is likely to be a great deal of diversity in the forms of interpersonal relationships.

Ethics, violence, law and government

Efficient IP markets

What makes information and knowledge markets inefficient is that there is no way to prevent a buyer from re-sharing information. I.e. there is no barbed wire for IP. Information transactions in a Gen society may involve the implantation of a small program that prevents the buyer from doing so.

Also: memory editing can be used to eliminate information asymmetry, as they allow buyers to "try out" a product and then erase their memory of the usage.

Crimes to watch out for

• Child enslavement: Creating a Gen, then subjecting them to something their utility function does not prefer without allowing them to leave. A serious issue here is definitional -- remember how I suggested (under Memory editing) that one may choose to temporarily suppress their memories for an experience? What if I decide to temporarily replace my memories and torture myself? Is the person being tortured even me? Or is it my child? Am I allowed to program the memories of this person to disappear and be replaced with mine? Or would that be taking his life? 
• Kidnapping: Similar to above, but you sync your child's memories with someone so you've basically kidnapped them. 
• Mindless destruction: With such incredible computing power available to all, how do we make sure that someone doesn't just find a way to manufacture tons of antimatter and destroy the world with it? Or, you know, just capture someone and torture them? Sure, we can develop better defense mechanisms: but how do we make sure the good guys stay ahead of the bad guys?
• Breaking encryption: Once again with such incredible computing power available, our current encryption systems are obviously going to be broken easily. Sure, we also have a greater ability to come up with better systems, but how do we make sure the good guys stay ahead of the bad guys?
• Hacking: Hacking can cause serious trouble including memory editing, getting people stuck in the Duat, torture and death. Once again: we will also have the power to develop incredibly better security systems, but how do we make sure the good guys stay ahead of the bad guys?
• Deepfakes: A problem for law enforcement, if there even is a centralized law enforcement. Evidence will have to be of a fundamentally higher standard, if a justice system is even to be a thing.
• Strategic partial suicide for obstruction of law: It is easy to game whatever legal theory of being we use in such an exotic society. E.g. if a person is determined by his memories, then a criminal could temporarily erase his memory of committing a crime and copy them to a drive, so it would be his inviolable private property, rather than a criminal person.
• Overpopulation? I don't know what I think about overpopulation, or if it's a thing. Can someone just produce a massive number of Gens that require an incredible quantity of resources, starving all the Gens and causing the entire system to completely collapse? 

In general, if we don't adopt any regulation, whoever expends the most resources into becoming most powerful would become most powerful -- things would advance just way too fast for any government structure to keep up with. Keeping ahead of all the Gens who value nothing but criminal behavior might require other Gens to value almost nothing but preventing criminal behavior.

(Part of the question is also what is physically permissible -- how good can deepfakes get? How good can a justice system get in uncovering past events (e.g. could you just calculate past states of the world from the current state)?)

How do we solve this problem?

Note that solutions to this problem need to be general, targeted towards "any" immoral behavior or rights-violation, rather than catered to the specific enumerated crimes above, as the range of possible serious crimes can be far more extensive than the ones I've described, depending on the exact physical laws (e.g. if it turns out that time travel is possible, it's essential to make sure nobody does it). The solutions also need to be airtight, unlike the laws we have today, due to the sheer destructive potential of these crimes. 

(You might think: what if we just banned the development of General AI? Well, that will fail spectacularly. It's the standard "good guys must have nukes" argument. If you don't develop it first, someone else will, and they might be the bad guys. "Okay," you say, "But I just want to make sure that a Gen isn't developed in my lifespan, so I don't get tortured." Well, please be assured that the aggressive Gen will be perfectly capable of reviving you from the dead.)

There are two general modes of solution to this problem: (1) to create economic incentive systems to regulate behaviour, like we do right now with humans (2) to align the Gen's utility system to make sure it doesn't cause the destruction of property rights.

As far as I'm aware, no specific solution in the first category has been proposed.

The second is known as the value alignment problem.

Well, you should be able to see why this problem might be non-trivial:

• The utility system should be able to "recurse", being non-evil means you shouldn't produce evil children either.
• Most property isn't privatized, so formulating what it means to destroy property, when it comes to things like "eating the milky way", is complicated. 
• On a similar note: basically every action violates property rights to some infinitesimal extent, what is known as an "externality". You need some rational economic calculations of this stuff.
• You can't just scan a human brain or something, because humans are not infallible, and are perfectly capable of criminal behaviour (while we want our Gens to have a zero probability of significant violence), which may scale particularly badly with power/ability. 
• Perhaps we should aim for (Hofstader-style) superrationality between all human beings, to e.g. prevent the creation of basilisks and prevent possible Newcomb-style aliens from gaming us.

But the general idea is that you start with a few Gens with the correct utility functions, then develop some Police Gens to make sure no humans are producing evil Gens (because a non-evil Gen by definition does not produce evil Gens, as that would cause property rights violations). One thing that helps us is that non-violence is really the only thing we care about. Everything else is just personal preference, and a Gen will be economically productive if it wants anything from other people (like electricity). And if some Gens don't want anything from other people, then they can exist without trade anyway. 

Backpropagation and the chain rule

In order to define an function approximation, we need to define a loss function, such as the mean-squared-error often used in linear regression. To minimise such a (generally highly complicated) loss function, gradient descent is a natural numerical algorithm.

OK -- so how does one actually calculate these gradients efficiently? Consider the following neural network (where each box represents a tensor, e.g. vector $X$s, matrix $W$s, whatever $y$ and scalar $L$ -- and arrows indicate that something is being fed as input through a function):

Well, you might observe that the tensors are in a chain. What do we do when we see a chain and we want to differentiate stuff? We use the chain rule.

To be more specific, we're interested in differentiating -- i.e. taking the gradient of -- the function that takes in the weights and outputs the loss (i.e. the loss function). This function can be understood as the composition of several functions -- specifically all the red arrows. But really, descending in the gradient direction is the same as descending in the respective derivative directions of each parameter. I.e. it suffices to talk about:

$$\frac{\partial L}{\partial W_2}=\frac{\partial L}{\partial X_2}\frac{\partial X_2}{\partial W_2}$$$$\frac{\partial L}{\partial W_1}=\frac{\partial L}{\partial X_2}\frac{\partial X_2}{\partial X_1}\frac{\partial X_1}{\partial W_1}$$
Or in general, for a network with layers $X_0,...X_m$ and $L=:X_{m+1}$:

$$\frac{\partial L}{\partial W_i} = \left[\prod_{k=m}^i\frac{\partial X_{k+1}}{\partial X_k}\right]\frac{\partial X_i}{\partial W_i}$$
Note that each item in this product is a tensor, i.e. involves taking tensor derivatives. This is why you see ML programming packages labeled stuff like "TensorFlow" -- what they do is keep track of derivatives for you.

Keeping track of derivatives and computing products on the spot is better than trying to come up with a general expression for the derivatives, because a generic neural network may be much more complicated than the one we've described, and may have arrows that skip a layer, etc. for which the composition doesn't even yield a matrix multiplication. In general, we may have any sort of operations involved in the network (activation functions are an obvious one), and as long as we can differentiate them, we can keep track of what we need to multiply. This algorithmic use of the chain rule is called backpropagation. 

Note how the chain rule itself has no problem at all with computing gradients for multiple data points -- we still have a loss function that is a function of the network's parameters. But with our algorithmic use, we'll need to form an expression for the gradient of the batch from each feed-forward's gradient. At least if the loss function is additive, so are the gradients.