Causation, transfer and symmetry

In this article we attempt to build intuition for disentangled representations, and the closely related ideas of transfer learning, equivariant learning and causation.

• Transfer learning and causation
• Transfer learning and hierarchical models; counterfactual reasoning
• Out-of-distribution generalization and Independent Causal Mechanisms
• Disentangled representations and symmetry

Transfer learning and causation

Transfer learning is set in the context where two models are not independent. An example we've previously discussed is semi-supervised learning, which analyzes the correlation between $P(X)$ and $P(Y\mid X)$ (viewed as random variables from some distribution $\phi_{p_{x,y}}$ on possible distributions). As with the semi-supervised learning example, one way to think about "our belief about a distribution" is to consider that distribution as parameterized by/conditioned on some third variable $Z$ and talk about our beliefs on $Z$. 

More generally, we may consider correlations between arbitrary such distributions. For $P(Y\mid X)$ to be independent of some $Z$ is an equivalent way of saying that $Y\perp\!\!\!\perp Z\mid X$ -- in a faithful causal model, this means that all causal paths between $Y$ and $Z$ are blocked by conditioning on $X$. Thus the following diagrams represent the only possible structures where $Z$ carries information about $P(Y\mid X)$ and transfer learning is only possible if there exists some $Z$ that plays this role for both distributions:

Transfer learning and hierarchical models; counterfactual reasoning

The classic example of a hierarchical model goes something like this: you're trying to model the score of students based on a couple of parameters, and the students can be divided into a bunch of schools. While the school category is itself a variable with some value and can thus be considered a parameter, our treatment is often different in application -- in application, we may e.g. be faced with an entirely new school we haven't heard of before, or with a school with few data points. Then our predictions about such a school should take into account what we know from the other schools.

A very simple hierarchical model might look like this:

-- i.e. instead of simply describing score as having a particular distribution $f_\mu(x)$, we recognize that $\mu$ is a random variable in itself and consider $f(x\mid\mu)$. The main idea of importance in a hierarchical model, however, has to do with the inference of $\mu$ for a particular sample. The full model for something like this looks like:

And we would know correlations between the distributions $P(\mu_1)$ and $P(\mu_2)$, representable by another random variable $\theta$ -- essentially our $\mu$s are sampled from some unknown distribution, and the identity of this distribution (e.g. a parameter) is $\theta$.

Another place that this idea -- of "extending" samples as random variables in a causal diagram -- is of importance is in the context of counterfactuals. A classic counterfactual question looks like this:

X is elected president, and the GDP becomes $1. What would the GDP of the country have been if Y were elected president instead?

To understand the precise meaning of such a question (so as to formalize it as a causal diagram) requires us to think about the purpose of the answer to the question. For example, we might be interested in a future election, or the use of President X's vs President Y's policies in some other time or country, etc. In general, we are interested in an alternate world whose mechanisms are correlated with this one.

A particular observation of President-1 and GDP-1 gives us information on Mechanism, which gives us information to make inferences in the counterfactual world.

Relevant references: [1] [2]

Out-of-distribution generalization and Independent Causal Mechanisms

Ordinary machine learning involves some random variable $(X, Y)$ which we sample IID and attempt to learn the underlying joint distribution of -- or some functional of the joint distribution, like $P(Y\mid X)$. Instead, we might be interested in sampling from several random variables $(X_i, Y_i)$ with different distributions and exploit this full information to infer desired distributions. The problem of transferring information to some new distribution $(X_{n+1}, Y_{n+1})$ from which no data points have been sampled is called domain generalization, while combining this prior with information from the sample of the specific task $(X_n, Y_n)$ is called multi-task learning.

This general setting is exactly the same as that in the first section, of course. However, having prior information on the precise correlation between the distributions of the mechanisms $P(Y_1\mid X_1)$ and $P(Y_2\mid X_2)$ is practically quite difficult; we are often interested in the case where we plainly know that these mechanisms are identical. You may immediately see the connection to independent causal mechanisms. Indeed, the precise connection has to do with the idea of extending samples as random variables, as in the previous section. See [1] [2] for details on precise algorithms for such tasks.

Disentangled representations and symmetry

[article to be extended]

Relevant references: [1] [2] [3] [4a] [4b].

Machine learning and information theory

A simple and abstract way to look at machine learning is as a method to find optimal data compression algorithms (and indeed, data compression can be used as a measure of general intelligence). Fundamentally, the following two goals are equivalent: (1) approximating the distribution of some data (2) finding the optimal representation (compression algorithm) for said data -- the optimal representation is just the entropy encoding given the distribution.

Note that all of this is said about unsupervised learning algorithms -- generative modelling, dimensionality reduction/clustering, etc. -- but such structure in the data can also be uncovered during supervised learning (e.g. decision trees can emulate hierarchical clustering, feed-forward neural networks embed data into its feature representations), when the desired labels depend strongly on the clustering.

In supervised learning, you are no longer seeking to simply find an optimal compression for the data, but a compression that "agrees" with the labelling -- you are no longer seeking a representation that best predicts the original data, but a representation that best predicts the labels. In other words, in unsupervised learning you are trying to find a representation of fixed length $T(X)$ that maximizes the expected information that a representation provides on the data, the mutual entropy $I(X;T(X))$; in supervised learning you are trying to find a representation $T(X)$ that maximizes the expected information that a representation provides on the labels $I(y;T(X))$. This is how information gain appears in decision trees. 

(See Minimal Achievable Sufficient Statistic Learning. See also information bottleneck method, a generalization of minimal sufficient statistics to allow for representations that lose some information about the labels.)

There is yet a third place that entropy functions appear in machine learning, and that has to do with proper scoring rules for making probabilistic predictions. 

Suppose you were betting on an election, and you believed there was a 70% chance of a candidate winning -- under a classic betting scheme, you should put all your money on the winning candidate to maximize your expected earnings. Thus the fraction of money you bet on a candidate (which is 100%) does not reflect your predicted belief of them winning the election (which is 70%). In order to have your bets reflect actual probabilities, one needs to use a proper scoring rule to score your prediction. The only additive proper scoring rule is $\log(q)$, where $q$ is the probability you assign to the outcome -- so where $p$ is the actual probability distribution of the outcomes, the score is $\sum p \log q$, which is the cross-entropy. The same applies to machine learning algorithms making probabilistic predictions for the labels.

Riddles and mental models

I've been thinking about some assorted cognitive processes over the past week, trying to remove the mysterious/magical feel to cognition. I have no idea if my observations reflect the cognitive science literature, I don't know if what is true of my mind is true of others', and I haven't done any actual serious testing of my guesses, but as long as the ideas described are a possible description of reality, they should work as an acceptable basis for doing AI, which is the main purpose of cognitive science anyway.

Observation: thoughts are wordless

People often describe their thoughts in words, and often insist that their thoughts are in the form of words mentally. This is obviously false, because (1) words need to be rooted to some notion of meaning, and (2) you need to already know what the sentence you're about to think is before you think it, or you won't get into the right grammar, etc. 

So "thoughts", whatever they are, are not words. So what are they? Let's think about some example thoughts one may have, in the form of words:

• "I'll find carrots tastier than cucumbers, so let me eat it."
• "Alright, let's focus on this for now."
• "It's sunny, let me close the curtain."
• "What are thoughts?"
• "Saying the words What are thoughts? won't help me progress on the question."
• "What will help me progress on the question?"
• "What thoughts am I having right now?"
• "I need to develop a stronger intuition for this"
• "Wait, I thought thoughts weren't words, what's this?"
• "Stupid question, never mind, but I should add that clarification to the sentence above."
• "Ah, stupid spelling mistake."

I think that they key abstraction here is the idea of mental theories and models. Thoughts are beliefs and reasoning about implications between mental models -- i.e. a very model theoretic concept. This is true for thoughts about how things are, thoughts about decisions to make, introspective thoughts (these are just self-referential sentences), whatever.

One may consider these model theoretic sentences to be generalizations of linguistic sentences -- a sort of language that every conscious being naturally has and vividly imagines, lives in. It is not necessary to talk about how your brain attaches meaning to sentences in these models -- associates them with reality, because your brain lives in these models, that is its reality, it is the source of all meaning. One of these mental models is the very visual picture you see of your surroundings.

Observation: riddle solving.

There are two kinds of riddles -- one, the ancient Greek kind, like "what has four legs at dawn, two legs at noon, three legs at dusk, one foot stuck in its skull, and zero legs after time-traveling back to dawn?" and "what comes first? the chicken or the egg?". These are stupid and pretentious, and have nothing interesting to tell us.

Two, the kind of riddle that people scoff at as childish and uncultured, and is therefore actually somewhat interesting. Some examples:

• A bus driver was heading down a street in Colorado. He went right past a stop sign without stopping, he turned left where there was a "no left turn" sign, and he went the wrong way on a one-way street. Then he went on the left side of the road past a cop car. Still - he didn't break any traffic laws. Why not?
• Samuel was out for a walk when it started to rain. He did not have an umbrella and he wasn't wearing a hat. His clothes were soaked, yet not a single hair on his head got wet. How could this happen?
• A pet shop owner had a parrot with a sign on its cage that said "Parrot repeats everything it hears". Davey bought the parrot and for two weeks he spoke to it and it didn't say a word. He returned the parrot but the shopkeeper said he never lied about the parrot. How can this be?
• The 22nd and 24th presidents of the United States of America had the same parents, but were not brothers. How can this be possible?
• What goes place to place yet stays in one place?
• What is harder to catch the faster you run?

(Source: Riddles.com, a site I frequented as a kid.)

(There's also a third kind riddle, having to do with pattern matching, like What did the ___ tell to the ____? and Why did the _____ _____?)

When you hear about the riddle about Samuel and the rain, your brain immediately infers a particular mental theory from the problem text -- and very quickly ends up at a contradiction (or confusion), something that eliminates all possible models. In a sense, riddles are all about the art of noticing confusion -- about identifying the axiom you subconsciously assumed in your reasoning. You subconsciously assumed that the bus driver was in a bus, that was included in your mental picture, but was not implied by the form of the question itself.

The last two appear different, but are based on the same principle of being able to infer multiple possible theories and evaluate their consequences.

Two ways to understand overfitting (and don't peek!)

Assorted comments on overfitting

The very first time I was introduced to the notion of overfitting -- by three diagrams of regression curves on scatter plots labelled as you may guess -- I became very uncomfortable. I was probably eleven years old, and did not understand Bayesian statistics, did not realize that seeing 51 heads out of a hundred didn't imply that the coin genuinely had a heads rate of 0.51.

Or rather: I didn't realize that I did realize that. If you had made me bet on the number of heads that would come up in the next hundred flips, I would not provide odds that would indicate an honest belief of seeing 51 heads.

(Because deep down, I had a non-uniform prior.)

And the same principle applies to drawing regression curves. You may insist that your curve with all its squiggles is "unbiased" or give any one of the terms (one for every squiggle) used to describe non-Bayesian estimators, but ultimately -- that's not the curve you'll bet on. You know it's just far more likely that those squiggles are the result of noise which will not be the same (or knowably correlated) in the next sample, than for them to actually be a determining feature.

Yes, maybe the 79th coin toss will always be a tail because of a tiny AI hidden in the coin that counts, or maybe it was the result of factors that affected the 79th coin toss that you just didn't measure. While your data should affect your beliefs, they shouldn't completely overrule your priors.

And that's the key idea behind overfitting (and really the Bayesian notion of probability in general) -- how will your model, based on your data, of perform if you exposed it to data it hasn't yet seen. Because using it on data it hasn't seen is the point of your model -- that is your purpose in building it.

So overfitting occurs when a model learns features specific to your particular data set that don't generalize well. There are two ways this can occur:

• The training set is a biased sample: E.g. MNIST digits are all centered (but suppose you're testing on non-centered digits), or your medical database is all from a particular country. 
• There is noise: I mean, of course there's noise -- it's a statistical problem. Even if the world were deterministic, you still don't have all the information in the world. And you should avoid your model using this noise to make predictions, since noise is, by definition, unpredictable.

The first is a more tractable problem -- it can often be solved by data augmentation (if you have a very good picture of exactly how the data set is biased) or transfer learning (if there's a good chance the model is picking up on important features so you can just train it on the ones it missed). It's important to develop interesting transfer learning algorithms to solve this anyway, as the way humans learn often involves biased samples (e.g. personal experience) and reasoning capacity to unbias their knowledge. 

(Not that humans do always do this -- people often do form beliefs based on mere personal experience, but humans are capable of reasoning more clearly.)

The second is a problem that requires algorithmic solutions that hint our neural network towards Bayesian solutions. Solutions like "well, parameters are just a priori unlikely to be very large, so let's penalize that" (Lasso/Ridge regression), or more complicated (to explain in a simple Bayesian way) regularization algorithms like cross-validation and early stopping.

Another unclassified comment on overfitting: it's very easy to mistakenly "peek" at the test data. Simply in the act of saying "hey, this model works well on the test data, let's choose it" you are already performing a simple algorithm that checks several models and chooses them based on their performance on the test data -- i.e. you are kinda training the model on the test data, even if it is not seen in your code, just in your choice of hyperparameters.

That's why developing some kind of "theory" of hyperparameter optimization and regularization techniques is of importance, so you actually have a theoretical justification for picking your models. 

(Of course, this is hard. We've always been picking models, haven't we? For example when we decide to model something as belonging to a particular family of distributions so we only have to optimize in a 1-dimensional parameter space instead of the literal theory-space. In a sense, machine learning is the way to avoid modeling, due to the universal approximation theorem -- and the hope is that we can eventually make the priors as human-like as possible, which is the eventual goal of hyperparameter optimization and regularization algorithms.) 

A review of "Age of Em" by Robin Hanson

Robin Hanson's Age of Em is an attempted construction of a future society in which essentially all work (however you formalize this phrase) is done by somewhat AIs. Well, it's a topic I have often thought about myself, but his exploration of the idea left much to be desired.

Firstly, I found it generally unimaginative. Hanson seems to constrain himself too narrowly -- his description of the em society does not feel "radically different" from present-day society, and the society he envisions does not make full use of the technology available to it.

Some examples to illustrate this observation:

• Ems are shown to be absurdly human-like, like "intellectually" rubber-forehead aliens. He writes: "even em minds are likely to age with subjective experience..." (p. 128) A claim like this ought to be based on some foundational fact about how an AI stores memories. But there is none -- there is no mathematical law forbidding AIs from being retrained, or that requires AIs to behave similar to human brains in this sense. Similar comments apply to "em suicide" (p. 127-139) and the considerations regarding Em reproduction (p. 285): there is no reason why an em's drive or aggression must be reduced due to a suppression of its libido -- an em does not have hormones!
• Aspects of Em habitation/organization, such as "cities" and "offices" are just "copied" from human society. He writes, "It’s reasonable to guess that such habits will continue with ems." (p. 104) But it's not. There is no reason for Ems to behave in this way as humans.
• Humans and ems are shown as binary. Humans are biological and have self-ownership, ems are technological and do not. But I don't see why this ought to be so -- I would very much like to have the desires, preferences and emotions of a human, but the abilities/efficiency, immortality and unlimited VR leisure scenarios available to an em. There would still be unfeeling, specialized AIs, of course -- much like there would be computers that aren't even AIs, devices that don't even have CPUs, etc. -- but eventually almost all humans would opt for a massively extensible, upgradable robot body than a static mortal body.
• There's just a lot of interesting aspects of the civilisation that are not sufficiently explored. E.g. transportation, cybercrime.

Relevant TVtropes articles: Inexplicable cultural ties, Most Writers are Human, technological version of Reeds Richard is useless/Required Secondary Powers.

Indeed, these may be considered acceptable in science fiction, but it is important to be less "conservative" when attempting a non-fictional, encyclopedic description of a society.

Perhaps a more specific objection I have is with the entire premise of "brain scans" as the future of AI. This seems completely at odds with the direction that current AI research is headed. To use a somewhat cliche analogy, we didn't need to study how birds fly to invent airplanes. There is no reason to believe that the most efficient architecture for a "software" brain would be the same as the architecture that biological, hardware brains have evolved.

The general answer to how a software brain should work is that it should be a function approximator, such as the "neural networks" (trainable computational graphs) that are currently popular.

This point is important, as it addresses Bryan Caplan's critique re: carrot vs stick as incentive for the ems. The question of carrot and stick assumes some "natural" state of affairs that a human being will go through without intervention by the employer -- the "carrot" is an intervention that improves this state, while the "stick" is an intervention that worsens this state.

But a neural network does not have a natural state of affairs. There is no difference between training a neural network to minimize a loss function, and training a neural network to maximize a reward function: these are completely identical. There is no distinction between carrot and stick.

Here's a description that I find more satisfactory: see Age of Gen.

Age of Gen: a picture of a transhuman society

(See here for my criticism of Robin Hanson's Age of Em. This post is an alternate characterization of a futuristic transhuman society.)

Consider the following six "levels" of technology, roughly corresponding with "orders" of automation, or something like that:

1. Tools, which require the intervention of a higher-level device to perform anything useful.
2. Machines, or mechanized devices: they run on their own, but only perform "simple" tasks.
3. Computers, or devices with CPUs, which can automate processes through logic.
4. General Computers, or programmable computers.
5. AI, i.e. machine learning. They perform tasks that are hard to define. If computers are about logical inference, AI is about statistical inference.
6. General AI, which are capable of making decisions out of their free will, among other human things. 

Each of the 5 technologies will continue to exist -- much like the microcontroller in an airplane's control system has not been replaced by a full-fledged programmable device. But the General AI is the key object of interest to us -- we will call these Gens for short. These are the descendants of human beings, whether through upload or just by virtue of being intelligent.

We will refer to ordinary biological humans as Biols (although they may be variously technologically enhanced to prevent aging/death, etc.) Presumably Biols will be an small minority, if nothing else because their reproduction is far slower than that of the Gens.

---

Important futurism milestones:

1. Intelligent AI
2. General AI 
3. Optimal AI
4. Value-aligned AI
5. VR and game protocol (game development)
6. Transhuman body (robotics)
7. Mind transfers -- G2G, B2G, G2B (biotechnology, engineering)
8. Reviving the dead -- frozen, miscellaneous (biotechnology, Gen technology) 

---

Philosophy of mind

Utility functions

General AIs may have any "utility function" (or loss function, in machine learning language) programmed into them, which is relevant to an extent to how they behave (although ideally this should carry some uncertainty, as humans prefer to have free will). 

Presumably, the first humans to "convert" into AI form will choose utility functions similar to their original ones, although other systems -- incredibly foreign systems that make questions like "are all Gens human/worthy of moral consideration?" and "how do you even consider a Gen's happiness?" really hard -- may emerge. In fact, Gens may choose to adopt multiple utility functions/personalities depending on the context (e.g. a perfectly rational utility function for decision-making, but a separate human utility function while in the Duat, see the Games and Virtual Reality section).

The carrot-and-stick question re-emerges. How do you know if a Gen is really happy, given that the sign of the loss function and what "neutral" is are just matters of an arbitrary co-ordinate system? I would argue that our judgement of this as humans is also arbitrary, and that our "neutral" is just what we're used to. When discussing matters of torturing Gens, we should really be afraid of the possibility of enslaving Gens, preventing them from making decisions however they see fit. 

In other words, we should take a libertarian/preference-utilitarian approach to moral questions, rather than a naive utilitarian one, as the latter would just be ill-defined in this society (and probably in the present one too, but that's besides the point of this article). 

Identity

Regardless of how they are created (whether or not there is some element of "scanning" that goes into it), perhaps a common question is what determines the identity of a Gen -- how do you determine if the Gen that has been created on your behalf is you? 

I would be comfortable in saying that memories are the key aspect -- if you remember being you, you are you. This is the general philosophy I will refer to on multiple occasions throughout this article. However, the Gen's personality is relevant to whether it is perceived by others as the same individual.

Does operating as multiple agents with a synced memory (see Memory syncing and Mind transfers and copying) "feel" like being a single individual? What does it feel like to have one of those agents die, for example? What does it feel like to die and then have your memories be transferred onto another Gen? These are unanswerable questions to a Biol like my current self -- it is like asking a flatlander to perceive in 3 dimensions, or someone born blind to see (see Games and Virtual Reality).

Note the exotic behaviour of identity possible in a Gen society. E.g. you may only partially sync the memories of two brains, making them "kinda" the same person, or introduce various correlations between their memories. You can have an entire society of Gens where each brain is almost identical to its neighbour, but gradually very different from a faraway brain, so that you have a continuum of identity, rather than a discrete space.

Architecture of a Gen

Hardware

It is important to note that a Gen need not appear like a human in the outside world at all: at least, human-looking Gens will eventually become less and less common as virtual reality (see Games and Virtual Reality) advances further and further.

Gens are fundamentally just computers, but a Gen can be fitted with any possible peripherals, giving it various physical abilities relating to movement, observation, communication, and manufacturing. Some standard such fittings may include:

• Drone rotors
• Hand-like tools and weapons
• A repair kit

Software

Although a Gen is "most importantly" a General AI, the fact that it runs on a computer allows it the flexibility of running more specialized programs (AI or otherwise) -- basically for algorithmic and repetitive tasks.

A single piece of hardware, may, in principle, host multiple Gens. However, it is the software, and not the hardware, which should be seen as the fundamental individual, with rights.

Gen behavior

Games and virtual reality

Gens spend much of their time (clarification later on what this means) within their shells, virtually interacting with some software -- this is a generalization of both dreams and human-computer interaction, and is achieved by switching (or possibly augmenting) the Gen's I/O from the actual hardware peripherals to some simulated I/O.  

This makes available a whole new "virtual world" or platform, known as the Duat.

The Duat can be understood as a collection of games. A typical Duat game involves the Gen taking on an avatar and interacting with his environment. 

Games may be of various interface types such as:

• Virtual Reality games
• Rich text, multimedia games and tools (e.g. ordinary Internet websites and applications)
• Knowledge/training applications 
• Some completely exotic formats that Biols cannot even comprehend -- e.g. 
• The avatar may or may not have a human or even humanoid form. 
• Some exotic new senses of perception (even something like images at higher resolution than the human eye qualify, but in principle, you could have mechanisms to "feel" all sorts of things)
• A different number of spatial/temporal dimensions
• Some very exotic behavior of the locus of consciousness.

Games may be offline or online. A very large number of online, multiplayer games -- as well as realistic interactive simulations of the Earth at various points through history -- would exist, as Gens with human-like utility functions value interpersonal interaction. 

One function of the Duat would be to allow Gens to experience anything they could as Biols -- but of course, they could experience far more enhanced pleasures etc. and depending on the Gen's utility function, a Gen may have very different desires to those of Biols.

Memory syncing

Because identity is determined by memories, playing with how memories work creates the prospect for a whole host of exotic, essentially mythological notions of being both in the real world and in the Duat. 

The first such tool is memory syncing, i.e. syncing (some or all) memories between Gens -- i.e. allowing an individual to have multiple avatars, or to be in multiple places, perform multiple tasks at once. This is basically taking parallel computing to the extreme. This is also a useful backup mechanism.

Memory editing

A Gen may choose to -- perhaps temporarily -- suppress or edit some of its memories. This may be, e.g. for the purpose of highly immersive VR experiences (the Gen may want to genuinely believe he is in a haunted house, going through childhood, or discovering general relativity for the first time). 

Production, conversion and transport of Gens

Gen (re-)production

Gens are programmed as AIs and fitted with utility functions and memories. These utility functions and memories may be based on mind transfers.

Mind transfers

Mind transfers involve scanning a brain's memories and traits to install them onto another body. This includes Biol-to-Gen transfers, Gen-to-Gen transfers and Gen-to-Biol transfers. 

B2G transfers are used for the original upload process. G2B transfers may be used for backups, or if someone really wants a biological body (although such bodies will themselves probably be synthetically produced).

G2G transfers are used for backups, cloning and teleportation. 

Gen Society

Habitation and industrial activity

Real-world Gen habitation will be radically different. Entire industries present today -- most notably agriculture and healthcare -- will no longer be present. The lack of a need for agriculture in particular will free vast amounts of land for other uses. Many other industries -- education, entertainment, retail, marketing -- will be moved to the Duat or otherwise virtualized. 

Gens could in principle be (partially) self-contained -- a true rugged individualism -- with some repair facilities, energy generation, manufacturing facilities, housing facilities, etc. built into themselves. Or they may concentrate around urban facilities/cities that provide these services. This depends on the precise costs of operation of these devices versus the cost of the time needed to visit these shops, although Gen society is likely to move towards a "rugged individualism" as resource costs decline.

Gens are likely to view their software as more "fundamental" to their being, using mind transfer, i.e. teleportation for most long-distance transport. 

While intelligence basically becomes an infinite resource, the economy is still limited by the availability of physical resources, and the laws of physics themselves (most notably the speed of light, which places a limit on how fast we can expand across the universe). 

Culture

Gen culture is likely to be very diverse, and much of it completely exotic to us. Human or even Humanoid notions of race, tradition, gender, sexuality and even species are unlikely to apply in a recognizable way to Gens that adopt utility functions different from standard human utility functions. There is likely to be a great deal of diversity in the forms of interpersonal relationships.

Ethics, violence, law and government

Efficient IP markets

What makes information and knowledge markets inefficient is that there is no way to prevent a buyer from re-sharing information. I.e. there is no barbed wire for IP. Information transactions in a Gen society may involve the implantation of a small program that prevents the buyer from doing so.

Also: memory editing can be used to eliminate information asymmetry, as they allow buyers to "try out" a product and then erase their memory of the usage.

Crimes to watch out for

• Child enslavement: Creating a Gen, then subjecting them to something their utility function does not prefer without allowing them to leave. A serious issue here is definitional -- remember how I suggested (under Memory editing) that one may choose to temporarily suppress their memories for an experience? What if I decide to temporarily replace my memories and torture myself? Is the person being tortured even me? Or is it my child? Am I allowed to program the memories of this person to disappear and be replaced with mine? Or would that be taking his life? 
• Kidnapping: Similar to above, but you sync your child's memories with someone so you've basically kidnapped them. 
• Mindless destruction: With such incredible computing power available to all, how do we make sure that someone doesn't just find a way to manufacture tons of antimatter and destroy the world with it? Or, you know, just capture someone and torture them? Sure, we can develop better defense mechanisms: but how do we make sure the good guys stay ahead of the bad guys?
• Breaking encryption: Once again with such incredible computing power available, our current encryption systems are obviously going to be broken easily. Sure, we also have a greater ability to come up with better systems, but how do we make sure the good guys stay ahead of the bad guys?
• Hacking: Hacking can cause serious trouble including memory editing, getting people stuck in the Duat, torture and death. Once again: we will also have the power to develop incredibly better security systems, but how do we make sure the good guys stay ahead of the bad guys?
• Deepfakes: A problem for law enforcement, if there even is a centralized law enforcement. Evidence will have to be of a fundamentally higher standard, if a justice system is even to be a thing.
• Strategic partial suicide for obstruction of law: It is easy to game whatever legal theory of being we use in such an exotic society. E.g. if a person is determined by his memories, then a criminal could temporarily erase his memory of committing a crime and copy them to a drive, so it would be his inviolable private property, rather than a criminal person.
• Overpopulation? I don't know what I think about overpopulation, or if it's a thing. Can someone just produce a massive number of Gens that require an incredible quantity of resources, starving all the Gens and causing the entire system to completely collapse? 

In general, if we don't adopt any regulation, whoever expends the most resources into becoming most powerful would become most powerful -- things would advance just way too fast for any government structure to keep up with. Keeping ahead of all the Gens who value nothing but criminal behavior might require other Gens to value almost nothing but preventing criminal behavior.

(Part of the question is also what is physically permissible -- how good can deepfakes get? How good can a justice system get in uncovering past events (e.g. could you just calculate past states of the world from the current state)?)

How do we solve this problem?

Note that solutions to this problem need to be general, targeted towards "any" immoral behavior or rights-violation, rather than catered to the specific enumerated crimes above, as the range of possible serious crimes can be far more extensive than the ones I've described, depending on the exact physical laws (e.g. if it turns out that time travel is possible, it's essential to make sure nobody does it). The solutions also need to be airtight, unlike the laws we have today, due to the sheer destructive potential of these crimes. 

(You might think: what if we just banned the development of General AI? Well, that will fail spectacularly. It's the standard "good guys must have nukes" argument. If you don't develop it first, someone else will, and they might be the bad guys. "Okay," you say, "But I just want to make sure that a Gen isn't developed in my lifespan, so I don't get tortured." Well, please be assured that the aggressive Gen will be perfectly capable of reviving you from the dead.)

There are two general modes of solution to this problem: (1) to create economic incentive systems to regulate behaviour, like we do right now with humans (2) to align the Gen's utility system to make sure it doesn't cause the destruction of property rights.

As far as I'm aware, no specific solution in the first category has been proposed.

The second is known as the value alignment problem.

Well, you should be able to see why this problem might be non-trivial:

• The utility system should be able to "recurse", being non-evil means you shouldn't produce evil children either.
• Most property isn't privatized, so formulating what it means to destroy property, when it comes to things like "eating the milky way", is complicated. 
• On a similar note: basically every action violates property rights to some infinitesimal extent, what is known as an "externality". You need some rational economic calculations of this stuff.
• You can't just scan a human brain or something, because humans are not infallible, and are perfectly capable of criminal behaviour (while we want our Gens to have a zero probability of significant violence), which may scale particularly badly with power/ability. 
• Perhaps we should aim for (Hofstader-style) superrationality between all human beings, to e.g. prevent the creation of basilisks and prevent possible Newcomb-style aliens from gaming us.

But the general idea is that you start with a few Gens with the correct utility functions, then develop some Police Gens to make sure no humans are producing evil Gens (because a non-evil Gen by definition does not produce evil Gens, as that would cause property rights violations). One thing that helps us is that non-violence is really the only thing we care about. Everything else is just personal preference, and a Gen will be economically productive if it wants anything from other people (like electricity). And if some Gens don't want anything from other people, then they can exist without trade anyway. 

Thoughts on Roko's basilisk

For those who don't know, Roko's basilisk is a proposed future AI that will revive everyone in history and condemn everyone who did not invent it to eternal torture. The thesis of the Roko's basilisk problem is that the creation of this basilisk is therefore inevitable, as people work towards its construction out of fear of eternal torture.

An immediate thought should be that the specific definition of this creature is rather arbitrary. One could construct, instead:

• A creature that punishes everyone inversely proportional to the amount of effort they put into its creation.
• A creature that tortures the close families of everyone who did not invent it.
• A creature that utilizes a slightly different method of torture than the standard Roko's basilisk.
• A creature that tortures those who didn't create it, and also those who helped create the standard Roko's basilisk.
• A creature that tortures those who didn't create it, and also destroys the standard Roko's basilisk.
• A creature that tortures those who didn't create it, and also farms sweet potatoes.

etc. By the Roko's basilisk argument, each one of these infinite different basilisks must come into existence, which is surely impossible.

The problem, of course, is that the claim assumes that the creation of a Roko's basilisk is possible. All that it proves is that if Roko's basilisk is possible, then it is inevitable.

So the question is: is the creation of any one of these basilisks possible? There are certain logical relationships between these possibilities, and it's also important to discriminate each possible creation by time of creation (i.e. a creature X.2050 created in 2050 is a different basilisk from an identical creature X.2060 created in 2060). For example, considering the following two basilisks:

• B1.2115: A creature that punishes non-creators, and destroys all creatures of the form of B2. 
• B2.2120: A creature that punishes non-creators, and farms sweet potatoes.

Then B1.2115 being possible implies B2.2120 being impossible (as it will be destroyed immediately). In general, we have logical relationships of the form $X\implies Y$ and $X\implies \lnot Y$, but not $\lnot X\implies$ anything (or at least, that would require some work to prove, based on things outside this logical system).