An agent chooses actions based on their outcomes. Outcomes are whatever the agent has preferences over. If an agent does not prefer any outcome to any other outcome, it does not matter what the agent does. Initially, let’s consider outcomes without considering the associated actions. Assume there are only a finite number of outcomes.

Let’s define a preference relation over outcomes. Suppose $o_1$ and $o_2$ are outcomes. Outcome $o_1$ is weakly preferred to outcome $o_2$, written $o_1⪰o_2$, if outcome $o_1$ is at least as desirable as outcome $o_2$.

We write $o_1⋡o_2$ to mean the negation of $o_1⪰o_2$, that is, $o_1$ is not weakly preferred to outcome $o_2$.

Define $o_1\sim o_2$ to mean $o_1⪰o_2$ and $o_2⪰o_1$. That is, $o_1\sim o_2$ means outcomes $o_1$ and $o_2$ are equally preferred. In this case, we say that the agent is indifferent between $o_1$ and $o_2$.

Define $o_1\succ o_2$ to mean $o_1⪰o_2$ and $o_2⋡o_1$. That is, the agent weakly prefers outcome $o_1$ to outcome $o_2$, but does not weakly prefer $o_2$ to $o_1$, and is not indifferent between them. In this case, we say that outcome $o_1$ is strictly preferred to outcome $o_2$.

Typically, an agent does not know the outcome of its actions. A lottery is defined to be a finite distribution over outcomes, written

where each $o_i$ is an outcome and each $p_i$ is a non-negative real number such that ${\sum }_i{p}_i=1$. The lottery specifies that outcome $o_i$ occurs with probability $p_i$. In all that follows, assume that outcomes may include lotteries. This includes lotteries where the outcomes are also lotteries (called lotteries over lotteries).

The rationale for this axiom is that an agent must act; if the actions available to it have outcomes $o_1$ and $o_2$ then, by acting, it is explicitly or implicitly preferring one outcome over the other.

To see why this is reasonable, suppose it is false, in which case $o_1⪰o_2$ and $o_2\succ o_3$ and $o_3⪰o_1$. Because $o_2$ is strictly preferred to $o_3$, the agent should be prepared to pay some amount to get from $o_3$ to $o_2$. Suppose the agent has outcome $o_2$; then $o_1$ is at least as good so the agent would just as soon have $o_1$. $o_3$ is at least as good as $o_1$, so the agent would just as soon have $o_3$ as $o_1$. Once the agent has $o_3$, it is again prepared to pay to get to $o_2$. It has gone through a cycle of preferences and paid money to end up where it is. This cycle that involves paying money to go through it is known as a money pump because, by going through the loop enough times, the amount of money that an agent must pay can exceed any finite amount. It seems reasonable to claim that being prepared to pay money to cycle through a set of outcomes is irrational; hence, a rational agent should have transitive preferences.

It follows from the transitivity and completeness axioms that transitivity holds for mixes of $\succ$ and $⪰$, so that $\text{if }{o}_1⪰o_2\text{ and }{o}_2⪰o_3$, then $o_1⪰o_3$ and if one or both of the preferences in the premise of the transitivity axiom is strict, then the conclusion is strict. Thus, $\text{if }{o}_1\succ o_2\text{ and }{o}_2⪰o_3$, then $o_1\succ o_3$. See Exercise 12.1.

Note that, in this axiom, $\succ$ between outcomes represents the agent’s preference, whereas $>$ between $p$ and $q$ represents the familiar comparison between numbers.

The following axiom specifies that lotteries over lotteries only depend on the outcomes and probabilities.

This axiom specifies that it is only the outcomes and their probabilities that define a lottery. If an agent had a preference for gambling, that would be part of the outcome space.

These four axioms imply some structure on the preference between outcomes and lotteries. Suppose that $o_1\succ o_2$ and $o_2\succ o_3$. Consider whether the agent would prefer

• •

  $o_2$ or

• •

  the lottery $[p:o_1,(1-p):o_3]$

for different values of $p\in [0,1]$. When $p=1$, the agent prefers the lottery (because, by decomposability, the lottery is equivalent to $o_1$ and $o_1\succ o_2$). When $p=0$, the agent prefers $o_2$ (because the lottery is equivalent to $o_3$ and $o_2\succ o_3$). At some stage, as $p$ is varied, the agent’s preferences flip between preferring $o_2$ and preferring the lottery.

Figure 12.1 shows how the preferences must flip as $p$ is varied. On the $X$-axis is $p$ and the $Y$-axis shows which of $o_2$ or the lottery is preferred. The following proposition formalizes this intuition.

The preceding proposition does not specify what the preference of the agent is at the point $p_2$. The following axiom specifies that the agent is indifferent at this point.

The next axiom specifies that replacing an outcome in a lottery with an outcome that is not worse, cannot make the lottery worse.

A direct corollary of this is that outcomes to which the agent is indifferent can be substituted for one another, without changing the preferences.

This follows because $o_1\sim o_2$ is equivalent to $o_1⪰o_2$ and $o_2⪰o_1$, and we can use substitutability for both cases.

An agent is defined to be rational if it obeys the completeness, transitivity, monotonicity, decomposability, continuity, and substitutability axioms.

It is up to you to determine if this technical definition of rationality matches your intuitive notion of rationality. In the rest of this section, we show more consequences of this definition.

Although preferences may seem to be complicated, the following theorem shows that a rational agent’s value for an outcome can be measured by a real number. Those value measurements can be combined with probabilities so that preferences with uncertainty can be compared using expectation. This is surprising for two reasons:

• •

  It may seem that preferences are too multifaceted to be modeled by a single number. For example, although one may try to measure preferences in terms of dollars, not everything is for sale or easily converted into dollars and cents.

• •

  One might not expect that values could be combined with probabilities. An agent that is indifferent between the money $\$(px+(1-p)y)$ and the lottery $[p:\$x,(1-p)\$y]$ for all monetary values $x$ and $y$ and for all $p\in [0,1]$ is known as an expected monetary value (EMV) agent. Most people are not EMV agents, because they have, for example, a strict preference between $1,000,000 and the lottery $[0.5:\$0,0.5:\$2,000,000]$. (Think about whether you would prefer a million dollars or a coin toss where you would get nothing if the coin lands heads or two million if the coin lands tails.) Money cannot be simply combined with probabilities, so it may be surprising that there is a value that can be.

Thus, an ordinal preference that follows the axioms is a cardinal preference where utility defines the values to be compared.

In this proof the utilities are all in the range $[0,1]$, but any linear scaling gives the same result. Sometimes $[0,100]$ is a good scale to distinguish it from probabilities, and sometimes negative numbers are useful to use when the outcomes have costs. In general, a program should accept any scale that is intuitive to the user.

A linear relationship does not usually exist between money and utility, even when the outcomes have a monetary value. People often are risk averse when it comes to money: they would rather have $\$n$ in their hand than some randomized setup where they expect to receive $\$n$ but could possibly receive more or less.

Rationality does not impose any conditions on what the utility function looks like.

Utility for an agent is a function of outcomes or states. Representing utilities in terms of features or variables typically results in more compact representations that are easier to reason with and more natural to acquire.

Suppose each outcome can be described in terms of features $X_1,\mathrm{\dots },X_n$. An additive utility is one that can be decomposed into a sum of terms:

Such a decomposition is making the assumption of additive independence.

When this can be done, it greatly simplifies preference elicitation – the problem of acquiring preferences from the user. This decomposition is not unique, because adding a constant to one of the terms and subtracting it from another gives the same utility. A canonical representation for additive utility has a unique decomposition. Canonical forms are easier to acquire as each number can be acquired without considering the other numbers. To put additive utility into canonical form, for each feature $X_i$, define a local utility function $u_i(X_i)$ that has a value of 0 for the value of $X_i$ in the worst outcome and 1 for the value of $X_i$ in the best outcome, and a non-negative real weight, $w_i$. The $w_i$ weights should sum to 1. The utility as a function of the variables is

To elicit such a utility function requires eliciting each local utility function and assessing the weights. Each feature, if it is relevant, must have a best value for an agent and a worst value for the agent. Assessing the local functions and weights can be done as follows. Consider just $X_1$; the other features then can be treated analogously. For feature $X_1$, values $x_1$ and $x_1^{\prime }$ for $X_1$, and fixed values $x_2,\mathrm{\dots },x_n$ for $X_2,\mathrm{\dots },X_n$:

The weight $w_1$ can be derived when $x_1$ is the best outcome and $x_1^{\prime }$ is the worst outcome (because then $u_1(x_1)-u_1(x_1^{\prime })=1$). The values of $u_1$ for the other values in the domain of $X_1$ can be computed using Equation 12.1, making $x_1^{\prime }$ the worst outcome (as then $u_1(x_1^{\prime })=0$).

Assuming additive independence entails making a strong independence assumption. In particular, in Equation 12.1, the difference in utilities must be the same for all values $x_2,\mathrm{\dots },x_n$ for $X_2,\mathrm{\dots },X_n$.

Additive independence is often not a good assumption. Consider binary features $X$ and $Y$, with domains $\{x_0,x_1\}$ and $\{y_0,y_1\}$.

• •

  Two values of $X$ and $Y$ are complements if having both is better than the sum of having the two separately. More formally, values $x_1$ and $y_1$ are complements if getting one when the agent has the other is more valuable than when the agent does not have the other:

  $$u(x_1,y_1)-u(x_0,y_1)>u(x_1,y_0)-u(x_0,y_0).$$

• •

  Two values are substitutes if having both is not worth as much as the sum of having each one. More formally, values $x_1$ and $y_1$ are substitutes if getting one when the agent has the other is less valuable than getting one when the agent does not have the other:

  $$u(x_1,y_0)-u(x_0,y_0)>u(x_1,y_1)-u(x_0,y_1).$$

Additive utility assumes there are no substitutes or complements. When there is interaction, we require a more sophisticated model, such as a generalized additive independence model, which represents utility as a sum of terms, where each term can be a factor over multiple variables. Elicitation of the generalized additive independence model is much more involved than eliciting an additive model, because a variable can appear in many factors.

For Boolean features, it is possible to represent arbitrary utility using the analogy to the canonical representation for probability. This extends the additive utility to allow weights for the conjunctions of atoms (which corresponds to the product when true is 1 and false is 0), including a bias term. Complements have a positive weight for the conjunction, and substitutes have a negative weight for the conjunction.

It is common to start with representing just the simpler interactions, such as only representing the weights of single atoms and some pairwise products, and only introducing products of more atoms when necessary. If all conjunctions were included, there would be $2^n$ weights for $n$ propositions, which is the same as a table of all combinations of truth values. The canonical representation is useful when many of the weights are zero, and so don’t need to be represented at all.

Utility theory is a normative theory of rational agents that is justified by a set of axioms. Prospect theory is a descriptive theory of people that seeks to describe how humans make decisions. A descriptive theory is evaluated by making observations of human behavior and by carrying out controlled psychology experiments.

Rather than having preferences over outcomes, prospect theory considers the context of the preferences. The idea that humans do not perceive absolute values, but values in context, is well established in psychology. Consider the Müller-Lyer illusion shown in Figure 12.4.

The horizontal lines are of equal length, but in the context of the other lines, they appear to be different. As another example, if you have one hand in cold water and one in hot water, and then put both into warm water, the warm water will feel very different to each hand. People’s preferences also depend on context. Prospect theory is based on the observation that it is not the outcomes that people have preferences over; what matters is how much the choice differs from the current situation.

The relationship between money and value that is predicted by prospect theory is shown in Figure 12.5.

Rather than having the absolute wealth on the $x$-axis, this graph shows the difference from the current wealth. The origin of the $x$-axis is the current state of the person’s wealth. This position is called the reference point. Prospect theory predicts:

• •

  For gains, people are risk averse. This can be seen as the curve above the current wealth is concave.

• •

  For losses, people are risk seeking. This can be seen as the curve below the current wealth is convex.

• •

  Losses are approximately twice as bad as gains. The slope for losses is steeper than that for gains.

It is not just money that has such a relationship, but anything that has value. Prospect theory makes different predictions about how humans will act than does utility theory, as in the following examples from Kahneman [2011, pp. 275, 291].

Empirical evidence supports the hypothesis that prospect theory is better than utility theory in predicting human decisions. However, just because it better matches a human’s choices does not mean it is the best for an artificial agent. An artificial agent that must interact with humans should, however, take into account how humans reason. For the rest of this chapter we assume utility theory as the basis for an artificial agent’s decision making and planning.