1.3 Introduction to our model
1.3.2 Learning strategies
In the following, we will consider in more detail the learning strategies. Social learning will take the form of conformism. Individual learning will take the form of reinforcement learning with exponential discounting. Bayesian learning would suggest itself as an alternative individual learning strategy.
We will briefly discuss Bayesian learning and explain why it is not very adequate for the learning task.
1.3.2.1 Individual learning trough reinforcement
In our model, individual learning is modeled in a more realistic fashion than previously. This is possible because an individual faces more than just one decision per lifetime and can therefore rely on past experience to make in-formed decisions.
We chose to implement individual learning as reinforcement learning [97].
This is a rather broad term that includes several possible implementations, so we have to go into more details. The most simplistic implementation of reinforcement learning would be a strategy that sticks with the same option as last period when it was successful and switches options when unsuccessful – this strategy is called win-stay lose-shift [133, 145].
Figure 1.7: Overview of the model. Strategies are characterized by a decision mechanism. This mechanism results in a certain behaviorthat, for social lear-ners, is dependent on how others in the population behave. Furthermore, the beha-vior of the strategies depends on theenvironment, which varies over time. Com-bining the behavioral responses of the strategies with the environment results in cultural adaptation. Social learners sample randomly from the whole population.
Depending on the choices, strategies accumulatefitness. Strategies with higher fit-ness tend to increase in frequency, resulting in genetic adaptation. Changes in the strategies’ frequency feeds back on cultural adaptation.
Figure 1.8: The behavior of win-stay lose-shift over time. The proportion of A choices made by this strategy (•) is measured on the left y-axis. The solid line represents the environment in the form of pA−pB, measured on the right y-axis.
The horizontal line serves as a guide to the eye. A perfectly performing strategy would always choose A as long aspA−pB>0, that is, when thepA−pBline is below the horizontal line. It would never choose A in the opposite case of pA−pB <0.
Win-stay lose-shift tracks changes in the environment but is very “conservative”
because it never deviates very far from 50% A choice. This behavior hampers the performance of this strategy.
An illustration of the behavior of win-stay lose-shift can be found in fi-gure 1.8. Shown are the aggregate behavior of 1000 individuals who use this strategy and at the same time how the environment develops over time.
Behavior is shown as the average proportion of A choices made be the in-dividuals, measured on the left y-axis. The environment consists of pA and pB, but to simplify matters, we only plotted the difference pA−pB, which is measured on the right y-axis.
To understand why win-stay lose-shift is not a good performer, we first have to understand how a perfect strategy would behave. Obviously, the perfect strategy should always choose A whenpA> pBand B whenpA< pB
(if pA = pB, the choice is irrelevant). When we look, e.g., at period 200, we find that pA is larger than pB but still only approximately 60% of the population choose A. Win-stay lose-shift is very conservative in that it will stick close to 50% most of the time, at the detriment of its performance.
Still, win-stay lose-shift is capable of tracking changes in the environment reasonably well, resulting in a performance significantly better than expected from chance. Performance is defined here as the degree to which a strategy approaches optimal behavior. If a strategy has a performance of 60%, that could mean, e.g., that in each period, 60% of the individuals using this strategy choose the better option. But it could also mean that during 60%
of the periods, all individuals choose the better option and during 40% of the periods, none of them chooses the better option. In reality, we will always
find a mixture of these extremes. The performance of random choice would be 50%, performance of win-stay lose-shift is 54.40% (0.06% standard error of the mean), which is significantly higher.
Essentially, win-stay lose-shift is a reinforcement learning strategy with a memory of only one period length. It cannot take into consideration what happened two, three, or more periods in the past. However, the environment tends to vary slowly. Say that A has led to success in the past 10 periods except in the very last period. Win-stay lose-shift would immediately switch to B after this turn of events, as it ignores everything except the last ob-servation. However, such a string of successes makes it very likely that pA is high and should not prompt a premature switch. This is where memory comes into play.
We model reinforcement learning with memory by assuming that individ-uals have a propensity, Pi for each option i, i = {A,B}. The propensities are updated in each period by increasing them by an amount Ri if option iis reinforced, with Ri being equal to 1 if option iwas chosen and yielded success or -1 if it was chosen and did not yield success; it is 0 if the op-tion was not chosen in this period. Moreover, the propensity for an opop-tion depends on the discounted propensity of the last period, with the discount factor denoted as q. In sum, the propensity for optioniin period tis given as:
Pi(t) =q·Pi(t−1) +Ri(t−1)
The probability Pri(t) of choosing option iin periodt is then given as:
Pri(t) = exp (Pi(t)) P
jexp (Pj(t))
We only face two options, A and B. Therefore, we have:
PrA(t) = 1−PrB(t)
= exp PA(t)
exp PA(t)+ exp PB(t) (1.1)
= 1
1 + exp PB(t)−PA(t)
= 1
1 + exp (−∆P(t)) (1.2)
with
∆P(t)≡PA(t)−PB(t)
Moreover, for ∆P(t), we have:
∆P(t) = PA(t)−PB(t)
= q·PA(t−1) +RA(t−1)−q·PB(t−1)−RB(t−1)
= q· PA(t−1)−PB(t−1)+RA(t−1)−RB(t−1)
= q·∆P(t−1) +RA(t−1)−RB(t−1) (1.3) When choosing A in the last period led to a success, RA(t−1) = 1, while a failure with A results in RA(t−1) =−1; the same is true for B. On the other hand, the option that was not chosen is not reinforced. Therefore, RA(t−1)−RB(t−1) equals 1 if A was chosen and yielded success or if B was chosen and did not yield success. In contrast, RA(t−1)−RB(t−1) equals -1 if A was chosen and did not yield success or if B was chosen and did yield success. We can thus defineR(t−1) as:
R(t−1)≡
(1 A successful or B unsuccessful int−1
−1 B successful or A unsuccessful int−1 Inserting this in equation 1.3, we get:
∆P(t) =q·∆P(t−1) +R(t−1) (1.4) To calculate the probabilities, we thus insert equation 1.4 in 1.2:
PrA(t) = 1
1 + exp (−∆P(t))
= 1
1 + exp (−q·∆P(t−1)−R(t−1))
From these equations, it is clear that for the first period of the first gener-ation, a probability to choose A or B cannot be defined. This is why for all strategies, the very first choice is defined to be random. For the first period of subsequent generations, offspring use as first choice the last choice of its parent. This is the only form of vertical transmission in our model and it is also applied to all other strategies. Individual learners do not inherit the propensities of their parents, though, instead starting fresh with propensities of 0.
Often, an additional modification to reinforcement learning is made by introducing a sensitivity factor λ (see e.g. [104]). This parameter alters the steepness of the probability of choosing an option as a function of the difference in propensities, so that:
PrA(t) = 1
1 + exp (−λ·∆P(t))
Low λ imply that the probabilities to choose A or B are rather insensi-tive to changes in propensity. The higher λ, the steeper the adjustment of the probability – even small differences in propensities would lead to large differences in the probabilities to choose A and B. In the extreme case of λ=∞, this results in a simple threshold function that prescribes to choose A if propensity for A exceeds propensity for B and vice versa:
PrA(t) =
We found that this simple threshold function leads to a performance that is almost the best that an individual learner can achieve. For this reason, and for reasons of simplicity and parsimony, we adopt this threshold function and do not further analyzeλ.
The only free parameter that is left is thusq, the discount factor applied to previous propensities. For q ≤ 0.5, propensities could never exceed 1 in absolute values in finite time, so that the reinforcementR would always exceed previous propensities. In other words, for q < 0.5, previous experi-ence counts so little that it never affects the current decision. If previous experience counts that little, we effectively deal with win-stay lose-shift.
When q > 0.5, however, experience past the very last period may well affect the current choice, and the more so the higherq. We show an example of how individual learners with q = 0.9 behave in a given environment in figure 1.9. Individual learners withq= 0.9 too track the environment quite well but behave less “conservatively” than win-stay lose-shift (the right panel of the figure is a reproduction of fig. 1.8). For example, at around period 200,pAis clearly greater thanpB, yet only approximately 60% of individuals using win-stay lose-shift pick A. In contrast, approximately 80% of individual learners withq = 0.9 pick A. On the flip-side, individual learners sometimes fall a little behind, e.g. during the changes between period 50 and 100. We will later analyze how performance of individual learners depends onq.
1.3.2.2 Bayesian individual learning
A natural alternative to individual learning based on exponential discounting reinforcement learning would be a Bayesian learning algorithm. A Bayesian learner has a prior belief aboutpAand pB and updates it after each realiza-tion of an outcome. For example, assume that in periodt, the learner has a prior belief thatpA equals 0.6 of 0.7, Pr (pA= 0.6) = 0.7, and consequently chooses A. However, the learner is not successful with A. Then her posterior belief about A is:
Figure 1.9: The behavior of individual learners with q = 0.9 (left panel) and of win-stay lose-shift (right panel) over time. The proportion of A choices made by the strategies (•) is measured on the left y-axis. The line represents the environment in the form of pA−pB, measured on the right y-axis. Individual learners with q = 0.9 track changes in the environment quite well and are less “conservative”
than win-stay lose-shift but lag a little behind.
Pr (pA= 0.6|A fails) = Pr (A fails|pA= 0.6)·Pr (pA= 0.6)/Pr (A fails) The probability of A not succeeding when pA = 0.6 is equal to 0.4. As-suming that Pr (A fails) = 0.5, we have:
Pr (pA= 0.6|A fails) = 0.4·0.7/0.5
= 0.56
Using this approach, it is possible to design a Bayesian individual learner.
The problem that occurs is that a Bayesian learner has to start with some kind of prior. We will be very generous: We supply the Bayesian learner the full probability distribution of pA and pB, as determined by simulating those values for a length of 107 periods (see figure 1.10). In reality, no learner would live long enough to be able to draw on such extensive data, so this should give a Bayesian learner a huge advantage. Even more, other learners in our model do not even know that there is a constant probability distribution ofpAand pB, but as we said, we wanted to be generous towards Bayesian learners.
In each period, the Bayesian learner chooses an option, then this option is realized, resulting in either success or failure. After realization, the Bayesian learner updates her beliefs about the probability distribution ofpAif A was chosen or pB if B was chosen. The belief about the option that was not chosen is not updated. In the next period, the Bayesian learner chooses the
Figure 1.10:Probability distribution of pA and pB, the environment, for the de-fault parameters.
option that, according to her beliefs, has the higher expected value and the cycle continues.
We made three further adjustments to Bayesian learners. As after each period, pA and pB change their values by one incremental step, they take distinct sets of values depending on whether the period is currently even or odd. For example, in odd periods, pA can take the value 0.5 or 0.54 but never 0.52, whereas in even periods, the opposite is true. We allowed Bayesian learners to take this into account.
Another adjustment is to allow Bayesian learners to make inferences from their first period’s choice, which is the same as their parent’s last period’s choice. If the parent chose A in the last period, it is more likely than not that pA> pB. We simulated the final priors ofpAandpB conditional on whether the final choice was A or B and allowed Bayesian learners to use these biased priors as their initial priors, depending in whether they inherited A or B as their first choice.
Moreover, it is possible to give Bayesian learners the ability to predict the next step of the environment when they are given knowledge about how the environment changes from period to period. For example if they assign a probability to the event that currentlypA(t) = 0.5 and a probability to the event that currently pA(t) = 0.54, they can use these probabilities to calcu-late the probability ofpA(t+1) = 0.52 when they know how the environment changes. This way, Bayesian learners not only have huge statistical knowl-edge about the environment, as well as the possibility to bias their priors according to their parent’s last choice, but also a complete understanding of the process that generates the environment, a further advantage that no other strategy has.
The remaining conditions for Bayesian learners are the same as for indi-vidual learners. The very first choice is determined by chance. After each generation, offspring inherit their first choice from their parent. However,
they do not inherit the beliefs of their parents but instead again start with the prior as determined initially. The situation is similar to the situation of reinforcement learners who also do not inherit the propensities of their parent.
1.3.2.3 Social learning through conformism
In this chapter, we do not want to present any breathtaking new insights derived from our social learning model but instead establish its validity.
Therefore, before we begin to test new social learning strategies, we con-tend with studying a social learning strategy that is well explored already, conformism.
Conformism has always been an important part of the theory of social learning [21, 22, 86]. There is no single definition of conformism that is used by all researchers across disciplines, or even within disciplines. Boyd and Richerson write that “[Conformist] Individuals are assumed to be dis-proportionately likely to imitate the more common behavioral types among their cultural parents” [21], which is essentially how conformism (or “hyper-conformity”, as some would call it [34]) is used in this work.
“Disproportionally” in the quotation above means that the probability of adopting the most frequent option is higher than this option’s share in the population’s choice. For example, say that option A is chosen by 60% of the population and option B by 40%. If social learning consisted of random choice, the probability to adopt A would also be 60%, not more or less than the option’s frequency in the population. This would therefore not qualify as conformism. If a social learning strategy had an adoption probability of morethan 60%, even if only of 61%, it could be conformism.8 Similarly, since option B is chosen by only 40% of the population, it’s adoption probability has to be less than 40% for a strategy to qualify as conformist.
For our purposes, we define conformism as a strategy that samples a fixed number of individuals and then chooses the option that the majority of the sampled individuals chose. For example, if three individuals are sampled and two or three of them chose A, the conformist will also choose A. We could implement a sensitivity parameterλ. This parameter would smooth out the step function-like mechanism we designed. When two of three individuals choose A, we could, e.g., say that the conformist also chooses A with 80%
probability instead of 100%. But as for reinforcement learning, we found such a parameter to have little positive effect, and often even negative effects, so it is more parsimonious to drop it completely. This implementation of conformism thus corresponds to the example we discussed earlier in this chapter.
8A strategy with a less than 60% adoption probability would be called “anti-conformism”
or a “maverick” [46].