Adverse effects of higher sample sizes

In several instances, we found that increasing the sample size does not im-prove performance but instead deteriorates it. This finding may be surpris-ing at first, since taksurpris-ing into account more information should not lead to a worse result. There are, however, theoretical reasons why sometimes ig-norance is a bliss. For example, knowing too much may prevent the usage of the recognition heuristic. American students were better at estimating which of two German cities has the larger population than which of two

American cities has; similarly German students were better with American cities than with German cities [73, 74].⁷

Other reasons have been found for why knowing less can be advanta-geous. For example, sampling only a few times can lead the payoff difference between the studied variables to appear larger than it actually is [93]. Cor-relations between variables may appear larger and thus more distinct if the sample size is smaller [102]. These findings make sense under the assumption that an individual needs to cross a certain threshold before taking action;

sampling less may result in the threshold being crossed more easily. The findings do not explain why the individuals do not simply lower the required threshold instead of sampling less. This makes it dubious whether less sam-pling is really adaptive in these cases and not rather a constraint.

This section deals with explaining why in the context of social learning, we should not always expect that as much information as possible is acquired, even if information acquisition is costless.

2.4.3.1 PBSL with weights [1/0]

First, we found that PBSL that only values successes (i.e. with weight pairs [1/0], [2/0], etc.) perform worse when they sample too many individuals.

Given standard conditions and for a sample size of three, this strategy at-tained a performance of 71.7%; for a sample size of 7, performance dropped to only 60.8%. Other scoring-type PBSL strategies, such as those with weights [4/−1], did not suffer this decline.

To better understand why we see these opposite effects of higher sample size, we had a closer look at the behavior of the different strategies. As an example, we assumed thatpA= 0.55 and pB = 0.45. As A is slightly better than B, one would expect that in the best case, a strategy would converge to-wards only choosing A. PBSL with weights [4/−1] and sample size 3 indeed converges towards predominantly choosing A, but only with a probability of 65%; approximately one third of the population would incorrectly choose B (see left panel of figure 2.21). With a sample size of 7, we see instead that 94% of the population will eventually choose the better option A. For PBSL with weights [4/−1] and similar, it becomes easier to pick the better option when they observe a greater sample.

We applied the same conditions to PBSL with weights [1/0]. For a sample size of 3, 85% would converge towards choosing the better of the two options (right panel of figure 2.21), explaining why they perform so well for this sample size. For a sample size of 7, it is possible that 99.8% of the population converges towards the better option, which is even better. There is, however,

7The recognition heuristic works this way: If one of the compared cities is known by name but not the other, one should guess that the known city is larger, as this is typically true. In the best case, one knows half of the cities; knowing too many cities would prevent the subjects from using the heuristic.

also a strong conformist bias. If 66% or more of the population choose the worse option, the conformist bias would lure more individuals to choose the worse option and in the end, 98.8% of the population would do so. PBSL with weights [1/0] and sample size 7 are therefore often very decidedly on the right side but can also very decidedly end up on the wrong side. The conformist bias is too strong for a sample size of 7 and therefore, this high sample size is an impediment for high performance.

2.4.3.2 PBSL McElreath

We found that for PBSL McElreath, too, a sample size of 3 results in higher performance than higher sample sizes in most conditions. Here we explain why this is the case. PBSL McElreath works by averaging the observed payoffs of the choice options and choosing the option with the highest average payoff. This appears to be a reasonable heuristic but this heuristic may also fail in some conditions. Assume for example that the social learning strategy observes 100 individuals that chose A and 1 individual that chose B. Furthermore, assume that the one B choice led to a success and that of the 100 A choices, 99 were successful and 1 unsuccessful. It would now be very sound to deduce from this observation that A must have a very high chance of success, while we know almost nothing about B. Knowing nothing about B, we should conclude that the very successful option A is the better choice and thus confidently choose it over B. Yet, if we take the average payoff of A (assuming a success gives 1 and a failure 0), it will be 0.99, while the average payoff of B is 1 and therefore higher. PBSL McElreath would choose B, the option that is almost certainly worse.

A similar argument as above can be made for very unsuccessful options.

If A is observed 100 times, of which 99 were unsuccessful and one successful, and if B is observed once and was unsuccessful, then B is in almost all situations the better option. Still PBSL McElreath would calculate the average payoffs, 0.01 in case of A and 0 in case of B, and consequently choose A. This example is of less importance, though, since it is a priori less likely that the worse option is chosen overwhelmingly often in the population.

These examples showcase why the strict comparison between average pay-offs can sometimes go awry, especially if the observed sample is large. Even in less extreme examples, if A is successful 5 times and unsuccessful once, while B is successful once, A is arguably the better choice but PBSL McEl-reath would choose B. Indeed, given this setup, if we assume that the better option has a success probability of 60% and the worse option of 40%, then A is the better choice in 10 out of 13 cases. The success probability of the better option would have to exceed 98% in order for B being more likely to be the better choice.⁸

8This is because if the better option has a very very high success probability, it becomes unlikely to observe even one failure in a sample of 6.

The approach of simply comparing means is flawed for high sample sizes.

When many individuals choose the same option, because of the described effect, it becomes less likely that an additional individual will follow their lead. PBSL McElreath will display anti-conformist behavior when many individuals are sampled (see left panel of figure 2.22). This effect occurs even though McElreath et al. [120] included a bias towards choosing the more frequent option. This bias is, however, only applied in fringe cases – only if there is a tie in average payoffs – and thus cannot correct for these suboptimal decisions. Now it is also clear why observing less leads to better results for PBSL McElreath. If the sample consists of only three or four observations, the described error cannot occur.⁹

Baldini showed more formally that anti-conformism is expected for higher sample sizes when averaging-type PBSL is used [8]. He therefore called into question the usefulness of averaging-type PBSL as a social learning strategy in general. However, as we showed, the problem can be circumvented by using small sample sizes. In effect, performance was even quite high despite small sample sizes. There is no reason to believe that evolution would not settle for smaller samples, as sampling is not exogenously fixed but presum-ably subject to natural selection.

We argued that if one observes 5 successes with A and one failure with A, while observing one success with B, it is in most circumstances better to choose A. To implement this change, we modified PBSL McElreath so that if the average payoffs of the observed options are close but not necessarily equal, the more frequent of the two options will be chosen. In theory, this payoff-conformism trade-off should cure the described weakness of PBSL McElreath with high sample sizes. We found this change to remedy the anti-conformist behavior observed for higher sample sizes (see right panel of figure 2.22), resulting in an improvement in performance. For example, under default conditions and for a sample size of 6, performance increased from 61.6% (without trade-off) to 78.4% (with trade-off).

In summary, we thus find that if an option is chosen very frequently in the population, this can actually lead to that option becoming less likely to be chosen by PBSL McElreath when they sample many individuals. There-fore, PBSL McElreath performs better when only sampling 3 individuals.

The adverse effect can, however, be amended by introducing a true payoff-conformism trade-off (instead of a tie-breaking rule). As a consequence, we find the performance at higher sample sizes to dramatically improve, exceed-ing performance at sample size 3.

9If one observes A being successful twice and unsuccessful once, and B being successful once, then choosing B is not obviously wrong.