Two-Armed Bandit Task

The two-armed bandit task is a classic reinforcement learning (RL) paradigm in which a subject repeatedly chooses between two options (“bandits”), each delivering a reward according to an unknown probability distribution. The subject learns by trial and error which option yields the highest expected reward, balancing exploration (trying the inferior option to gather information) against exploitation (consistently choosing the superior option). The name derives from the colloquial term “one-armed bandit” for slot machines.

Formal Definition

On each trial t, the subject selects an action a ∈ {A, B} and receives a reward rₜ sampled from a probability distribution associated with that action. The subject maintains an estimate of the expected value Qₜ(a) for each action, updated according to a learning rule:

$Q_t=Q_{t-1}+\alpha (r_{t-1}-Q_{t-1})$

where α is the learning rate (0 < α ≤ 1) that scales how much new outcomes update the estimated value. Choice between actions is typically modelled via a softmax decision rule:

$P(A{)}_t=\frac{1}{1+\exp (-\beta (Q_{t,A}-Q_{t,B})-\xi \cdot \text{sign}(a_{t-1}))}$

where β is the choice stochasticity (inverse temperature) and ξ captures any choice repetition bias.

The Noisy Learning Model

Recent work (Findling et al., 2019) demonstrated that more than half of seemingly stochastic choices can be explained by learning noise rather than choice stochasticity. The noisy learning model extends the standard Q-learning update by adding a noise term εₜ:

$Q_t=Q_{t-1}+\alpha (r_{t-1}-Q_{t-1})+{\epsilon }_t$

where εₜ is drawn from a normal distribution with mean 0 and standard deviation σₜ = ζ|rₜ₋₁ − Qₜ₋₁| (i.e., the noise scales with the prediction error, similar to Weber’s law). The parameter ζ captures learning noise — the imprecision with which reward information is incorporated into value estimates — which is now recognised as a stable, meaningful individual-difference variable dissociable from choice stochasticity (Skvortsova & Hauser, 2022).

Applications

The two-armed bandit is widely used across species (humans, nonhuman primates, rodents) to study:

• Reward-based decision-making and the explore/exploit dilemma
• Individual differences in learning rate, learning noise, and choice stochasticity
• Clinical populations — altered learning noise distinguishes impulsivity in ADHD from compulsivity in OCD (Skvortsova & Hauser, 2022)
• Neural correlates — dopamine-dependent reward prediction errors, ventromedial prefrontal value coding, striatal reward processing
• Cognitive flexibility — related to but distinct from intradimensional-extradimensional-shift tasks, which test attentional set-shifting rather than trial-by-trial value updating

Key Study: Vejlø et al. (2026)

A large-scale quantitative MRI study (Vejlø, Nikolova, Banellis, Tyrer, Skvortsova, Hauser & Allen, 2026; bioRxiv) used a gamified two-armed bandit task — the Milky Way game (Brain Explorer app; see www.brainexplorer.net) — with 248 participants to investigate the microstructural brain correlates of learning parameters.

Task Design

• Gamified version on the Brain Explorer app: two “space cows” (brown and black-and-white) as bandits
• Goal: accumulate points (“space milk”) by choosing the cow with the higher reward
• Self-paced choice, outcome shown only for the chosen cow
• 72 trials split into two rounds of 36
• Reward magnitudes (1–99 points) sampled from two probability distributions with independently drifting means (Skvortsova & Hauser, 2022)

Computational Model

The noisy learning model was fit to each participant’s choices, extracting three parameters per participant: learning rate (α) for the chosen and unchosen object, learning noise (ζ), and choice stochasticity (β).

Neuroimaging Methods

• Whole-brain quantitative MRI at 3T (Magnetom Prisma)
• Multi-parameter mapping (MPM) protocol: R1 (myelin density), R2* (iron concentration), MT (myeloarchitecture), PD (water content)
• Voxel-Based Quantification (VBQ) with TFCE correction (pFWE < .05)
• Nuisance covariates: age, sex, BMI, total intracranial volume

Key Findings

Parameter	Brain Region	Microstructural Measure
Learning rate (chosen object α)	Right cerebellum exterior (k = 35,385)	↑ R1 (myelination)
Learning noise (ζ)	Right cerebellum (k = 24,875)	↑ R1 (myelination)
Learning noise (ζ)	Left precentral gyrus, medial segment	↑ R1 (myelination) + ↑ R2* (iron)

1. Learning rate was positively associated with myelination in the right cerebellum — consistent with the cerebellum’s emerging role in forming internal models for reward prediction (Nicholas et al., 2024).
2. Learning noise was associated with myelination and iron concentration in the left precentral gyrus (motor cortex) — a novel finding suggesting that imprecision in value updating may propagate into motor planning and selection systems.
3. Learning noise was also associated with cerebellar myelination, overlapping with but anatomically distinct from the learning rate cluster.

No significant associations were found for choice stochasticity (β) or MT values, suggesting these relationships are specific to learning noise rather than choice noise, and to myelin density/iron rather than broader myeloarchitecture.

Significance

The findings provide the first evidence that distinct microstructural profiles (cerebellar myelin vs. motor cortex myelin + iron) underlie variability in learning efficiency vs. learning noise. This suggests reward learning is not a unitary process but is shaped by separable neurobiological pathways supporting learning precision and noise, with potential implications for ADHD, OCD, and other disorders of decision-making.

Data & Code

• Code and demographics: GitHub
• Statistical brain maps: NeuroVault collection HSOJRRQV

Related Study: Tyrer et al. (2025) — Exploration vs Exploitation

A companion study from the same group (Tyrer, Nikolova, Dubois, Banellis, Vejlø, Hauser & Allen, 2025; bioRxiv) used a multi-armed bandit (3-bandit) paradigm — Maggie’s Farm — with 122 participants from the same Visceral Mind Project cohort to investigate the microstructural correlates of exploration-exploitation decision strategies.

Task Design

• Maggie’s Farm (Dubois et al., 2021): three bandits depicted as trees bearing apples
• Each bandit associated with a normally-distributed reward distribution (apple size)
• Bandits had high (3 samples), limited (1 sample), or no prior reward information
• Short horizon (1 draw) vs long horizon (6 draws) trials
• Goal: maximise total reward (apples collected before “sunset”)

Computational Model

Thompson sampling model with two heuristic additions: ε-greedy (value-free random exploration) and novelty bonus η. Parameters fitted separately for short and long horizon trials.

Key Neuroimaging Findings

Contrast	Brain Region	Microstructural Measure	Trial Horizon
Value-free random exploration (ε)	Right superior frontal gyrus (k = 15,138) + right middle frontal gyrus (k = 3,467)	↑ R1 (myelination)	Long horizon only
Value-free random exploration (ε)	Right postcentral gyrus (k = 5,840) + right superior parietal lobule (k = 3,883)	↓ MT saturation	Short horizon only

1. R1 myelination in right frontal regions (SFG, MFG) was positively associated with value-free random exploration specifically when exploration was beneficial (long horizon trials). These frontal areas have prior links to impulsivity (Hu et al., 2016).
2. MT saturation in right postcentral gyrus and superior parietal lobule was negatively associated with value-free exploration in short horizon trials (where exploration is not beneficial), suggesting greater myeloarchitectural integrity supported optimal exploitation.
3. No significant associations with R2* (iron concentration), contrasting with the Vejlø et al. finding that learning noise (but not exploration) was linked to cortical iron.

Together, the two studies from this cohort show that learning parameters and exploration strategies have dissociable microstructural signatures: cerebellar myelin → learning rate, motor cortex myelin+iron → learning noise, frontal myelin → value-free exploration.

Data & Code

• Code: GitHub (shared repository)
• Statistical brain maps: NeuroVault collection HSOJRRQV

|## References

• Findling, C., Skvortsova, V., Dromnelle, R., Palminteri, S., & Wyart, V. (2019). Computational noise in reward-guided learning drives behavioral variability in volatile environments. Nature Neuroscience, 22(12), 2066–2077.
• Skvortsova, V., & Hauser, T. (2022). Distinct computational mechanisms underlying cognitive flexibility deficits in impulsivity and compulsivity. Research Square.
• Vejlø, M., Nikolova, N., Banellis, L., Tyrer, A., Skvortsova, V., Hauser, T. U., & Allen, M. G. (2026). Dissociable Microstructural Correlates of Learning Rate and Learning Noise in Gamified Reward-Based Decision-Making. bioRxiv. https://doi.org/10.64898/2026.02.24.707646 ¹
• Tyrer, A., Nikolova, N., Dubois, M., Banellis, L., Vejlø, M., Hauser, T. U., & Allen, M. (2025). Cortical Microstructural Variations Correlate with Individual Differences in Gamified Exploration–Exploitation Behaviours. bioRxiv. https://doi.org/10.1101/2025.10.08.681181 ²

Related Pages

intradimensional-extradimensional-shift | nonverbal-cognitive-tasks | active-inference | parametric-ied-stimulus-design

Footnotes

1. raw/papers/vejlo-2026-learning-rate-noise-microstructure.md ↩

2. raw/papers/tyrer-2025-exploration-exploitation-microstructure.md ↩