Report

dopamine and prediction error TD error t rt Vt 1 Vt L R Vt (t ) R no prediction 1 prediction, reward prediction, no reward Schultz 1997 humans are no different • dorsomedial striatum/PFC – goal-directed control • dorsolateral striatum – habitual control • ventral striatum – Pavlovian control; value signals • dopamine... in humans… 5 stimuli: 40¢ 20¢ 0/40¢ 0¢ 0¢ < 1 sec 5 sec ISI 0.5 sec You won 40 cents 19 subjects (dropped 3 non learners, N=16) 3T scanner, TR=2sec, interleaved 234 trials: 130 choice, 104 single stimulus randomly ordered and counterbalanced 2-5sec ITI what would a prediction error look like (in BOLD)? prediction errors in NAC unbiased anatomical ROI in nucleus accumbens (marked per subject*) raw BOLD (avg over all subjects) can actually decide between different neuroeconomic models of risk * thanks to Laura deSouza Polar Exploration Peter Dayan Nathaniel Daw John O’Doherty Ray Dolan Exploration vs. exploitation Classic dilemma in learned decision making For unfamiliar outcomes, how to trade off learning about their values against exploiting knowledge already gained Exploration vs. exploitation Reward Time • Exploitation – Choose action expected to be best – May never discover something better Exploration vs. exploitation Reward Time • Exploitation – Choose action expected to be best – May never discover something better • Exploration: – Choose action expected to be worse Exploration vs. exploitation Reward Time • Exploitation – Choose action expected to be best – May never discover something better • Exploration: – Choose action expected to be worse – If it is, then go back to the original Exploration vs. exploitation Reward Time • Exploitation – Choose action expected to be best – May never discover something better • Exploration: – Choose action expected to be worse Exploration vs. exploitation Reward Time • Exploitation – Choose action expected to be best – May never discover something better • Exploration: – Choose action expected to be worse – If it is better, then exploit in the future Exploration vs. exploitation Reward Time • Exploitation – Choose action expected to be best – May never discover something better • Exploration: – – – – Choose action expected to be worse Balanced by the long-term gain if it turns out better (Even for risk or ambiguity averse subjects) nb: learning non trivial when outcomes noisy or changing Bayesian analysis (Gittins 1972) • Tractable dynamic program in restricted class of problems – “n-armed bandit” • Solution requires balancing – Expected outcome values – Uncertainty (need for exploration) – Horizon/discounting (time to exploit) – Choose best sum of value plus bonus – Bonus increases with uncertainty Value • Optimal policy: Explore systematically • Intractable in general setting – Various heuristics used in practice Action Experiment • How do humans handle tradeoff? • Computation: Which strategies fit behavior? – Several popular approximations • Difference: what information influences exploration? • Neural substrate: What systems are involved? – PFC, high level control • Competitive decision systems (Daw et al. 2005) – Neuromodulators • dopamine (Kakade & Dayan 2002) • norepinephrine (Usher et al. 1999) Task design Subjects (14 healthy, righthanded) repeatedly choose between four slot machines for points (“money”), in scanner Trial Onset Slots revealed Task design Subjects (14 healthy, righthanded) repeatedly choose between four slot machines for points (“money”), in scanner Trial Onset +~430 ms Slots revealed Subject makes choice chosen slot spins. + Task design Subjects (14 healthy, righthanded) repeatedly choose between four slot machines for points (“money”), in scanner Trial Onset +~430 ms Slots revealed Subject makes choice chosen slot spins. + + obtained 57 points +~3000 ms Outcome: Payoff revealed Task design Subjects (14 healthy, righthanded) repeatedly choose between four slot machines for points (“money”), in scanner Trial Onset +~430 ms Slots revealed Subject makes choice chosen slot spins. + + obtained 57 points + +~3000 ms Outcome: +~1000 ms Payoff revealed Screen cleared Trial ends Payoff structure Noisy to require integration of data Subjects learn about payoffs only by sampling them Payoff structure Noisy to require integration of data Subjects learn about payoffs only by sampling them Payoff Payoff structure Payoff structure Nonstationary to encourage ongoing exploration (Gaussian drift w/ decay) Analysis strategy • Behavior: Fit an RL model to choices – Find best fitting parameters – Compare different exploration models • Imaging: Use model to estimate subjective factors (explore vs. exploit, value, etc.) – Use these as regressors for the fMRI signal – After Sugrue et al. Behavior Behavior Behavior Behavior Behavior Behavior model 1. Estimate payoffs mgreen , mred etc sgreen , sred etc 2. Derive choice probabilities Pgreen , Pred etc Choose randomly according to these Behavior model Kalman filter Error update 1. Estimate payoffs (like TD) Exact inference mgreen , mred etc sgreen , sred etc 2. Derive choice probabilities Pgreen , Pred etc Choose randomly according to these Behavior model 2. Derive choice probabilities x x payoff Kalman filter Error update 1. Estimate payoffs (like TD) Exact inference mgreen , mred etc sgreen , sred etc x trial t Pgreen , Pred etc Choose randomly according to these t+1 Behavior model 2. Derive choice probabilities x x payoff Kalman filter Error update 1. Estimate payoffs (like TD) Exact inference mgreen , mred etc sgreen , sred etc x trial t Pgreen , Pred etc Choose randomly according to these t+1 Behavior model 2. Derive choice probabilities x x x payoff Kalman filter Error update 1. Estimate payoffs (like TD) Exact inference mgreen , mred etc sgreen , sred etc x trial t t+1 mred mred Pgreen , Pred etc Choose randomly according to these Behavior model 2. Derive choice probabilities x x payoff Kalman filter Error update 1. Estimate payoffs (like TD) Exact inference mgreen , mred etc sgreen , sred etc x x x x trial t t+1 mred mred Pgreen , Pred etc Choose randomly according to these Kalman filter Error update 1. Estimate payoffs (like TD) Exact inference mgreen , mred etc sgreen , sred etc 2. Derive choice probabilities Pgreen , Pred etc payoff Behavior model trial t t+1 mred mred 2 2 2 sred / sre s d o 2 2 sred (1 )sred Choose randomly according to these Behrens & volatility Behavior model Kalman filter 1. Estimate payoffs mgreen , mred etc sgreen , sred etc Compare rules: How is 2. Derive choice exploration probabilities directed? Pgreen , Pred etc Choose randomly according to these Behavior model mgreen , mred etc sgreen , sred etc Compare rules: How is 2. Derive choice exploration probabilities directed? Pgreen , Pred etc Choose randomly according to these mgreen , mred etc sgreen , sred etc Compare rules: How is 2. Derive choice exploration probabilities directed? (dumber) Value Behavior model Action (smarter) mgreen , mred etc sgreen , sred etc Compare rules: How is 2. Derive choice exploration probabilities directed? Value Behavior model Action Probability Randomly “e-greedy” 1 3e Pred e (dumber) if mred max(all m ) otherwise (smarter) mgreen , mred etc sgreen , sred etc Compare rules: How is 2. Derive choice exploration probabilities directed? Action By value “softmax” Probability Randomly “e-greedy” Value Behavior model 1 3e Pred e (dumber) if mred max(all m ) otherwise Pred exp( βμred ) (smarter) mgreen , mred etc sgreen , sred etc Compare rules: How is 2. Derive choice exploration probabilities directed? By value “softmax” Action By value and uncertainty “uncertainty bonuses” Probability Randomly “e-greedy” Value Behavior model 1 3e Pred e (dumber) if mred max(all m ) otherwise Pred exp( βμred ) Pred exp( β[ μred +φσred ]) (smarter) Model comparison • Assess models based on likelihood of actual choices – Product over subjects and trials of modeled probability of each choice – Find maximum likelihood parameters • Inference parameters, choice parameters • Parameters yoked between subjects • (… except choice noisiness, to model all heterogeneity) Behavioral results -log likelihood (smaller is better) # parameters e-greedy softmax uncertainty bonuses 4208.3 3972.1 3972.1 19 19 20 • Strong evidence for exploration directed by value • No evidence for direction by uncertainty – Tried several variations Behavioral results -log likelihood (smaller is better) # parameters e-greedy softmax uncertainty bonuses 4208.3 3972.1 3972.1 19 19 20 • Strong evidence for exploration directed by value • No evidence for direction by uncertainty – Tried several variations Imaging methods • 1.5 T Siemens Sonata scanner • Sequence optimized for OFC (Deichmann et al. 2003) • 2x385 volumes; 36 slices; 3mm thickness • 3.24 secs TR • SPM2 random effects model • Regressors generated using fit model, trial-by-trial sequence of actual choices/payoffs. Imaging results L vStr • TD error: dopamine targets (dorsal and ventral striatum) x,y,z= 9,12,-9 dStr • Replicate previous studies, but weakish – Graded payoffs? p<0.01 p<0.001 x,y,z= 9,0,18 Value-related correlates L vmPFC vmPFC p<0.01 p<0.001 % signal change probability (or exp. value) of chosen action: vmPFC probability of chosen action x,y,z=-3,45,-18 L mOFC p<0.01 p<0.001 x,y,z=3,30,-21 mOFC % signal change payoff amount: OFC payoff Exploration • Non-greedy > greedy choices: exploration • Frontopolar cortex • Survives whole-brain correction L rFP rFP p<0.01 p<0.001 x,y,z=-27,48,4; 27,57,6 LFP Timecourses Frontal pole IPS Checks • Do other factors explain differential BOLD activity better? – Multiple regression vs. RT, actual reward, predicted reward, choice prob, stay vs. switch, uncertainty, more – Only explore/exploit is significant – (But 5 additional putative explore areas eliminated) • Individual subjects: BOLD differences stronger for better behavioral fit Frontal poles • “One of the least well understood regions of the human brain” • No cortical connections outside PFC (“PFC for PFC”) • Rostrocaudal hierarchy in PFC (Christoff & Gabrielli 2000; Koechlin et al. 2003) • Imaging – high level control – Coordinating goals/subgoals (Koechlin et al. 1999, Braver & Bongiolatti 2002; Badre & Wagner 2004) – Mediating cognitive processes (Ramnani & Owen 2004) – Nothing this computationally specific • Lesions: task switching (Burgess et al. 2000) – more generic: perseveration Interpretation • Cognitive decision to explore overrides habit circuitry? Via parietal? – Higher FP response when exploration chosen most against the odds – Explore RT longer • Exploration/exploitation are neurally distinct • Computationally surprising, esp. bad for uncertainty bonus schemes – proper exploration requires computational integration – no behavioral evidence either • Why softmax? Can misexplore – Deterministic bonus schemes bad in adversarial/multiagent setting – Dynamic temperature control? (norepinephrine; Usher et al.; Doya) Conclusions • Subjects direct exploration by value but not uncertainty • Cortical regions differentially implicated in exploration – computational consequences • Integrative approach: computation, behavior, imaging – Quantitatively assess & constrain models using raw behavior – Infer subjective states using model, study neural correlates Open Issues • model-based vs model-free vs Pavlovian control – environmental priors vs naive optimism vs neophilic compulsion? • environmental priors and generalization – curiosity/`intrinsic motivation’ from expected future reward