Report

Tutorial: Optimal Learning in the Laboratory Sciences Working with nonlinear belief models December 10, 2014 Warren B. Powell Kris Reyes Si Chen Princeton University http://www.castlelab.princeton.edu 1 Slide Slide 1 Lecture outline Nonlinear belief models 2 Knowledge Gradient with Discrete Priors The knowledge gradient can be hard to compute: x KG ,n E m ax y F ( y , K The expectation can be hard to compute when the belief model is nonlinear. n 1 ( x )) m ax y F ( y , K ) n The belief model is often nonlinear, such as the kinetic model for fluid dynamics. This has motivated research into how to handle these problems. 3 Knowledge Gradient with Discrete Priors Proposal: Assume a finite number of truths (discrete priors), e.g. L=3 possible candidate truths Utility curve depends on kinetic parameters, e.g 1, 2 , 3 We maintain the weights of each of the possible candidates to represent how likely it is the truth, e.g. p1=p2=p3=1/3 means equally likely 4 Knowledge Gradient with Discrete Priors Utility curve depends on kinetic parameters. The weights on the candidate truths are also on the choice of kinetic parameters: Knowledge Gradient with Discrete Priors Estimation: a weighted sum of all candidate truths Knowledge Gradient with Discrete Priors There are many possible candidate truths For each candidate truths, the measurements are noisy Utility curve depends on kinetic parameters. Knowledge Gradient with Discrete Priors Suppose we make a measurement Knowledge Gradient with Discrete Priors Weights are updated upon observation Observation More likely based on observation. Less likely based on observation Knowledge Gradient with Discrete Priors Estimate is then updated using our observation Average Marginal of Information Best estimate: maximum utility value Marginal value of information Average marginal value of information: average across all candidate truths and noise Best estimate before the experiment Best estimate after the experiment Knowledge Gradient with Discrete Priors KGDP makes decisions by maximizing the average marginal of information After several observations, the weights can tell us about the truth 12 Candidate Truths (2D) Beliefs on parameters produces family of surfaces 13 Before any measurements 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 5 6 7 8 9 Oil droplet diameter (nm) Do we explore? The KG map shows us where we learn the most. 10 Prior Estimate Inner water droplet diameter (nm) Inner water droplet diameter (nm) KG “Road Map” 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 5 6 7 8 9 Oil droplet diameter (nm) … or do we exploit? This is the region where we think we will get the best results (but we might be wrong). This is the classic exploration vs. exploitation problem 10 Before any measurements 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 5 6 7 8 9 Oil droplet diameter (nm) Do we explore? The KG map shows us where we learn the most. 10 Prior Estimate Inner water droplet diameter (nm) Inner water droplet diameter (nm) KG “Road Map” 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 5 6 7 8 9 Oil droplet diameter (nm) … or do we exploit? This is the region where we think we will get the best results (but we might be wrong). This is the classic exploration vs. exploitation problem 10 Before any measurements 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 5 6 7 8 9 Oil droplet diameter (nm) 10 Prior Estimate Inner water droplet diameter (nm) Inner water droplet diameter (nm) KG “Road Map” 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 5 6 7 8 9 Oil droplet diameter (nm) 10 After 1 measurement 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 5 6 7 8 9 Oil droplet diameter (nm) 10 Posterior Estimate Inner water droplet diameter (nm) Inner water droplet diameter (nm) KG “Road Map” 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 5 6 7 8 9 Oil droplet diameter (nm) 10 After 2 measurements 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 5 6 7 8 9 Oil droplet diameter (nm) 10 Posterior Estimate Inner water droplet diameter (nm) Inner water droplet diameter (nm) KG “Road Map” 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 5 6 7 8 9 Oil droplet diameter (nm) 10 After 5 measurements 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 5 6 7 8 9 Oil droplet diameter (nm) 10 Posterior Estimate Inner water droplet diameter (nm) Inner water droplet diameter (nm) KG “Road Map” 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 5 6 7 8 9 Oil droplet diameter (nm) 10 After 10 measurements 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 5 6 7 8 9 Oil droplet diameter (nm) 10 Posterior Estimate Inner water droplet diameter (nm) Inner water droplet diameter (nm) KG “Road Map” 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 5 6 7 8 9 Oil droplet diameter (nm) 10 After 20 measurements 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 5 6 7 8 9 Oil droplet diameter (nm) 10 Posterior Estimate Inner water droplet diameter (nm) Inner water droplet diameter (nm) KG “Road Map” 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 5 6 7 8 9 Oil droplet diameter (nm) 10 After 20 measurements Posterior Estimate 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 5 6 7 8 Oil droplet diameter (nm) 9 10 Inner water droplet diameter (nm) Inner water droplet diameter (nm) Truth 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 5 6 7 8 9 Oil droplet diameter (nm) 10 Kinetic parameter estimation Besides learning where optimal utility is, the KG policy can help learn kinetic parameters. Distribution on candidate truths induces a distribution on their respective parameters. Candidate Probability Probability 0.7 0.6 Uniform prior distribution 0.5 0.4distribution of possible parameter vectors… Uniform 0.3 0.2 0.1 0 0 Parameter Probability 10 20 30 Candidate Truth 40 1 1 … translates to random sample of a 0.5 uniform distribution for an individual parameter. 0.5 0 50 7 8 9 10 11 0 0.52 0.54 0.56 0. Kinetic parameter estimation 0.5 Probability 0 7 8 9 10 11 0 1 1 0.5 0.5 0 26 28 30 0 1 0.5 0.5 Probability 0.5 1 0 0.52 0.54 0.56 0.58 0.82 0.84 0.86 0.88 Prior distribution 7 8 9 10 11 0 1 1 0.5 0.5 Probability 1 Probability 1 0 26 28 30 0 0.52 0.54 0.56 0.58 0.82 0.84 0.86 0.88 After 20 measurements Kinetic parameter estimation Probability 1 1 Low prefactor/low barrier 0.5 0.5 0 7 8 9 10 11 0 0.52 0.54 0.56 0.58 High prefactor/high barrier 1 1 0.5 0.5 Probability • Most probable prefactor/ energy barriers come in pairs. • Yield similar rates at room temperature. • KG is learning these rates. 0 26 28 30 0 0.82 0.84 0.86 0.88 After 20 measurements Kinetic parameter estimation After 50 measurements, distribution of belief about vectors… … distribution of belief about k ripe : k ripe After 50 measurements, distribution of belief about vectors… … distribution of belief about one parameter: k coalesce 28 Collaboration with McAlpine Group Opportunity Cost Percentage opportunity cost: difference between estimated and true optimum value w.r.t the true optimum value 29 Rate Error Rate error (log-scale): difference between the estimated rate and the true optimal rate 30