Report

Bayesian Optimization with Experimental Constraints Javad Azimi Advisor: Dr. Xiaoli Fern PhD Proposal Exam April 2012 1 Outline • Introduction to Bayesian Optimization • Completed Works – Constrained Bayesian Optimization – Batch Bayesian Optimization – Scheduling Methods for Bayesian Optimization • Future Works – Hybrid Bayesian optimization • Timeline 2 Bayesian Optimization • We have a black box function and we don’t know anything about its distribution • We are able to sample the function but it is very expensive • We are interested to find the maximizer (minimizer) of the function • Assumption: – lipschitz continuity Introduction to BO Constrained BO Batch BO Scheduling 3 Future Work Big Picture Current Experiments Posterior Model Select Experiment(s) Run Experiment(s) Introduction to BO Constrained BO Batch BO Scheduling 4 Future Work Posterior Model (1): Regression approaches • Simulates the unknown function distribution based on the prior – Deterministic (Classical Linear Regression,…) • There is a deterministic prediction for each point x in the input space – Stochastic (Bayesian regression, Gaussian Process,…) • There is a distribution over the prediction for each point x in the input space. (i.e. Normal distribution) – Example • Deterministic: f(x1)=y1, f(x2)=y2 • Stochastic: f(x1)=N(y1,0.2) f(x2)=N(y2,5) Introduction to BO Constrained BO Batch BO Scheduling 5 Future Work Posterior Model (2): Gaussian Process • Gaussian Process is used to build the posterior model Points with high output expectation – The prediction output at any point is a normal random variable – Variance is independent from observation y Points with high output variance – The mean is a linear combination of observation y 6 Selection Criterion • Goal: Which point should be selected next to get to the maximizer of the function faster. MM MUI MPI MEI • Maximum Mean (MM) – Selects the points which has the highest output mean – Purely exploitative • Maximum Upper bound Interval (MUI) – Select point with highest 95% upper confidence bound – Purely explorative approach • Maximum Probability of Improvement (MPI) – It computes the probability that the output is more than (1+m) times of the best current observation , m>0. – Explorative and Exploitative • Maximum Expected of Improvement (MEI) – Similar to MPI but parameter free – It simply computes the expected amount of improvement after sampling at any point Introduction to BO Constrained BO Batch BO Scheduling 7 Future Work Motivating Application: Fuel Cell This is how an MFC works Nano-structure of anode significantly impact the electricity production. e- e- Oxidation products (CO2) bacteria Fuel (organic matter) SEM image of bacteria sp. on Ni nanoparticle enhanced carbon fibers. O2 H+ H2 O Cathode Anode We should optimize anode nano-structure to maximize power by selecting a set of experiment. 8 Introduction to BO Constrained BO Batch BO Scheduling Future Work Other Applications • • • • • Financial Investment Reinforcement Learning Drug test Destructive tests And … Introduction to BO Constrained BO Batch BO Scheduling 9 Future Work Constrained Bayesian optimization (AAAI 2010, to be submitted Journal) Introduction to BO Constrained BO Batch BO Scheduling 10 Future Work Problem Definition(1) • BO assumes that we can ask for specific experiment • This is unreasonable assumption in many applications – In Fuel Cell it takes many trials to create a nanostructure with specific requested properties. – Costly to fulfill Introduction to BO Constrained BO Batch BO Scheduling 11 Future Work Problem Definition(2) • It is less costly to fulfill a request that specifies ranges for the nanostructure properties • E.g. run an experiment with Averaged Area in range r1 and Average Circularity in range r2 • We will call such requests “constrained experiments” Averaged Area Space of Experiments Constrained Experiment 1 • large ranges • low cost • high uncertainty about which experiment will be run Constrained Experiment 2 • small ranges • high cost • low uncertainty about which experiment will be run Average Circularity Introduction to BO Constrained BO Batch BO Scheduling 12 Future Work Proposed Approach • We introduced two different formulation • Non Sequential – Select all experiments at the same time • Sequential – Only one constraint experiment is selected at each iteration • Two challenges: – How to compute heuristics for constrained experiment? – How to take experimental cost into account?(which has been ignored by most of the approaches in BO) Introduction to BO Constrained BO Batch BO Scheduling 13 Future Work Non-Sequential • All experiments must be chosen at the same time • Objective function: – A sub set of experiments (with cost B) which jointly have the highest expected maximum is selected, i.e. E[Max(.)] Introduction to BO Constrained BO Batch BO Scheduling 14 Future Work Submodularity • It simply means adding an element to the smaller set provides us with more improvement than adding an element to the larger set • Example: We show that max (.) is submodular – S1={1, 2, 4}, S2={1, 2, 4, 8}, (S1 is a subset of S2), g=max(.) and x=6 – g(S1, x) - g(S1)=2, g(S2,x)-g(S2)=0 • E[max(.)] over a set of jointly normal random variable is a submodular function • Greedy algorithm provides us with a “constant” approximation bound Introduction to BO Constrained BO Batch BO Scheduling 15 Future Work Greedy Algorithm Introduction to BO Constrained BO Batch BO Scheduling 16 Future Work Sequential Policies • Having the posterior distribution of p(y|x,D) and px(.|D) we can calculate the posterior of the output of each constrained experiment which has a closed form solution Discretization Level Input space • Therefore we can compute standard BO heuristics for constrained experiments – There are closed form solution for these heuristics Introduction to BO Constrained BO Batch BO Scheduling 17 Future Work Budgeted Constrained • We are limited with Budget B. • Unfortunately heuristics will typically select the smallest and most costly constrained experiments which is not a good use of budget -Low uncertainty -High uncertainty -Better heuristic value -Lower heuristic value -Expensive -Cheap • How can we consider the cost of each constrained experiment in making the decision? – Cost Normalized Policy (CN) – Constraint Minimum Cost Policy(CMC) Introduction to BO Constrained BO Batch BO Scheduling 18 Future Work Cost Normalized Policy • It selects the constrained experiment achieving the highest expected improvement per unit cost • We report this approach for MEI policy only Introduction to BO Constrained BO Batch BO Scheduling 19 Future Work Constraint Minimum Cost Policy (CMC) • Motivation: 1. Approximately maximizes the heuristic value 2. Has expected improvement at least as great as spending the same amount of budget on random experiments • Example: Cost=10 random Cost=4 random Poor heuristic value: not select due to 1st condition Introduction to BO Very expensive: 10 random experiments likely to be better Constrained BO Batch BO Cost=5 random Selected Constrained experiment Scheduling 20 Future Work Results (1) Real Fuel Cell CMC-MEI Cosines Rosenbrock Introduction to BO Constrained BO Batch BO Scheduling 21 Future Work Results (2) Real Rosenbrock Introduction to BO Constrained BO Fuel Cell NS Cosines Batch BO Scheduling 22 Future Work Batch Bayesian Optimization (NIPS 2010) Sometimes it is better to select batch. (Javad Azimi) 23 Motivation • Traditional BO approach request a single experiment at each iteration • This is not time efficient when running an experiment is very time consuming and there is enough facilities to run up to k experiments concurrently • We would like to improve performance per unit time by selecting/running k experiments in parallel • A good batch approach can speedup the experimental procedure without degrading the performance Introduction to BO Constrained BO Batch BO Scheduling 24 Future Work Main Idea • We Use Monte Carlo simulation to select a batch of k experiments that closely match what a good sequential policy selection in k steps Given a sequential Policy and batch size k x21 x31 xn1 x12 x22 x32 xn2 x13 ... .. x23 ... .. x33 ... .. xn3 ... .. x1k x2k x3k ... ... .. x11 xnk Return B*={x1,x2,…,xk} Introduction to BO Constrained BO Batch BO Scheduling 25 Future Work Objective Function(1) • Simulated Matching: – Having n different trajectories with length k from a given sequential policy – We want to select a batch of k experiments that best matches the behavior of the sequential policy • This objective can be viewed as minimizing an upper bound on the expected performance difference between the sequential policy and the selected batch. • This objective is similar to weighted k-medoid Introduction to BO Constrained BO Batch BO Scheduling 26 Future Work Supermodularity • Example: Min(.) is a supermodual function – B1={1, 2, 4}, B2={1, 2, 4, -2}, f=min(.) and x = 0 – f(B1) -f(B1, x)=1, f(B2)-f(B2, x)=0 • Quiz: What is the difference between submodular and supermodular function? – If the inequality is changed then we have submodular function • The proposed objective function is a supermodular function • The greedy algorithm provides us with an approximation bound Introduction to BO Constrained BO Batch BO Scheduling 27 Future Work Algorithm Introduction to BO Constrained BO Batch BO Scheduling 28 Future Work Results (5) Greedy Introduction to BO Constrained BO Batch BO Scheduling 29 Future Work Scheduling Methods for Bayesian Optimization (NIPS 2011(spotlight)) 30 Extended BO Model Time Horizon h Lab 1 x1 Lab 2 x2 x4 x5 x8 x3 Lab 3 xn-1 x7 x6 Lab l Problem: xn We consider the following: • Concurrent experiments (up to l exp. at any time) • Stochastic exp. durations (known distribution p) • Experiment budget (total of n experiments) • Experimental time horizon h Stochastic Experiment Durations Schedule when to start new experiments and which ones to start Introduction to BO Constrained BO Batch BO Scheduling 31 Future Work Challenges Objective 2: maximize info. used in selecting each experiments (favors minimizing concurrency) Lab 1 x1 x4 Lab 2 x2 x5 x3 x6 Lab 3 Lab 4 x4 x1 x2 ⋯ xn x7 We present online and offline approaches that effectively trade off these two conflicting objectives Introduction to BO Constrained BO Batch BO Scheduling 32 Future Work Objective Function • Cumulative prior experiments (CPE) of E is measured as follows: • Example: Suppose n1=1, n2=5, n3=5, n4=2, Then CPE=(1*0)+(5*1)+(5*6)+(2*11)=57 • We found a non trivial correlation between CPE and regret Introduction to BO Constrained BO Batch BO Scheduling 33 Future Work Offline Scheduling • Assign start times to all n experiments before the experimental process begins • The experiment selection is done online • Two class of schedules are presented – Staged Schedules – Independent Labs Introduction to BO Constrained BO Batch BO Scheduling 34 Future Work Staged Schedules • There are N stage and each stage is represent as <ni,di> such that – CPE is calculated as: – We call an schedule uniform if |ni-nj|<2 n1=4 n2=3 n3=4 n4=3 x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 x11 x12 x13 x14 d1 d2 h d3 d4 • Goal: finding a p-safe uniform schedule with maximum number of stages. Introduction to BO Constrained BO Batch BO Scheduling 35 Future Work Staged Schedules: Schedule Introduction to BO Constrained BO Batch BO Scheduling 36 Future Work Independent Lab (IL) • • • • Assigns mi experiment to each lab i such that Experiments are distributed uniformly within the labs Start times of different labs are decoupled The experiments in each lab have equal duration to maximize the finishing probability within horizon h • Mainly designed for policy switching schedule Lab1 x11 x12 x13 x14 Lab2 x21 x22 x23 x24 Lab3 x31 x32 x33 Lab4 x41 x42 x43 h Introduction to BO Constrained BO Batch BO Scheduling 37 Future Work Online Schedules • p-safe guarantee is fairly pessimistic and we can decrease the parallelization degree in practice • Selects the start time of experiments online rather than offline • More flexible than offline schedule Introduction to BO Constrained BO Batch BO Scheduling 38 Future Work Baseline online Algorithms • Online Fastest Completion policy (OnFCP) – Finish all of the n experiments as quickly as possible – Keeps all l labs busy as long as there are experiments left to run – Achieves the lowest possible CPE • Online Minimum Eager Lab Policy (OnMEL) – OnFCP does not attempt to use the full time horizon – use only k labs, where k is the minimum number of labs required to finish n experiments with probability p Introduction to BO Constrained BO Batch BO Scheduling 39 Future Work Policy Switching (PS) • PS decides about the number of new experiments at each decision step • Assume a set of policies or a policy generator is given • The goal is defining a new policy which performs as well as or better than the best given policy at any state s • The i-th policy waits to finish i experiments and then call offIL algorithm to reschedule • The policy which achieves the maximum CPE is returned • The CPE of the switching policy will not be much worse than the best of the policies produced by our generator Introduction to BO Constrained BO Batch BO Scheduling 40 Future Work Experimental Results Setting: h=4,5,6; pd=Truncated normal distribution, n=20 and L=10 Best Performance Best CPE in each setting Introduction to BO Constrained BO Batch BO Scheduling 41 Future Work Future Work Introduction to BO Constrained BO Batch BO Scheduling 42 Future Work Traditional Approaches •Sequential: – Only one experiment is selected at each iteration – Pros: Performance is optimized – Cons: Can be very costly when running one experiment takes long time • Batch: – k>1 experiments are selected at each iteration – Pros: k times speed-up comparing to sequential approaches – Cons: Can not performs as well as sequential algorithms Introduction to BO Constrained BO Batch BO Scheduling 43 Future Work Batch Performance (Azimi et.al NIPS 2010) k=5 k=10 Introduction to BO Constrained BO Batch BO Scheduling 44 Future Work Hybrid Batch • Sometimes, the selected points by a given sequential policy at a few consequent steps are independent from each other • Size of the batch can change at each time step (Hybrid batch size) Introduction to BO Constrained BO Batch BO Scheduling 45 Future Work First Idea (NIPS Workshop 2011) • Based on a given prior (blue circles) and an objective function (MEI), x1 is selected • To select the next experiment, x2 , we need, y1=f(x1) which is not available • The statistics of the samples inside the red circle are expected to change after observing at actual y1 • We set y1 =M and then EI of the next x2 x3 x1 step is upper bounded • If the next selected experiment is outside of the red circle, we claim it is independent from x1 Introduction to BO Constrained BO Batch BO Scheduling 46 Future Work Next • Very pessimistic to set Y=M and then the speedup is small • Can we select the next point based on any estimation without degrading the performance? • What is the distance of selected experiments in batch and the actual selected experiments by sequential policy? Introduction to BO Constrained BO Batch BO Scheduling 47 Future Work TimeLine • Spring 2012: Finishing the Hybrid batch approach • Summer 2012: Finding a job and final defend (hopefully ) Introduction to BO Constrained BO Batch BO Scheduling 48 Future Work Publications 49 And • I would like to thank Dr. Xaioli Fern and Dr. Alan Fern Introduction to BO Constrained BO Batch BO Scheduling 50 Future Work Introduction to BO Constrained BO Batch BO Scheduling 51 Future Work Results (1) Random Introduction to BO Constrained BO Batch BO Scheduling 52 Future Work Results (2) Sequential Introduction to BO Constrained BO Batch BO Scheduling 53 Future Work Results (3) EMAX Introduction to BO Constrained BO Batch BO Scheduling 54 Future Work Results (4) K-means Introduction to BO Constrained BO Batch BO Scheduling 55 Future Work Constrained BO: Results Real Fuel Cell CMC-MUI Random Cosines Rosenbrock Introduction to BO Constrained BO Batch BO Scheduling Future Work 56 Constrained BO: Results Real Fuel Cell CN-MEI Cosines Rosenbrock Introduction to BO Constrained BO Batch BO Scheduling Future Work 57 Constrained BO: Results Real Fuel Cell CMC-MPI(0.2) Cosines Rosenbrock Introduction to BO Constrained BO Batch BO Scheduling Future Work 58 PS Performance Bound • ( s , t ) is our policy generator at each time step t and state s • State s is the current running experiments with their starting time and completed experiments. • ( s , t , ) denotes is the policy switching result whereis the base policy selected in the last step • The decision by is returned by N independent simulations. • C t (s) is the CPE of policy with error Introduction to BO Constrained BO Batch BO Scheduling 59 Future Work