Report

Rewarding Crowdsourced Workers Panos Ipeirotis New York University and Google Twitter: @ipeirotis Joint work with: Jing Wang, Foster Provost, Josh Attenberg, and Victor Sheng; “A Computer Scientist in a Business School” http://behind-the-enemy-lines.com Example: Build an Web Page Classifier Need a large number of labeled sites for training Get people to look at sites and label them as: G (general audience) PG (parental guidance) R (restricted) X (porn) Cost/Speed Statistics Undergrad intern: 200 websites/hr, cost: $15/hr Mechanical Turk: 2500 websites/hr, cost: $12/hr Challenges We do not know the true category for the objects – Available only after (costly) manual inspection We do not know quality of the workers We want to label objects with true categories We want (need?) to know the quality of the workers Expectation Maximization Estimation Iterative process to estimate worker error rates 1. Initialize “correct” label for each object (e.g., use majority vote) 2. Estimate error rates for workers (using “correct” labels) 3. Estimate “correct” labels (using error rates, weight worker votes according to quality) 4. Go to Step 2 and iterate until convergence Challenge: From Confusion Matrixes to Quality Scores Confusion matrix for spammer worker P[X → X]=0.847% P[G → X]=0.053% P[X → G]=99.153% P[G → G]=99.947% Confusion matrix for good worker P[X → X]=99.847% P[G → X]=4.053% P[X → G]=0.153% P[G → G]=95.947% How to check if a worker is a spammer using the confusion matrix? (hint: error rate not enough) Challenge 1: Spammers are lazy and smart! Confusion matrix for spammer Confusion matrix for good worker P[X → X]=0% P[X → G]=100% P[G → X]=0% P[G → G]=100% P[X → X]=80% P[G → X]=20% P[X → G]=20% P[G → G]=80% Spammers figure out how to fly under the radar… In reality, we have 85% G sites and 15% X sites Error rate of spammer = 0% * 85% + 100% * 15% = 15% Error rate of good worker = 85% * 20% + 85% * 20% = 20% False negatives: Spam workers pass as legitimate Challenge 2: Humans are biased! Error rates for legitimate (but biased) employee P[G → G]=20.0% P[P → G]=0.0% P[R → G]=0.0% P[X → G]=0.0% P[G → P]=80.0% P[P → P]=0.0% P[R → P]=0.0% P[X → P]=0.0% P[G → R]=0.0% P[P → R]=100.0% P[R → R]=100.0% P[X → R]=0.0% P[G → X]=0.0% P[P → X]=0.0% P[R → X]=0.0% P[X → X]=100.0% We have 85% G sites, 5% P sites, 5% R sites, 5% X sites Error rate of spammer (all G) = 0% * 85% + 100% * 15% = 15% Error rate of biased worker = 80% * 85% + 100% * 5% = 73% False positives: Legitimate workers appear to be spammers (important note: bias is not just a matter of “ordered” classes) Solution: Fix bias first, compute error rate afterwards Error Rates for legitimate (but biased) employee P[G → G]=20.0% P[P → G]=0.0% P[R → G]=0.0% P[X → G]=0.0% P[G → P]=80.0% P[P → P]=0.0% P[R → P]=0.0% P[X → P]=0.0% P[G → R]=0.0% P[P → R]=100.0% P[R → R]=100.0% P[X → R]=0.0% P[G → X]=0.0% P[P → X]=0.0% P[R → X]=0.0% P[X → X]=100.0% When biased worker says G, it is 100% G When biased worker says P, it is 100% G When biased worker says R, it is 50% P, 50% R When biased worker says X, it is 100% X Small ambiguity for “R-rated” votes but other than that, fine! Solution: Fix bias first, compute error rate afterwards Error Rates for spammer P[G → G]=100.0% P[P → G]=100.0% P[R → G]=100.0% P[X → G]=100.0% P[G → P]=0.0% P[P → P]=0.0% P[R → P]=0.0% P[X → P]=0.0% P[G → R]=0.0% P[P → R]=0.0% P[R → R]=0.0% P[X → R]=0.0% P[G → X]=0.0% P[P → X]=0.0% P[R → X]=0.0% P[X → X]=0.0% When spammer says G, it is 25% G, 25% P, 25% R, 25% X When spammer says P, it is 25% G, 25% P, 25% R, 25% X When spammer says R, it is 25% G, 25% P, 25% R, 25% X When spammer says X, it is 25% G, 25% P, 25% R, 25% X [note: assume equal priors] The results are highly ambiguous. No information provided! Expected Misclassification Cost • High cost: probability spread across classes • Low cost: probability mass concentrated in one class Assigned Label Corresponding “Soft” Label Label Cost Spammer: G <G: 25%, P: 25%, R: 25%, X: 25%> 0.75 Good worker: P <G: 100%, P: 0%, R: 0%, X: 0%> 0.0 [***Assume misclassification cost equal to 1, solution generalizes to arbitrary misclassification costs across categories] Quality A scalar measure of quality QualityScore: Score • A spammer is a worker who assigns labels randomly, regardless of what the true class is. Cost ( Worker) QualityScore( Worker) 1 Cost (Prior ) • Quality score useful for ranking workers • Unaffected by systematic biases • Scalar, so no need to examine confusion matrices Quality Score Question: How to pay workers? Challenges • Thresholding has wrong incentive structure: • Decent (but still useful) workers remain unused • If you are above the threshold, no need to improve • Uncertainty: The quality score is not really a fixed number • Fluctuations in payment are puzzling for workers • Best to have only increases in payment Two Types of Workers Divide workers into two groups – Qualified Workers • The quality satisfies the target quality – Unqualified Workers • The qualify fails to meet the target quality levels A Simple Pricing Model for Qualified Workers – p: the price paid to all qualified workers – fW(w): the pdf distribution of worker reservation wage – FW(w): the cdf distribution of worker reservation wage – R: the fixed price paid by external client for each qualified object Example Optimal Worker Salary = 21 Revenue fW(w) 15 fW(w): LogNormal(3,1), Selling price R=50 The Value of Unqualified Workers Binary classification (1:1) Worker confusion matrix “Accept” classification cost <=0.1 Number of Workers 1 3 5 7 9 11 Classification Cost 0.300 0.216 0.163 0.126 0.099 0.079 We need ~9 workers to achieve the required quality Value: 1/9 A Pricing Model for Workers – p: the price paid to all qualified workers – fW(w): the pdf distribution of worker reservation wage FW(w): the cdf distribution of worker reservation wage – Adjust for the presence of “unqualified” workers, and each unqualified worker “counts” as 1/k of a qualified one – R: the fixed price paid by external client for each qualified object Optimal Prices p* p*/3 p*/9 Quality Score Question: How to pay workers? Challenges • Thresholding has wrong incentive structure: • Decent (but still useful) workers remain unused • If you are above the threshold, no need to improve • Uncertainty: The quality score is not really a fixed number • Fluctuations in payment are puzzling for workers • Best to have only increases in payment Bayesian Estimates for Uncertainty Worker A P[0 → 0]=Beta(2,1) P[0 → 1]=Beta(1,2) P[1 → 0]=Beta(1,2) P[0 → 0]=Beta(2,1) Worker B P[0 → 0]=Beta(101,1) P[0 → 1]=Beta(1,101) P[1 → 0]=Beta(1,101) P[0 → 0]=Beta(101,1) Real-Time Payment and Reimbursement Example of the piece-rate payment of a worker # Tasks 10 20 30 40 Infinity Piece-rate Payment (cents) 11 18 21 23 40 Fair Payment Real-Time Payment and Reimbursement Example of the piece-rate payment of a worker # Tasks 10 20 30 40 Infinity Piece-rate Payment (cents) 11 18 21 23 40 Piece-rate Payment Fair Payment Potential “Bonus” Payment 10 Number of Tasks Real-Time Payment and Reimbursement Example of the piece-rate payment of a worker # Tasks 10 20 30 40 Infinity Piece-rate Payment (cents) 11 18 21 23 40 Piece-rate Payment Fair Payment Potential “Bonus” Reimbursement Payment Payment 10 20 Number of Tasks Real-Time Payment and Reimbursement Example of the piece-rate payment of a worker # Tasks 10 20 30 40 Infinity Piece-rate Payment (cents) 11 18 21 23 40 Piece-rate Payment Fair Payment Potential “Bonus” Reimbursement Reimbursement Payment Payment Payment 10 20 30 Real-Time Payment and Reimbursement Example of the piece-rate payment of a worker # Tasks 10 20 30 40 Infinity Piece-rate Payment (cents) 11 18 21 23 40 Piece-rate Payment Fair Payment Potential “Bonus” Reimbursement Reimbursement Reimbursement Payment Payment Payment Payment 10 20 30 40 Synthetic Experiment Setup N=10,000 tasks R=200 cents Cost <= 0.01 Labeling Process: Workers – Arrival frequency: every 600 seconds – Number of each arrival: 10 workers – Submitting speed: 30 seconds per task The evaluation criterion is Unit Time Profit: (Synthetic) Experimental Setup Scatter Plot of Confusion Matrix, Reservation Wage Qualify level and reservation wage independently distributed Qualify level and reservation wage positively correlated Experimental Results 35 Average Profit per Second 30 25 Quality-Based Price 24.6% Uniform Price: 4.4(30%-quantile) 20 Uniform Price: 5.7(40%-quantile) 15 159.6% Uniform Price: 7.4(50%-quantile) Uniform Price: 9.5(60%-quantile) 10 Uniform Price: 12.5(70%-quantile) 5 0 No Correlation Positive Correlation Workers reacting to bad rewards/scores Score-based feedback leads to strange interactions: The “angry, has-been-burnt-too-many-times” worker: “F*** YOU! I am doing everything correctly and you know it! Stop trying to reject me with your stupid ‘scores’!” The overachiever worker: “What am I doing wrong?? My score is 92% and I want to have 100%” 29 National Academy of Sciences Dec 2010 “Frontiers of Science” conference Your workers behave like my mice! An unexpected connection… 30 Your workers behave like my mice! Eh? 31 Your workers want to use only their motor skills, not their cognitive skills 32 The Biology Fundamentals 33 Brain functions are biologically expensive (20% of total energy consumption in humans) Motor skills are more energy efficient than cognitive skills (e.g., walking) Brain tends to delegate easy tasks to part of the neural system that handles motor skills An unexpected connection at the NAS “Frontiers of Science” conf. Your workers want to use only their motor skills, not their cognitive skills Makes sense 34 An unexpected connection at the NAS “Frontiers of Science” conf. And here is how I train my mice to behave… 35 The Mice Experiment 36 Cognitive Solve maze Find pellet Motor Push lever three times Pellet drops How to Train the Mice? Confuse motor skills! Reward cognition! I should try this the moment that I get back to my room 37 Punishing Worker’s Motor Skills Punish bad answers with frustration of motor skills (e.g., add delays between tasks) – – – “Loading image, please wait…” “Image did not load, press here to reload” “404 error. Return the HIT and accept again” →Make this probabilistic to keep feedback implicit 38 Rewarding (?) Cognitive Effort Reward good answers by rewarding the cognitive part of the brain – – – – – 39 Introduce variety Introduce novelty Give new tasks fast Show score improvements faster (but not the opposite) Show optimistic score estimates 40 Experiments 41 Web page classification Image tagging Email & URL collection Experimental Summary (I) Spammer workers quickly abandon – – – Good workers unaffected – – – 42 No need to display scores, or ban Low quality submissions from ~60% to ~3% Half-life of low-quality from 100+ HITs to less than 5 No significant effect on participation of workers with good performance Lifetime of participants unaffected Longer response times (after removing the “intervention delays”; that was puzzling) Experimental Summary (II) Remember, scheme was for training the mice… 15%-20% of the spammers start submitting good work! ???? 43 Two key questions Why response time was slower for some good workers? Why some low quality workers start working well? ???? 44 System 1: “Automatic” actions System 2: “Intelligent” actions 45 System 1 Tasks 46 System 2 Tasks 47 Not Performing Well? Disrupt and Engage System 2 Performing Well? Out Status: Usage of System 1 Status: Usage of System 2 (“Automatic”) (“Intelligent”) Performing Well? Check if System 1 can Handle, remove System 2 stimuli 48 Not Performing Well? Hell/Slow ban Thanks! Q & A?