Is it hard to build a service for 100M user? Short answer – yes /dev/viber What would you do if you had 1,000,000 users after 3 days? 1. Run away 2. Get drunk 3. Pray 4. All of the above /dev/viber Some numbers… • 100M registered users • 2.5B text, 1.7B voice minutes monthly • 50 employees in Belarus, about 120 worldwide /dev/viber What do we do? • Free calls, messages and group messages (more coming!) • Your number is your “user id” • Always On… /dev/viber Why Viber became popular • People love to get something for free • First one to do it “right” • It’s purple :) /dev/viber Reception • 1M after 3 days • Very high growth ratio • Give us more (platforms, features) – “Where is the Blackberry version?” /dev/viber What we saw as a result? PROBLEMS!!! /dev/viber Building a service that can handle 100M users in 1.5 years is difficult. Really. /dev/viber Why problems? • More usage - more load on backend • More platforms: – More development – Can’t deliver all features on all platforms at the same time. /dev/viber Scaling “Scaling is replacing all components of a car while driving it at 100MPH” [Mike Krieger, Instagram] /dev/viber When developers say “fully scalable” they don’t really mean it… /dev/viber Backend • Got to 1M connected users in less than 2 weeks • Used Amazon EC2, but still you have to design for scalability • Started from 30 servers (and we thought it was a LOT) • So we started to rewrite /dev/viber Why is it so hard? • • • • • • Address books (100M) Reverse address entries (3B) 300,000 data requests per second Active dataset 1.5TB 15-20M concurrent connected users (TCP) Over 100,000 concurrent phone calls /dev/viber How to handle all the data • • • • NoSQL (MongoDB, Redis)! Sharding MongoDB uses auto-sharding but it’s slow Simple in-memory solutions require lot of work /dev/viber We are our biggest enemy • A small mistake on a mobile client can lead to a massive DDOS • One bug in Android client = 100 times load on backend /dev/viber Say goodbye to life • • • • You are always in the office Sometime the office is at home Sometime at the beach Sometime... /dev/viber Shit happens • Stuff will break • Stay cool and fix • Communicate with your users /dev/viber Monitoring • • • • Collect tons of stats Look at them Look at them again Trending matters. Things don’t just happen. /dev/viber Backend now • Constantly rewriting and optimizing • Runs on 300 servers (and we think it's not too much) • Using and improving open source software /dev/viber Backend - technologies • • • • • Proprietary front-end servers (C++, really fast) Redis – in memory NoSQL, fast MongoDB – persistent NoSQL, not so fast MySQL – registrations, pretty slow Always looking for alternatives (CouchBase, DynamoDB, etc) /dev/viber Platforms • Users wants to communicate with their friends, but they keep buying different phones • Started from iOS and had Android community bashing us for a year • Now: iOS, Android, Blackberry, WindowsPhone 7 • More coming… /dev/viber Client challenges - iOS • Support numerous iOS versions (we’re finally dropping support for iOS 3) • Support several device type (3G, 3GS, 4, 4S) • Need to react very fast (iOS upgrade)! • Hard to test on large scale (no beta testing, although there is a one-time solution) /dev/viber iOS Device Breakdown 3.60% 7.16% 6% 36.60% 11.57% iPhone 4S iPhone 4 iPhone 3GS iPhone 3G 34.77% iPod iPad 6 users registered with iPhone 5 /dev/viber iOS versions breakdown 1.10% 13.90% iOS3 iOS4 84.80% iOS5 /dev/viber Mobile upgrade speed Desktop application upgrade Mobile application upgrade /dev/viber Client challenges - Android • Devices/platform fragmentation (we saw more then ??? devices) • Lot of sound-related bugs caused by it • Different screens and device caps • APIs not always work as documented and not always documented /dev/viber Client challenges - voice & video infrastructure • • • • • Have to choose/write a proper engine Now we are using and modifying WebRTC Has to work smoothly on mobile devices 3G is a pain Has to optimize for several platforms /dev/viber Lessons learned • • • • Always plan for success Monitor your system Hire best people It’s VERY REWARDING! [email protected] /dev/viber Questions?