CLOUD COMPUTING: HADOOP SECURITY DESIGN -2009 *ALL OPINIONS AND INFORMATION ARE MINE AND DO NOT REPRESENT THE VIEW(S) OF MY EMPLOYER Kaveh Noorbakhsh Kent State: CS Owen O’Malley | Kan Zhang | Sanjay Radia | Ram Marti | Christopher Harrell | Yahoo! Brief History: Cloud Computing as a Service 1961: 1969 •John McCarthy •ARAPANET Introduces Concept of Cloud Computing as a business model 1997 •“Cloud Computing” coined by Ramnath Chellappa 1999 2002 2004 2006 2008 2009 2011 •Saleforce.com •Amazon Web •HDFS & •Google Docs •Eucalyptus •MS Azure •Amazon RDS st Services Map/Reduce •Enterprise •Amazon EC2 •1 Open •Amazon RDS supports in Nutch Oracle Applications Source AWS •Yahoo hires •MySQL via simple API for •Office 365 Doug Cutting supported web interface Private Clouds •OpenNebula •Private and Hybrid clouds •Hadoop hits web scale Hadoop – Funny Name, Big Impact Map/Reduce An Introduction Map/Reduce allows computation to scale out over many “cheap” systems rather than one expensive super computer Divide and Conquer “Work” Partition w1 w2 w3 “worker” “worker” “worker” r1 r2 r3 “Result” Combine Two Layers MapReduce: Code runs here HDFS: Data lives here Advantages of the Cloud Database as a Service = DBaaS Infrastructure as a Service = Iaas Software as a Service = SaaS Platform as a Service = PaaS Share hardware and energy costs Share employee costs Fast spin-up and tear down Expand quickly to meet demands Costs ideally proportional to usage Scalability Cloud Services Spending Billions of Dollars 160 140 120 100 2009 2010 2014 80 60 40 20 0 Cloud Services Revenue Cloud vs Total IT Spending Billions of Dollars 4500 4000 3500 3000 2500 2009 2010 2014 2000 1500 1000 500 0 Cloud Services Revenue Total IT Expendetures Security Challenges of the Cloud Where is my data living? Where is my data going? You may not know where you data is exactly since the data can be distributed among many physical disks In the cloud, especially in map/reduce, data is constantly in moving from node to node and nodes may be across multiple mini-clouds Who has access to my data? There may be other clients using the cloud, as well as, administrators and others who maintained the cloud that could have access to the data if it is not properly protected. Hadoop Security Concerns Hadoop services do not authenticate users or other services. (a) A user can access an HDFS or MapReduce cluster as any other user. This makes it impossible to enforce access control in an uncooperative environment. For example, file permission checking on HDFS can be easily circumvented. (b) An attacker can masquerade as Hadoop services. For example, user code running on a MapReduce cluster can register itself as a new TaskTracker. DataNodes do not enforce any access control on accesses to its data blocks. This makes it possible for an unauthorized client to read a data block as long as she can supply its block ID. It’s also possible for anyone to write arbitrary data blocks to DataNodes. Security Requirements for Hadoop Users are only allowed to access HDFS files that they have permission to access. Users are only allowed to access or modify their own MapReduce jobs. User to service mutual authentication to prevent unauthorized NameN- odes, DataNodes, JobTrackers, or TaskTrackers. Service to service mutual authentication to prevent unauthorized services from joining a cluster’s HDFS or MapReduce service. The degradation of performance should be no more than 3%. 1. A ppl icat i ons accessing ﬁ les on H D F S cl ust er s Non-MapReduc plicat ions, including hadoop fs, access ﬁles st ored on one or more H clust ers. T he applicat ion should only be able t o access ﬁles and ser t hey are aut horized t o access. See ﬁgure 1. Variat ions: Proposed Solution – Use Case 1 (a) Access HDFS direct ly using HDFS prot ocol. Accessing Data (b) Access HDFS indirect ly t hough HDFS proxy servers via t he H FileSyst em or HT T P get . 1) User/App requests access to a data block. 2) Name Node authenticates and gives the user a block token. 3) User/App uses block token on Data Node to access block for READ, WRITE, COPY or REPLACE. (joe) b r e k Application Name Node delg(jo e ) MapReduce Task kerb(hdfs) bloc k toke n Data Node o ck t o l b ken Figure 1: HDFS High-level Dat aﬂow 2. A ppl icat i ons accessi ng t hi r d-par t y ( non-H ado op) ser v ices MapReduce applicat ions and MapReduce t asks accessing ﬁles or o t ions support ed by t hird party services. An applicat ion should on able t o access services t hey are aut horized t o access. Examples of t Proposed Solution – Use Case 2 2.2 High Level Use Cases 2 USE CAS Submitting Jobs 1) A user may obtain a delegation token through Kerberos. Application kerb(joe) Job Tracker kerb(mapreduce) 2) Token given to user jobs for subsequent authentication to NameNode as the user. 3) Jobs can use the delegation token to access data that user/app has access to Task Tracker job token Task HDFS HDFS HDFS ) (joe g l de other credential trus t Other Service NFS Figure 2: MapReduce High-level Dat aﬂow Core Principles Analysis Confidentiality Users/Apps will only have access to the data blocks they should have via block tokens Analysis Pass Core Principles Analysis Integrity Data is only available at the block level if the block token matches. There is an assumption that the data is good because the blocks are not checked Analysis Pass Fail Core Principles Analysis Availability Job Tracker and Name Nodes are single points of failure for system. Tokens persist for a small period of time so the system is resilient to short outages of Name Node and Job Tracker Analysis Fail Pass Conclusion The token method for authentication for both data and process access makes sense in a highly distributed system like hadoop. However, the fact that tokens have so much power and are not constantly re-checked leaves this design open to very serious TOCTOU attacks. As compared to the currently model(aka no security) this represents a major step forward. THE END Questions?