Huawei Big Data Storage N9000 Jan Stibor Storage & Server Product Manager 1 Definition of Big Data 2 HUAWEI Big Data Storage Solution 3 Application Scenarios of HUAWEI Solution 2 Era of Big Data Is Coming 2.9 million emails per second globally 250 million pictures per day Videos of 28,800 hours per day 24 PB data processed per day 200 PB per safe city 6.3 million orders per day 50 million messages per day More than 1 PB per 3D movie By 2015, data amount will reach 8 ZB with the valued of $16.9 billion. Page 3 Page 3 What Is Big Data? Mass data Exponential growth of unstructured data Quick data flow and dynamic data system Real-time analysis, not post analysis Various data types Unstructured data such as files, emails, and videos accounting for 90% of the data amount generated in the next 10 years Veracity of Great data Prediction and analysis of the future and the behavior mode IDC: four characteristics of big data Page 4 Page 4 Driving Force of Big Data 1 • Collection Storage — open architecture, gradual scale out, and integration of near-line systems and offline systems 2 • Storage Big Data Analysis — compound data, unified storage of multiple data sources, unified query, and sharing of storage resources 3 • Management Archiving — lifecycle management of mass data at a low cost and high energy efficiency 4 Analysis • Maintenance — easy to use Page 5 Page 5 Contents 1 Definition of Big Data 2 HUAWEI Big Data Storage Solution 3 Application Scenarios of HUAWEI Solution 6 CSS/CSE HUAWEI Big Data Storage Solution High-performance solution Analysis Large-capacity solution Big Data Storage Archiving Mass data query and analysis solution Page 7 Page 7 HUAWEI OceanStor N9000—X in 1 •Quick data import •Support for JDBC and ODBC •SQL 92/2003 •Data compression Scale-out NAS •Linear expansion from 3 to 288 nodes •More than 3,000,000 OPS •Capacity up to 100 PB •NFS/CIFS interfaces •HDFS interface Scale-out DB Scale-out Backup Page 8 •Global online deduplication •Support for LTFS Page 8 HUAWEI OceanStor N9000 Architecture Application layer Media asset Internet HPC … Behavior prediction, real-time analysis, … …… Unstructured data, object data, structured data NAS interface Object interface Big data sharing Archiving interface BI Database interface Big data analysis Map-reduce HDFS Unified management and namespace Lifecycle policy Index of quick file search Remote replication and Global deduplication snapshot Storage Analysis Storage Archiving 10GE/Infiniband layer Node Node Node Node Page 9 Periodic data inspection Node Node Page 9 N9000 Hardware Front view High-performance node： The scene of frequent Rear view Front view Rear view Front view Rear view read and write small files High-bandwidth node： The scene of large file sequential read and write Large-capacity node： Near-line storage Acceleration node Page 10 2 U high 25 disk slots 2-channel hexa-core CPU 48 GB memory at least SSDs for storing metadata 4 U high 36 disk slots 2-channel hexa-core CPU 48 GB memory at least 4 U high 36 disk slots 2-channel quad-core CPU 16 GB memory at least 1 U high Page 10 N9000 Software — Wise Series Self-developed distributed file system Mandatory, one-off purchase Wise Data Protection (WiseDP) 2 WiseLink Cluster load balancing software Support for multiple load balancing policies Optional, allowing for separate purchase 3 WiseQuota Quota management software that sets space usage limits on directories or users. Optional, allowing for separate purchase DST software Support for policy-based automatic migration Optional, allowing for separate purchase 1 WushanFS WushanFS WushanFS: Wushan File System 4 WiseTier Page 11 Page 11 Extensive Flexibility: WiseDP +1 Number ODC RDC of Nodes 3 4 4 6 5 8 6 4 8 6 10 8 14 12 14 10 20 12 +2 ECC ECC Switch +3 +4 ECC ECC RDN 2 2 2 2 2 2 2 4 4 Max. No. of Max. No. of Utilization Allowed Faulty Allowed Faulty Disks Nodes 1 2 1 67% 1 2 1 75% 1 2 1 80% 2 2 2 67% 2 2 2 75% 2 2 2 80% 2 2 2 86% 4 4 4 71% 4 4 4 75% Original data counts (ODC) Redundant data counts (RDC) Redundant data nodes (RDN) N+1 to N+4 data protection File-level data protection 1 hr/TB data recovery Page 12 Page 12 Outstanding Performance: WiseLink ... Automatic load balancing among existent and newly added nodes Single node expansion within 60s Without human intervention and manual modifications Page 13 Page 13 Outstanding Performance: WiseTier • Enables DST to work among diverse types of nodes. • Adopts tiering policies based on I/O access popularity. High-OPS nodes • Adopts hotspot anti-shake policy to prevent unnecessary data migration. • Seamlessly adapts to service changes without interrupting ongoing applications. Large-capacity nodes Hot data Cold data • Consolidates multiple storage systems into a single storage system, eliminating the possibilities of frequent data migration and simplifying management. Page 14 Page 14 Simplified Management: Refined Space Management WiseQuota: sets quotas for subdirectories to facilitate space management. 100 TB Fast automatic deployment Comprehensive performance monitoring Unified topology display 10 TB 20 TB User User 40 TB 30 TB User group User group 20 TB 10 TB User User Centralized management of a single file system of 100 PB Page 15 Page 15 HUAWEI 9000 — Adapting to Changes and Converging for Easy Management Extensive Flexibility Scaling from 3 to 288 nodes Support up to 100 PB of capacity Linear scaling of performance and capacity Outstanding Performance World's leading file system access performance of more than 3 million OPS Optimal Integration Distributed scale-out architecture Integration of scale-out NAS, scale-out DB, and scale-out backup Page 16 Streamlined Management Unified management of all nodes on a single interface LCM Page 16 Contents 1 Definition of Big Data 2 HUAWEI Big Data Storage Solution 3 Application Scenarios of HUAWEI Solution 17 HUAWEI OceanStor N9000 Scenario— Media Asset Management Animation rendering Non-linear editing system … … … HD editing SD editing HPC VOD system LAN a strong need for data sharing N9000 highbandwidth nodes and large-capacity nodes VOD server cluster PC Stable bit steams and low single-stream bandwidth, with Animation rendering solution N9000 highperformance nodes TV set + set-top box TV station non-linear editing system solution • ... Cataloging Focusing on five aspects • Frame image file storage, with a strong need for high performance in sequential read of small-sized I/Os VOD solution • Concurrent reads by multiple users, with a strong need for high bandwidth PC Smart phone Media asset management solution Post-production system Audio Editing Effect Media asset management system Query/Cataloging • Large capacity, high bandwidth, and data sharing Movie post-production solution … • Stable bid streams and high-bandwidth throughput, providing data sharing Page 18 Page 18 HUAWEI ENTERPRISE ICT SOLUTIONS A BETTER WAY Copyright©2012 Huawei Technologies Co., Ltd. All Rights Reserved. The information in this document may contain predictive statements including, without limitation, statements regarding the future financial and operating results, future product portfolio, new technology, etc. There are a number of factors that could cause actual results and developments to differ materially from those expressed or implied in the predictive statements. Therefore, such information is provided for reference purpose only and constitutes neither an offer nor an acceptance. Huawei may change the information at any time without notice.