NCAR’s Data-Centric Supercomputing Environment Yellowstone December 21, 2011 Anke Kamrath, OSD Director/CISL [email protected] Overview • Strategy – Moving from Process to Data-Centric Computing • HPC/Data Architecture – What we have today at ML – What’s planned for NWSC • Storage/Data/Networking – Data in Flight – WAN and High-Performance LAN – Central Filesystem Plans – Archival Plans 2 Evolving the Scientific Workflow • Common data movement issues – Time consuming to move data between systems – Bandwidth to archive system is insufficient – Lack of sufficient disk space • Need to evolve data management techniques – Workflow management systems – Standardize metadata information – Reduce/eliminate duplication of datasets (ie – CMIP5) • User Education – Effective methods for understanding workflow – Effective methods for streamlining workflow Traditional Workflow Process Centric Data Model Evolving Scientific Workflow Information Centric Data Model Current Resources @ Mesa Lab GLADE BLUEFIRE LYNX NWSC: Yellowstone Environment Geyser & Caldera Yellowstone GLADE HPC resource, 1.55 PFLOPS peak Central disk resource 11 PB (2012), 16.4 PB (2014) DAV clusters High Bandwidth Low Latency HPC and I/O Networks FDR InfiniBand and 10Gb Ethernet NCAR HPSS Archive 100 PB capacity ~15 PB/yr growth 1Gb/10Gb Ethernet (40Gb+ future) Science Gateways RDA, ESG Data Transfer Services Remote Vis Partner Sites XSEDE Sites NWSC-1 Resources in a Nutshell • Centralized Filesystems and Data Storage Resource (GLADE) – >90 GB/sec aggregate I/O bandwidth, GPFS filesystems – 10.9 PetaBytes initially -> 16.4 PetaBytes in 1Q2014 • High Performance Computing Resource (Yellowstone) – IBM iDataPlex Cluster with Intel Sandy Bridge EP† processors with Advanced Vector Extensions (AVX) – 1.552 PetaFLOPs – 29.8 bluefire-equivalents – 4,662 nodes – 74,592 cores – 149.2 TeraBytes total memory – Mellanox FDR InfiniBand full fat-tree interconnect • Data Analysis and Visualization Resource (Geyser & Caldera) – Large Memory System with Intel Westmere EX processors • 16 nodes, 640 cores, 16 TeraBytes memory, 16 NVIDIA Kepler GPUs – GPU-Computation/Vis System with Intel Sandy Bridge EP processors with AVX • 16 nodes, 128 SB cores, 1 TeraByte memory, 32 NVIDIA Kepler GPUs – Knights Corner System with Intel Sandy Bridge EP processors with AVX • 16 nodes, 128 SB cores, 992 KC cores, 1 TeraByte memory - Nov’12 delivery † “Sandy Bridge EP” is the Intel® Xeon® E5-2670 GLADE • 10.94 PB usable capacity 16.42 PB usable (1Q2014) Estimated initial file system sizes – collections ≈ 2 PB RDA, CMIP5 data – scratch ≈ 5 PB shared, temporary space – projects ≈ 3 PB long-term, allocated space – users ≈ 1 PB medium-term work space • Disk Storage Subsystem – 76 IBM DCS3700 controllers & expansion drawers • 90 2-TB NL-SAS drives/controller • add 30 3-TB NL-SAS drives/controller (1Q2014) • GPFS NSD Servers – 91.8 GB/s aggregate I/O bandwidth; 19 IBM x3650 M4 nodes • I/O Aggregator Servers (GPFS, GLADE-HPSS connectivity) – 10-GbE & FDR interfaces; 4 IBM x3650 M4 nodes • High-performance I/O interconnect to HPC & DAV – Mellanox FDR InfiniBand full fat-tree – 13.6 GB/s bidirectional bandwidth/node NCAR Disk Storage Capacity Profile Total Centralized Filesystem Storage (PB) GLADE (NWSC) GLADE (ML) bluefire 18 16 Total Usable Storage (PB) 14 12 10 GLADE (at NWSC) 8 6 4 2 GLADE (Mesa Lab) 0 Jan-10 Jan-11 Jan-12 Jan-13 Jan-14 Jan-15 Jan-16 Yellowstone NWSC High-Performance Computing Resource • Batch Computation – – – – – 4,662 IBM dx360 M4 nodes – 16 cores, 32 GB memory per node Intel Sandy Bridge EP processors with AVX – 2.6 GHz clock 74,592 cores total – 1.552 PFLOPs peak 149.2 TB total DDR3-1600 memory 29.8 Bluefire equivalents • High-Performance Interconnect – – – – Mellanox FDR InfiniBand full fat-tree 13.6 GB/s bidirectional bw/node <2.5 µs latency (worst case) 31.7 TB/s bisection bandwidth • Login/Interactive – 6 IBM x3650 M4 Nodes; Intel Sandy Bridge EP processors with AVX – 16 cores & 128 GB memory per node NCAR HPC Profile Peak PFLOPs at NCAR IBM iDataPlex/FDRIB (yellowstone) 30x Bluefire performance 1.5 yellowstone Cray XT5m (lynx) IBM Power 575/32 (128) POWER6/DDR-IB (bluefire) 1.0 IBM p575/16 (112) POWER5+/HPS (blueice) IBM p575/8 (78) POWER5/HPS (bluevista) 0.5 IBM BlueGene/L (frost) bluesky frost bluevista blueice IBM POWER4/Colony (bluesky) lynx bluefire frost upgrade 0.0 Jan-04 Jan-05 Jan-06 Jan-07 Jan-08 Jan-09 Jan-10 Jan-11 Jan-12 Jan-13 Jan-14 Jan-15 Jan-16 Geyser and Caldera NWSC Data Analysis & Visualization Resource • Geyser: Large-memory system – 16 IBM x3850 nodes – Intel Westmere-EX processors – 40 cores, 1 TB memory, 1 NVIDIA Kepler Q13H-3 GPU per node – Mellanox FDR full fat-tree interconnect • Caldera: GPU computation/visualization system – 16 IBM x360 M4 nodes – Intel Sandy Bridge EP/AVX – 16 cores, 64 GB memory, 2 NVIDIA Kepler Q13H-3 GPUs per node – Mellanox FDR full fat-tree interconnect • Knights Corner system (November 2012 delivery) – Intel Many Integrated Core (MIC) architecture – 16 IBM Knights Corner nodes – 16 Sandy Bridge EP/AVX cores, 64 GB memory, 1 Knights Corner adapter per node – Mellanox FDR full fat-tree interconnect Erebus Antarctic Mesoscale Prediction System (AMPS) 0° • IBM iDataPlex Compute Cluster – – – – – 84 IBM dx360 M4 Nodes; 16 cores, 32 GB Intel Sandy Bridge EP; 2.6 GHz clock 1,344 cores total – 28 TFLOPs peak Mellanox FDR InfiniBand full fat-tree 0.54 Bluefire equivalents • Login Nodes – 2 IBM x3650 M4 Nodes – 16 cores & 128 GB memory per node • Dedicated GPFS filesystem – 57.6 TB usable disk storage – 9.6 GB/sec aggregate I/O bandwidth 90° E 90° W 180° Erebus, on Ross Island, is Antarctica’s most famous volcanic peak and is one of the largest volcanoes in the world – within the top 20 in total size and reaching a height of 12,450 feet. Yellowstone Software • Compilers, Libraries, Debugger & Performance Tools – Intel Cluster Studio (Fortran, C++, performance & MPI libraries, trace collector & analyzer) 50 concurrent users – Intel VTune Amplifier XE performance optimizer 2 concurrent users – PGI CDK (Fortran, C, C++, pgdbg debugger, pgprof) 50 conc. users – PGI CDK GPU Version (Fortran, C, C++, pgdbg debugger, pgprof) for DAV systems only, 2 concurrent users – PathScale EckoPath (Fortran C, C++, PathDB debugger) 20 concurrent users – Rogue Wave TotalView debugger 8,192 floating tokens – IBM Parallel Environment (POE), including IBM HPC Toolkit • System Software – LSF-HPC Batch Subsystem / Resource Manager • IBM has purchased Platform Computing, Inc. (developers of LSF-HPC) – – – – Red Hat Enterprise Linux (RHEL) Version 6 IBM General Parallel Filesystem (GPFS) Mellanox Universal Fabric Manager IBM xCAT cluster administration toolkit NCAR HPSS Archive Resource • NWSC – Two SL8500 robotic libraries (20k cartridge capacity) – 26 T10000C tape drives (240 MB/sec I/O rate each) and T10000C media (5 TB/cartridge, uncompressed) initially; +20 T10000C drives ~Nov 2012 – >100 PB capacity – Current growth rate ~3.8 PB/year – Anticipated NWSC growth rate ~15 PB/year • Mesa Lab – Two SL8500 robotic libraries (15k cartridge capacity) – Existing data (14.5 PB): • 1st & 2nd copies will be ‘oozed’ to new media @ NWSC, begin 2012 – New data @ Mesa: • Disaster-recovery data only – T10000B drives & media to be retired – No plans to move Mesa Lab SL8500 libraries (more costly to move than to buy new under AMSTAR Subcontract) Plan to release an “AMSTAR-2” RFP 1Q2013, with target for first equipment delivery during 1Q2014 to further augment the NCAR HPSS Archive. Yellowstone Physical Infrastructure Resource # Racks Yellowstone 65 - iDataPlex Racks (72 nodes per rack) 9 - 19” Racks (9 Mellanox FDR core switches) 1 - 19” Rack (login, service, management nodes) GLADE 20 - NSD Server, Controller and Storage Racks 1 - 19” Rack (I/O aggregator nodes, management , IB & Ethernet switches) Geyser & Caldera 1 - iDataPlex Rack (GPU-Comp & Knights Corner) 2 - 19” Racks (Large Memory, management , IB switch) Erebus (AMPS) 1 - iDataPlex Rack 1 - 19” Rack (login, IB, NSD, disk & management nodes) Total Power Required ~2.13 MW Yellowstone ~1.9 MW GLADE 0.134 MW Geyser & Caldera 0.056 MW Erebus (AMPS) 0.087 MW Yellowstone allocations (% of resource) NCAR’s 29% represents 170 million core-hours per year for Yellowstone alone (compared to less than 10 million per year on Bluefire) plus a similar fraction of the DAV and GLADE resources. Yellowstone Schedule Current Schedule Production Science ASD & early users Acceptance Testing 15 Oct 1 Oct 17 Sep 3 Sep 20 Aug 6 Aug 23 Jul 9 Jul 25 Jun 11 Jun 28 May 14 May 30 Apr 16 Apr 2 Apr 19 Mar 5 Mar 20 Feb 6 Feb 23 Jan 9 Jan 26 Dec 12 Dec 28 Nov 14 Nov 31 Oct Integration & Checkout Production Systems Delivery Test Systems Delivery & Installation Storage & InfiniBand Delivery & Installation Infrastructure Preparation Data in Flight to NWSC • NWSC Networking • Central Filesystem – Migrating from GLADE-ML to GLADE-NWSC • Archive – Migrating HPSS data from ML to NWSC 20 a 21 BiSON to NWSC Initially three 10G circuits active Two 10G connections back to Mesa Lab for internal traffic One 10G direct to FRGP for general Internet2 / NLR / Research and Education traffic Options for dedicated 10G connections for high performance computing to other BiSON members System is engineered for 40 individual lambdas Each lambda can be a 10G, 40G, or 100G connection Independent lambdas can be sent each direction around the ring (two ADVA shelves at NWSC – one for each direction) With a major upgrade system could support 80 lambdas 100Gbps * 80 channels * 2 paths = 16Tbps 22 High performance LAN Data Center Networking (DCN) high speed data center computing with 1G and 10G client facing ports for supercomputer, mass storage, and other data center components redundant design (e.g., multiple chassis and separate module connections) future option for 100G interfaces Juniper awarded RFP after NSF approval Access switches: QFX3500 series Network core and WAN edge: EX8200 series switch/routers Includes spares for lab/testing purposes Juniper training for NETS staff early Jan-2012 Deploy Juniper equipment late Jan-2012 23 Moving Data… Ugh 24 Migrating GLADE Data • Temporary work spaces (/glade/scratch, /glade/user) – No data will automatically be moved to NWSC • Allocated project spaces (/glade/projxx) – New allocations will be made for the NWSC – No data will automatically be moved to NWSC – Data transfer option so users may move data they need • Data collections (/glade/data01, /glade/data02) – CISL will move data from ML to NWSC – Full production capability will need to be maintained during the transition Network Impact - Current storage max performance is 5GB/s - Can sustain ~2GB/s for reads while under a production load - Will move 400TB in a couple of days, however we will saturate a 20Gb/s network link 25 Migrating Archive • What Migrates???? • • • • Data: 15PBs and counting… Format: MSS to HPSS Tape: Tape B (1TB tapes) to Tape C (5TB tapes) Location: ML to NWSC • HPSS at NWSC to become primary site in Spring 2012 • • • • 1 day outage when metadata servers get moved ML HPSS will remain as Disaster Recovery Site Data Migration will take until early 2014 Will throttle migration to not overload network 26 WHEW… A LOT OF WORK AHEAD! QUESTIONS?