Network & Services Overview June 2012 Jeff Ambern [email protected] Agenda • • • • Indiana GigaPOP Overview Services Commodity Internet Usage Trends Monon100 (100Gbps to Chicago) Indiana GigaPOP • Established in Indianapolis in 1998 • Partnership between Indiana University and Purdue University • Advanced high-speed, high-availability, feature-rich network • Drives down the costs and increases the connection speeds for Indiana’s top research colleges and universities • Serves as an aggregation point for Indiana’s universities to access (regional, national and international) R&E and Commodity Internet. GigaPOP Participants • • • • • Indiana University Purdue University I-Light Notre Dame RLHEC • • • • SLIAC ENA CSPAN Archives NCAA GigaPOP Providers & Peers Providers (Commodity) • Cogent • TimeWarner • Iquest (Local only) • Smithville (Local only) • Wintek (Local only) • Internet2 TRCPS • WiscNet RPS • Akamai (Cache) • Google (Cache) Peers (R&E) • Internet2 • National LambdaRail • MREN • CIC • ESnet Network Elements • Juniper MX960 (Core Nodes) • Fully redundant chassis hardware (cooling, power supplies, Routing Engines, SCB’s) • • • • HP 5400 (Layer2 Backbone Switches) Brocade MLXe (Layer2 Backbone Switch) HP 3500 (Management Switches for OOB) Mix of Cisco Routers (Terminal Servers, VPN, Mgt.) Current Topology GigaPOP Capabilities • Layer3 • • • • R&E Access (National, Regional and Local) Commodity Internet Access L3VPN/MPLS Multicast • Layer2 • L2VPN/MPLS • VLANs • Layer1/2 • 100G Ethernet (Fiber) • 10G Ethernet (Fiber) • 1G Ethernet (Fiber, Copper) Access to Research Networks R&E (Base Service) • National/International/Regional Networks (Internet2 (100G), NLR,CIC, MREN, Esnet) • Local R&E (Gigapop Participants, State Universities) R&E Service includes IPv4 plus: • IPv6 (we can provide IPv6 address space to members) • Multicast • Jumbo Frames (9000 bytes) Commodity Internet (Optional to Participants) • • • Full Transit • Complete Internet Routing Table • ~410K Routes Settlement-Free Internet • Partial Internet Routing Table • ~240K Routes • Commercial Entities agree to peering arrangements with I2 (TRCPS) or other GigaPOPs to save on upstream BW charges Internet Caching Servers • Similar to a Settlement-Free peering arrangement but involving only one entity (i.e. Google, Akamai, Netflix (future peering)) • Server Farm and cached data is local to customer • Does not consume Internet Transit bandwidth for cached data (except during incremental updates) Caching Services Why is caching beneficial to Participants? • Offloading commercial traffic to local servers reduces our costs that we would have to pay to our per Mbps transit providers • Reduces the amount of Commodity traffic that would congest our upstream links and force us into costly upgrades (I2 TRCPS, Cogent and TWTC) • Reducing our costs allows us to pass savings on to our Participants Caching Services Akamai server farm added late 2009 • 1Gbps 95% • 500Mbps daily average we would have to pay to our upstream providers or consume BW on our settlement-free links Google Caching Services added 12/2011 • 1.3Gbps 95% • 730Mbps average Netflix Caching Services / Peering - Late 2012 • Currently working with Netflix on traffic analysis • Depending on usage will either Peer at IXP or install cache node in Indpls. Commodity Usage - Gbps • Cogent (10G) & TimeWarner (10G) • Full Internet Transit • Fully Redundant Upstreams • Full IPv4 & IPv6 Transit • Internet2 TRCPS Commodity Peering Service (10G) • Reduces cost through aggregation and settlement-free peering • WisNet Regional Peering Service (10G) – Via our CIC membership • Akamai Caching Service (10G) • Google Caching Service (10G) TimeWarner 1.28 1.82 Cogent 1 2.67 Wiscnet Regional Peering Service I2 Transitrail CPS 4.09 Akamai 3.03 Google Bandwidth Trends Commodity Internet Usage Trends (Aggregate Upstream) • 40-70% increase in usage per year 2009-2012 • • • • Sept 2009 – 2.5G Avg (Cost/Gbps $10000) – 25K/Month Sept 2010 – 6.5G Avg (Cost/Gbps $6000) Sept 2011 - 9.5G Avg (Cost/Gbps $5000) July 2012 – 13.0G Avg (New Pricing $1750/Gbps) – 22.27K/Month • Able to keep member cost down due to decreases in provider pricing and Settlement-Free Peers Aggregate Commodity (Gbps) $/Gbps 14 12000 12 10000 10 8000 8 6000 6 4000 4 2000 2 0 0 2009 2010 2011 2012 2009 2010 2011 2012 Commodity Bandwidth 20092012 24hr Avgs Monon100 Monon100 • Monon100 is named after the Monon Rail line that connected Indiana's higher education institutions to the rest of the world through Chicago • The Monon Rail served six colleges and universities along its line (from Chicago to Louisville): o o o o o o Purdue University in West Lafayette, Indiana Wabash College in Crawfordsville, Indiana DePauw University in Greencastle, Indiana Indiana University in Bloomington, Indiana Butler University in Indianapolis, Indiana St. Joseph's College in Rensselaer, Indiana • Links the Indiana GigaPoP to Internet2, NLR, CIC, MREN and other networks. • 10 times faster than our previous network path to Chicago • Resource available to all Indiana GigaPoP members Gigapop - Internet2 Update • Connected at 100G to Internet2 in Chicago • Upgrading optical nodes between Indianapolis and Chicago to support 100G • Temporary Internet2 100G Wave until Optical upgrades completed Optical Transition • Phase 1 o Rebuild IPGrid Optical System to be Monon100 Optical System • Phase 2 o Transition services from temp I2 DWDM to Monon100 Optical System • Indiana o Future growth will include 100GE to IUB via I-Light • CIC o Future growth will include 100GE to CIC in Chicago Monon100 – Core Upgrades • Installed new Brocade MLXe 16 slot switch in Chicago • Core Juniper routers have been upgraded to 12.1R1.9 o o o o o Power supplies upgraded to 4100W Enhanced MX SCB (3 per router) Added High Capacity Fan Trays 100G Cards online (3 at ICTC and 1 at LL) MPC Type 3 - Modular Port Concentrator CFP-100G-LR4 Monon100/Indiana GigaPOP Ilight to Chicgo I-light access to Chicago Questions?