Building Reliable, Secure and Manageable Substation Communications Dragan Dokic | CCIE, CISSP, MCSE Introduction - Experience • Dragan Dokic | President, Summit Energy Tech • Focus on utility sector – Infrastructure systems management – Custom business systems software development • 16 years of experience in IT industry • 10 years in utility sector – Managed network operations for PNGC Power [Portland, OR] from September 2002 to October 2011 – Presentation focuses on lessons learned in field network reliability, security and manageability from this experience Introduction • PNGC’s 2001 – 2011 field network – 92 office, substation and repeater sites at 11 distribution utilities in Oregon, Idaho • System mission – Gather real-time load data 24/7 for power scheduling operation in Portland – Support local utility SCADA/AMI/Site Security operations PNGC Power WAN – July 2011 Toledo, OR Boardman, Oregon Junction City, Oregon Lewiston, ID Malta, ID The Moon Areas of Focus Reliability Security Manageability Presentation available for download at in the Events section Reliability – Network Design • Keys to success – Diversity in media • Combine land lines, fixed wireless [private/public], mobile wireless and satellite – Diversity in providers • Local and national – Dynamic Routing [OSPF] • Routers exchange knowledge of local network with neighboring routers • Enterprise grade routers / switches a requirement • Perfect world configuration – Private wired/wireless ‘island’ with two Internet gateways using distinct media and distinct providers Connectivity overview Backup router Primary router Link cost overview Backup Primary Link cost calculation Sub A -> Main Office via Satellite tunnel: 3+1=4 Link cost calculation Sub A -> Main Office via 900Mhz+DSL tunnel: 1+1+1=3 Open Shortest Path Link cost via Satellite tunnel  higher than via DSL tunnel; therefore, packets will traverse 900Mhz/DSL tunnel in normal operation Normal Operation Open Shortest Path From substation A to Main Office Normal Operation Open Shortest Path From substation B to Main Office Link down operation If DSL tunnel is down, packets will traverse satellite tunnel; Sub A Main Office X Link down operation If DSL tunnel is down, packets will traverse satellite tunnel; Sub B Main Office X Questions? Security – Overview • Wireless link encryption • Function specific VLANs • No default routes! Wireless Link Encryption • Media device level [e.g. Radio, Modem] – WEP, WPA, WPA2 • Routing device level [e.g. Cisco 891 router] – IPSEC • End device level [e.g. DIGI TS4 port server] – SSL At what level to secure data? Security - Wireless Link Encryption [continued] • Most secure option? – Use all three if management overhead is not an issue • Most efficient but secure enough option? – Use routing device site-to-site VPN capabilities – Advantages: • Support for best commercially available security technologies [e.g., AES-256] • Comprehensive change logging capabilities • Standardized configuration throughout the system [less management overhead] Security – Function Specific VLANs • Define VLAN’s per business function – SCADA, AMI, Security System, Wireless, VOIP, Network Mgmt. • Firewall traffic between VLANs on need-to-access basis – E.g., Prevent personnel attached to substation wireless VLAN to access documentation stored on a server at the main office from accessing recloser controls in the SCADA VLAN • Reliability advantages – Non-critical VLANs [e.g. AMI, security] can be shut down automatically/remotely if link quality is too poor to carry all traffic, but good enough to carry SCADA One VLAN per business function High-speed link outage scenario Security – No Default Route! • Do not use default routes through service providersupplied gateways • Define a single host route back to the main office, then establish default route through VPN tunnel • This is the most effective method to prevent attacks sourced from the Internet • Always use in conjunction to regular firewall configuration lists [not a substitute!] Less secure Provider gateway More secure Provider gateway Questions? Manageability - Overview • Tools – network management systems • Addressing – developing a scheme • Watchdog system – preventing lockout Manageability – Tools • Network Management Systems [NMS] • Protocols used • SNMP, Syslog, ICMP, HTTP • Applications • PRTG • Solarwinds Syslog Manageability – Tools [continued] • How to collect data? Push vs. Pull – Pull: Poll devices using SNMP/HTTP/ICMP at regular intervals [e.g., every – Push: Devices send data per defined event triggers – SNMP traps – Syslog messages • What data to collect? – – – – Availability [ping] Network utilization Input voltages RSSI [radio link quality] Manageability – Tools [continued] • Pull example: – 5 minute SNMP poll of UPS for input voltage – If voltage drops below threshold of 108VAC for a duration of time longer than 5 minutes, an alert will be triggered by NMS [e-mail, text message, event log] – But what if voltage drops for 2 minutes only in between polls? You may not know it even happened. • Push comes to rescue: – UPS sends SNMP trap to NMS as soon as voltage drops below 108VAC – Alert is triggered by NMS when trap is received Paessler PRTG – Screen shot Solarwinds Kiwi Syslog – Screen shot Manageability – Addressing • Develop consistent scheme to use system wide • Recommended private range: 10.0.0.0/8 – – – – First octet: same for entire system Second octet: site ID [e.g. 8=Springfield Sub] Third octet: business function ID [e.g., 4=AMI] Fourth octet: device itself [e.g., Collector #1] Subnet Mask [255.255.255.0] 1st octet ‘fixed’ 2nd octet = site ID 4th octet = device 3rd octet = vlan/business function Manageability – Addressing [continued] • Large network? – Group sites by region using second octet – Allows for address summarization if needed. • Example: – Eastern division region: • 10.64-127.0.0 • Summary address: 10.64.0.0/10 – Western division region: • 10.128-191.0.0 • Summary address: 10.128.0.0/10 Manageability – Watchdog System • General concept – Reboot key remote communications devices if connectivity to central site is interrupted • Benefit – Prevent unnecessary site visits due to • Operator error • Device lock-up [e.g., buggy firmware, heat issues] Manageability – Watchdog System [continued] • Hardware requirements: – SNMP-capable switched PDU with task scheduling and delayed power cycling command capabilities – Example: APC AP7900 8-port 15A PDU • Software capability requirements: – Centralized command override mechanism using NMS – Send SNMP ‘Set’ to cancel pending power cycling command Manageability – Watchdog System Example • ‘Delayed’ power cycle schedule is defined on PDU: – Outlets to power cycle: – Frequency: – Command execute delay: 1,2 [e.g., radio, router] 60 minutes 30 minutes • Network management system running at main office sends an SNMP delayed power-cycle command cancel message – Frequency: every 5 minutes • Process – If delayed power cycle cancel command cannot reach the PDU at least one time during the 30 minute reboot delay period, outlets 1 and 2 will be power cycled and communication will (hopefully!) be restored Questions? Thank you!