Data Integration Strategy Air Force Personnel Operations Agency – Human Resources December 2, 2011 Copyright © 2011 SAS Institute Inc. All rights reserved. Agenda - TOC Who is SAS? Where is SAS in the Air Force Essential Business Process Agility Data Integration and Information Management Data Quality: Cleansing and Enrichment Data Integration/Management Benefits Why SAS for AFPOA Resources and Further Contact Information 2 Copyright © 2011, SAS Institute Inc. All rights reserved. SAS – Leader in Applied Analytics Largest privately held software company in the world 2010 Revenue: $2.43 Billion – 35 Years of consecutive growth 24% reinvestment in R&D for 2010 50,000 deployments worldwide 12,000 employees 90 of the top 100 of Fortune Global 500 All 15 U.S. Departments and 90% of U.S. Federal Agencies All 50 U.S. State Governments Government agencies in 80 countries Outstanding Technical Support Services Life Sciences: 6% Manufacturing: 5.5% Retail: 4.5% Communications: 7% Education: 3% Services: 11% Energy & Utilities: 3% Other: 1% Government: 15% Financial Services: 42% Copyright © 2011, SAS Institute Inc. All rights reserved. 3 Where SAS is in the U.S. Air Force In a new framework for AF Human Resources, these SAS installations could be streamlined if needed or leveraged. Malmstrom AFB Hanscom AFB Offutt AFB Hill AFB Wright-Patterson AFB Langley AFB Chantillly, VA & Washington, D.C. Scott AFB Peterson AFB Nellis AFB Edwards AFB LA AFB Tinker AFB Kirtland AFB Huntsville Robins AFB Maxwell AFB-Gunter Barksdale AFB Brooks, Randolph Lackland AFBs Eglin AFB Hurlburt Field Patrick AFB 4 Copyright © 2011, SAS Institute Inc. All rights reserved. Essential for Business Process Agility Data Integration defined in terms of Information Data Access/Integration/Quality Data Governance for Airman Personnel PII Master Data Best Record Airman Personnel DATA INTEGRATION UNIFIED PLATFORM for Information Decision Management Support Performance and Cost Analytical Model Management and Governance Recalibrate or New DATA QUALITY DATA ACCESS ANALYTIC MGMT. MASTER DATA DECISION MGMT. 5 Copyright © 2011, SAS Institute Inc. All rights reserved. Data Integration and Information Management The Opportunity Data Diversity: Big Data Information Management Use Case Diversity Unified Data Management Role Diversity: IT/Business Acceleration is required to meet the current demands in the Business and Technical landscape Managing information for Air Force personnel who are the most valuable asset includes the ability to value information appropriately Deployment Diversity: Cloud Consumption Diversity: Mobility Architecture Diversity: Enterprise Architecture Temporal Diversity: Right-Time Application Diversity: Operational Analytics 6 Copyright © 2011, SAS Institute Inc. All rights reserved. Data Management and Integration Capability View Agile And Responsive To Changing Requirements Linking IT Strategy to Business Strategy DATA SERVICES DATA & ANALYTIC SERVICES DATA GOVERNANCE ANALYTICS GOVERNANCE Analytics Management Data Management DATA INTEGRATION DATA QUALITY MDM DECISION MANAGEMENT MODEL MANAGEMENT & MONITORING MODEL DEPLOYMENT & INTEGRATION INFRASTRUCTURE SUPPORT: Text & Unstructured Data Support, Events, Security, Meta-data & Lineage, Monitoring & Deployment Infrastructure Support ENTERPRISE DATA ACCESS 7 Copyright © 2011, SAS Institute Inc. All rights reserved. Data Management UNIFIED PLATFORM INFORMATION MANAGEMENT DataManagement Management Data Data Integration Data Quality Master Data Management Enterprise Data Access and Federation Resolves Architecture is rigid; not flexible to changes in business needs Managing end-to-end processes is challenging 8 Copyright © 2011, SAS Institute Inc. All rights reserved. Why SAS/DataFlux for Data Management SAS the Original Data Warehouse Pioneer UNIFIED PLATFORM INFORMATION MANAGEMENT DataFlux 35 years Data Management Experience Thousands of Data Management Customers worldwide End-to-end Solution Available Proven Market Leader in Data Management Recognized as a leading provider of enterprise data quality, data integration, data governance and MDM solutions Gartner Leader for DQ Gartner Leader for DI Gartner DI Leader Best-in-Class Data Access Technology Over 2,300 customers worldwide Shortest Time to delivery for MDM Scalable Grid Enabled Platform Only Data Management vendor with integrated analytic scoring Offices throughout the US, the UK, France, Germany and Australia World Class Partner Network Thousands of consultants worldwide Growth through consistent R&D Founded in 1997 Acquired by SAS in 2000 Operates as a wholly owned subsidiary 9 Copyright © 2011, SAS Institute Inc. All rights reserved. SAS’ Data Management Portfolio SAS Data Integration Server UNIFIED PLATFORM INFORMATION MANAGEMENT SAS Data Integration Server SAS Enterprise Integration Server SAS Data Quality Solution (Server minus DI Studio) DataFlux Data Management Platform Data Management Studio Data Management Server Data Federation Server qMDM – Master Data Management DataFlux Solution Accelerators DataFlux Event Stream Processor Supporting SAS Technologies SAS / Access » Interface to <insert favorite RDBMS > / Interface to PC Files SAS Integration Technologies » Message Queuing APIs, Web Services, Publishing Framework 10 Copyright © 2011, SAS Institute Inc. All rights reserved. SAS Data Management Advantage – Access? DATA ACCESS Any Data, Anywhere, Anytime: Best-in-Class Data Access Technology SAS/Access What is it? SAS/ACCESS Interfaces are out-of-the-box solutions that provide enterprise data access and integration between SAS and third-party databases. SAS/ACCESS interfaces are highly optimized to enable your SAS® solutions to read, write and update data regardless of its native database or platform. Because the data appears native to SAS, there is no need to learn Structured Query Language (SQL) or any other databasespecific query languages. All data is presented unified to users, complex activities like bulk loading are simplified, cross source integration is seamless. » SAS leverages the underlying RDBMS for you transparently » For example, a single option is used to invoke bulk loaders » DATA MyORA.ORACLE_TABLE(BULK_LOAD=YES) Copyright © 2011, SAS Institute Inc. All rights reserved. 11 SAS Data Management Advantage Any Data, Anywhere, Anytime: Best-in-Class Data Access Technology RDBMS DATA ACCESS Non-Relational SAS/ACCESS Interface to DB2 SAS/ACCESS Interface to ADABAS SAS/ACCESS Interface to Informix SAS/ACCESS Interface to DATACOM/DB SAS/ACCESS Interface to Microsoft SQL SAS/ACCESS Interface to IDMS/R SAS/ACCESS Interface to MySQL SAS/ACCESS Interface to IMS-DL/I SAS/ACCESS Interface to ODBC SAS/ACCESS Interface to PC Files SAS/ACCESS Interface to OLE DB SAS/ACCESS Interface to SYSTEM 2000 SAS/ACCESS Interface to Oracle SAS/ACCESS Interface to Sybase SAS/ACCESS Interface to Sybase IQ SAS/ACCESS Interface to Teradata SAS Data Surveyor for SalesForce.com MPP Database Appliances SAS/ACCESS Interface to Aster Data nCluster SAS/ACCESS Interface to Greenplum SAS/ACCESS Interface to Netezza Enterprise Applications SAS Data Surveyor for Oracle Applications SAS Data Surveyor for SAP SAS Data Surveyor for Siebel OLAP Message Queues SAS/ACCESS Interface to Exadata XML SAS/ACCESS Interface to ODBC Web Services Unstructured, Semi-Structured And more… (ParAccel, Vertica, Microsoft PDW) Enterprise-wide, netcentric to meet the operational needs of the AF 12 Copyright © 2011, SAS Institute Inc. All rights reserved. SAS Data Integration and Management Benefits and Features Data Integration Tasks SAS DataFlux Other ETL Tools Table Creation Low Effort High Effort Fast Extract Automatic Separate Utility Batch & Real-Time Low Effort Multi Step Deployment Data Quality Integrated Separate Tool Data Services Low Effort Custom Code High Performance Low Performance SQL Pushdown Automatic High Effort Federated Views Automatic Separate Tool Parallelism Low Effort / Any Job High Effort / Limited Job Sets Scoring Integrated Custom Code Customization Low Effort High Effort Sorts & Join DATA INTEGRATION 13 Copyright © 2011, SAS Institute Inc. All rights reserved. SAS Data Management Workbench DATA INTEGRATION Data Integration via a Flexible and Powerful Interface Execute Data Quality Workflows Metadata Driven Environment » Deploy governance within the IT infrastructure » Determine integration method » Real-time » Batch » Virtual » Reuse and redeploy the same set of business rules across applications Review execution results » Exceptions » Debugging » Tuning » Optimization DM System Reporting 14 14 Copyright © 2011, SAS Institute Inc. All rights reserved. SAS Data Management Advantage Scalable : In-Database EL-T DATA INTEGRATION Visual Indicators show you where processing takes place. Push-down is a simple option set on the SQL transform. Users can control where processing takes place, two clicks enable users to “move first then process” via bulk loader. 15 Copyright © 2011, SAS Institute Inc. All rights reserved. SAS Data Management Advantage Analytic Integration : Integrated Scoring DATA INTEGRATION Integration of Predictive analytics from SAS’ Enterprise Miner into Data Management tasks, takes just two clicks. Shared Metadata provides the lineage of model dependencies and where models are consumed With Scoring Accelerator Scoring Functions can be Pushed “In-Database” and called via SQL Function calls from DI Studio 16 Copyright © 2011, SAS Institute Inc. All rights reserved. Metadata Management Sophisticated metadata mapping technologies accelerates development Impact Analysis/Data Lineage to see process and data relationships Multi-user Collaboration with DEV/TEST/PROD promotion management Powerful metadata reporting including runtime and performance analysis User Interface Roles and Security DATA INTEGRATION 17 Copyright © 2011, SAS Institute Inc. All rights reserved. SAS Data Management Advantage Best in Class: DataFlux Data Quality Embedded into batch, near-time and real-time processes. Rules callable through message queues, Web services and custom exits. Data cleansing provided in native languages with localizations. Metadata built and shared across the entire SAS Platform. Generate and append postal addresses, geo-coding, demographic data or facts from other sources of information. Profile operational data and monitor ongoing data activities. DATA QUALITY 18 Copyright © 2011, SAS Institute Inc. All rights reserved. Data Cleansing and Enrichment Data quality, including integrated profiling, exploration, business rules creation, entity resolution and monitoring. Process orchestration layers for jobs, SAS code and SQL. Data quality is embedded into batch, near-time and real-time processes. Out-of-the-box standardization rules; or you can create customized, reusable data quality business rules. Data quality rules are callable through message queues, Web services and custom exits. Data enrichment and augmentation. Real-time transaction cleansing using standard business rules. Data quality monitoring lets you continuously examine data in real time and over time to discover when quality falls below acceptable limits. Alerts are issued when corrective action is needed. DATA QUALITY 19 Copyright © 2011, SAS Institute Inc. All rights reserved. Business Rules to Track Data Events DATA QUALITY Events from business rules are SOA enabled 20 Copyright © 2011, SAS Institute Inc. All rights reserved. Monitor Data Quality Monitor on the Web; Multiple repositories Workflow capabilities DataFlux Data Management Web Studio Monitor Filter: Triggered rules: Apply Clear General * Open Table Reason Total count: 24 Assigned User Integration Franz Kafka 4378 Unresolved Aging Ernest Hemingway 31 4023 Unresolved Integration Joseph Conrad 6 856 3874 Unresolved Organizati … Ernest Hemingway 6/5/2010 1:1:1 PM 3 343 3689 Unresolved Process Joseph Conrad Rule 6 6/4/2010 1:1:1 PM 1 11 3241 Unresolved Transform … Ernest Hemingway Rule 6 6/3/2010 1:1:1 PM 1 51 3241 Assigned Usage Joseph Conrad Rule 3 6/2/2010 1:1:1 PM 2 92 4378 Assigned User Error Joseph Conrad Rule 3 6/1/2010 1:1:1 PM 1 31 4023 Assigned Aging Joseph Conrad Rule Date Importanc e Rule 1 3 563 5002 Unresolved Date: This week 6/10/2010 1:1:1 PM Rule 2 6/9/2010 1:1:1 PM 6 786 4789 Rule: Rule 3 6/8/2010 1:1:1 PM 2 42 Rule 3 6/7/2010 1:1:1 PM 1 Task: Rule 4 6/6/2010 1:1:1 PM Source: Rule 5 Repository: % Triggers # Triggers Rows Status Processed Description: Importance : Summary 5/30/2010 1:1:1 PM Rule 4 5/30/2010 1:1:1 PM Status Term Table Trigger Percentage Run Trigger Values Rule 4 Monitor Statistics DATA QUALITY Rule Code Status 6 6 History Graph 696 3874 Resolved 696 3874 Resolved Trigger History (Percentage of Rows Processed) Integration Integration Mary Cassatt Mary Cassatt 10 8 6 4 2 0 9/7/2009 10/7/2009 11/7/2009 12/7/2009 1/7/2010 2/7/2010 3/7/2010 4/7/2010 5/7/2010 6/7/2010 7/7/2010 Date Copyright © 2011, SAS Institute Inc. All rights reserved. 21 Entity Resolution for Data Services Knowing your Airman Cust. Id Legacy 30391-244 William CRM Online ERP Data Warehouse 1001 First Name Middle James MASTER DATA Last Name DOB SSN Sosulski 563-49-1234123 Oak St., Eves, IL 30319 04/12/39 Address Cust. Id First Name Middle Last Name DOB SSN 30391244 William Sosulski 563491234 123 Oak St., Eves, IL Cust. Id First Name Middle 14239 Bubba Cust. Id First Name Middle Last Name DOB SSN 3721B Willaim Corp. 56349123 3224 Pkwy G, Los Osos Cust. Id First Name Middle 30391-244 William 30391-244 30391244 14239 J. Last Name DOB J. James James 4-12-39 SSN April 12 April 12 Last Name DOB Sosulski Address Address BubbaJ@bubbagroup.com SSN Address Address 04/12/1939 563-49-1234123 Oak St., Eves, IL 30319 3721B 30391-244 William James Sosulski 04/12/1939 563491234 123 Oak Street Eves 22 Copyright © 2011, SAS Institute Inc. All rights reserved. CA 91403 22 SAS Data Management Benefits Control & Lower TCO DATA INTEGRATION Integrated DQ Functions Specialized DQ Interfaces SOA Enabled Lineage Reporting from Source to Report Over 300 Functions 70 built-in Transformations Analytic Integration & Scoring Scalable Partition-Based Parallelism Record Setting File I/O Best-in-Class Data Access Technology /w/ Widest Array of Data Sources and Targets Version Control 23 Copyright © 2011, SAS Institute Inc. All rights reserved. Why SAS for AFPOA? Flexibility Agile and responsive to meet the needs of the Air Force’s most important asset: Air Force Personnel Why it Matters Feature Data quality rules automatically exposed as RESTful and SOAP Web Services All tools are built on a 4th Generation SAS Programming Language and C++ Unified runtime across computing environments Seamless integration with AFPOA enterprise architecture and workflow Enables AFPOA to more easily adapt SAS/DataFlux data management capabilities to changing mission needs Enables AFPOA to attain cloud computing environments 24 Copyright © 2011, SAS Institute Inc. All rights reserved. SAS Can Be Your Trusted Partner PERFORMANCE #1 World Leader in Business Analytics 50,000+ Customers 12,000 Employees Worldwide CULTURE Relentless Innovation Voted #1 Place to Work in U.S. Trusted Partner to Governments and Leading Business Organizations EXPERIENCE 50,000 SAS Sites in 127 Countries 93 of the Top 100 Companies in 2011 Fortune Global 500 35 Years Leading Analytics Solutions THOUGHT LEADER SAS Advanced Analytics Lab Provides Business Leadership Domain Expertise in Key Industries Culture of Innovation: 24% R&D Reinvestment 25 Copyright © 2011, SAS Institute Inc. All rights reserved. Resources Here are the URLs and titles of downloadable materials we refer you to for more information: SAS Data Management - http://www.sas.com/software/datamanagement/ From this webpage, you can access downloadable materials and drill down deeper into Data Integration Data Quality Enterprise Data Access Master Data Management Here are two additional white papers you might find helpful which were provided to Bloor. “Data Quality Remediation” (DataFlux) “DataFlux Data Management Methodology: A Do-It-Yourself Guide for High-Value Data Across the Enterprise” (DataFlux) 26 Copyright © 2011, SAS Institute Inc. All rights reserved. SAS® Data Integration Thank you! If you are interested in hearing more about SAS and how we address Information and Data Integration Management, please contact: Gail Bamford Industry Marketing Manager for Defense & Intelligence Gail.Bamford@sas.com 571-227-7000 x51715 27 Copyright © 2011, SAS Institute Inc. All rights reserved.