Ivan Neganov About Ivan Neganov Founder and CEO of SoftForte, Inc. 11 years of experience in developing WCM solutions based on ASP.NET and SharePoint platforms. Focusing on SharePoint since 2007. Blog: neganov.blogspot.com the Science of Quality Web: www.softforte.com Email: [email protected] 2 Agenda Part I – Planning for Performance Part II – Planning for Throughput 3 Part I – Planning for Performance Performance Defined 4 How Fast is “Fast”? Human Psysiology Factor 2006 Akamai/Jupiter Research Under 0.1 sec – virtually unnoticeable. Under 1 sec – perceived as interactive Under 10 sec – willing to focus on a task 33% of broadband consumers will wait no longer than 4 sec for a page to load. 2009 Akamai/Forrester Research 2 sec. – average expectation of online shopper 3 sec. – max time 40% shoppers are willing to wait for a page to load http://www.akamai.com/html/about/press/releases/2009/press_091409.html KB40 – Keynote Business 40. Keynote Systems, Inc. maintains index of fastest business internet sites: http://www.keynote.com/keynote_competitive_research/performance_indices/ WM100 – Webmetrics maintains index of top 100 sites by performance http://www.webmetrics.com/resources/benchmarking.html 5 SharePoint Response Time Guidance Type of operation Common operation Uncommon operation Rare operation Examples Acceptable user response time · Browsing to the home page <3 seconds · Browsing to a document library · Creating a subsite <5 seconds Creating a list · Uploading a document to a document library · Backing up a site <7 seconds · Creating a site collection http://technet.microsoft.com/en-ca/library/cc262787.aspx 6 How Fast is “Fast” in my Company? Study publicly available metrics Study organization’s historical metrics Estimate average and peak traffic Define a matrix of PLT1 and PLT2: For various pages For various authentication groups For peak and average usage 7 Response Time Page Load Time (PLT) or User Response Time (URT) – time until a page fully renders. Microsoft uses PLT1 and PLT2 – the very first access to the page, and subsequent access to the same page. 8 Part I – Planning for Performance CNS Model: Client Network Server 9 URT Formula 10 URT Formula (Netforecast) R – response time Payload – total size of page and all its resources AppTurns – round trips made at application level (excluding TCP handshake/congestion control round trips & authentication) RTT – round trip time Cs – constant server time component Cc – constant client time component Reference: http://www.webperformancematters.com/journal/2007/7/24/latencybandwidth-and-response-times.html 11 Need for Testing Simply applying the formula will lead to significant errors. You need to calibrate every part of it. Testing produces data for calibration. 12 Part I – Planning for Performance Client Performance 13 Client Scripting Performance J-Query profiler from John Resig allows to measure performance by method and calculate Big-O breakdown. http://ejohn.org/blog/function-callprofiling/ Profiling script from within script is very imprecise, partly due to platform implementation. For example, on Windows XP timer would show intervals shorter than 15ms as 0. Profilers: YSlow for Firebug - http://developer.yahoo.com/yslow/ JScript Profiler http://blogs.msdn.com/ie/archive/2008/09/11/introducing-theie8-developer-tools-jscript-profiler.aspx DynaTrace profiler (can profile script parsing time!) Article: http://ejohn.org/blog/deep-tracing-of-internet-explorer/ 14 Part I – Planning for Performance Network Performance 15 Network Performance – the Bottleneck Bandwidth limitations – can be addressed via technology Latency limitations – Speed of Light RTT/2 = (36,000 *2)/300,000 RTT ~ 0.5 sec. TCP limitations Signal strength/QoS 16 Latency and Bandwidth Overall link bandwidth = 3 Mbit/s What is my actual bandwidth & latency? www.speedtest.net detects your local bandwidth and latency. 17 TCP Communication A max. packet size on Ethernet is 1500 bytes, aka MTU or max. transfer unit. On IPv4 networks IP overhead takes 40 bytes, hence max payload equals 1460 bytes, aka MSS or max. segment size. TCP requires acknowledgement (ACK) of all packets sent but allows sending a number of packets without waiting for ACK to improve speed. Eventually ACK must arrive. If some packets are lost, i.e. there is no ACK within a timeout, then packets are re-transmitted. 18 TCP Communication: Naïve Model 19 TCP Communication: Realistic Model 20 TCP Communication: TCP Window 21 TCP Window TCP Window is a number of bytes a receiver can accept without sending ACK immediately. Too large window means network congestion >> lost packets >> retransmission >> performance degradation Too small window means low bandwidth utilization >> performance degradation 22 TCP Slow Start Optimal window size is twice the amount of data that can be “in flight” on the wire from sender to receiver at any given time: RWIN = 2 * (Bandwidth * RTT/2), or RWIN = 2 * BDP BDP – bandwidth-delay product. RWIN – TCP receive window buffer. TCP detects bandwidth and latency and dynamically sets window size. Usually initial RWIN = 64KB. Once connection is established, TCP increases RWIN, process aka “Slow Start”. ”. On a slower WAN it can take up to 12 round trips to optimize the receive window. Initial RWIN size on W2K3: http://msdn.microsoft.com/enus/library/ms819736.aspx 23 TCP Congestion Control Sender maintains congestion window, CWND and constantly tweaks it according to bandwidth and delay to avoid congestion: Effective bandwidth = CWND/RTT Various congestion control algorithms are known, ex. Tahoe, Reno. Windows Vista, 7 and 2008 use CTCP. It is advantageous over WAN, enabled by default on 2008, but not on Vista and Windows 7. Reference: http://technet.microsoft.com/enus/library/bb726965.aspx 24 TCP Congestion Window Scaling 25 TCP Packet Loss Packet loss may occur for many reasons, ex. when network is congested or equipment is misconfigured, or there is a signal loss, etc. Packet loss severely impacts throughput: Throughput <= 0.7 * MSS/(RTT * Sqrt(Ploss)) MSS – Max. segment size, 1460 bytes for IPv4, 1440 bytes for IPv6 on Ethernet. Ploss – probability of a packet loss. Example: At 100ms round trip time and 10-4 probability of a packet loss you would get no more than 8Mbit/s throughput. Contemporary networks have very low packet loss probability, yet some packet loss occurs on long links. WAN testing is sometimes done assuming 1 – 3% of packet loss. 26 Addressing TCP Limitations Using UDP instead of TCP Minimizing number of round trips Using few large files vs. many small files Using multiple browser connections Using HTTP persistent connections Using client-side caching Using Content Delivery Networks (CDN) Using WAN accelerators & offloading devices 27 Multiple Browser Connections Contemporary browsers use multiple TCP connections per hostname: IE6, IE7 – 2 connections max; IE8, FireFox 3.5 – 6 connections max. Open multiple (source) ports for multiple TCP connections. Despite having multiple connections a lot of sequential loading still takes place. IE8 is the first browser to download multiple script files in parallel. 28 HTTP Persistent Connections HTTP 1.1 supports persistent connections through Keep-Alive header. The goal is to re-use underlying TCP connection with its current CWND avoiding having to go through Slow Start again. Enabled by default on most browsers and on IIS 6, 7. Keep-alive timeout is 1 min for IE and 15 sec. for FireFox, and is adjustable. For changing timeout on IE6, 7 see: http://support.microsoft.com/kb/813827 Enabling Keep-Alive in IIS7: http://technet.microsoft.com/enus/library/cc772183(WS.10).aspx 29 Content Delivery Networks CDNs distribute cached content on multiple servers, which are close to end users. Internet traffic is redirected to the closest CDN server instead of the origin server. Advantages: Disadvantages: Very expensive, typically affordable to large enterprises only. Ex. $0.5/GB on 50 TB monthly ~25,000$/month Less efficient for highly volatile content. It can be technically difficult to invalidate CDN cache explicitly. Free CDNs, primarily AJAX support: Low latency & high bandwidth when accessing a CDN server result in much better performance for the end users. As a result of many users hitting CDN cache the load on original server is reduced. Excellent for media streaming. Google AJAX Libraries API - http://code.google.com/apis/ajaxlibs/ Microsoft AJAX CDN http://weblogs.asp.net/scottgu/archive/2009/09/15/announcing-the-microsoft-ajaxcdn.aspx More Info about CDNs: http://en.wikipedia.org/wiki/Content_delivery_network#Free_CDNs 30 WAN Accelerators & Offloading Devices Use packet compression, differencing, caching, optimal route calculation algorithms, reducing packet loss. Solutions include Cisco, Citrix, Packeteer, Riverbed, F5, Brocade. Microsoft’s ISA and IAG, and their successor Unified Access Gateway (UAG 2010) provide caching, offloaded compression, differencing and authentication delegation. 31 Determining Network Performance Nature of network transmission complicates its mathematical modeling and projection of results between different networks. This increases amount of calibration testing needed. Create a reference set of web pages and test them on various networks. Calibrate earlier discussed CNS formulas using these test results. Tools are available: http://www.webpagetest.org/ http://kite.keynote.com/ http://msdn.microsoft.com/en-us/magazine/dd188562.aspx http://www.fiddler2.com/fiddler2 http://www.aptimize.com 32 Part I – Planning for Performance Server Performance 33 Server Performance Create baseline measurement for various load profiles and PLT1/PLT2 Use Performance counters: ASP.NET Request Execution Time ASP.NET Request Wait Time Server Response Time (SRT) = the sum of the two. Essential performance counters: http://support.microsoft.com/kb/815159 34 SharePoint 2010 Performance Improvements More load on WFE, SQL & Client PLT performance improvements and optimization for WAN, early page rendering “Cobalt” protocol – asynchronous uploading of an office file from client cache to server. Developer Dashboard – improves bottleneck diagnostics 35 Performance Measurement Tools SharePoint Developer Dashboard Measuring PLT1 & PLT2 36 Part II – Planning for Throughput Objectives Models Rules of Thumb Selecting Hardware SharePoint 2010 & Capacity Management 37 About Capacity Planning Objectives: Know expected load levels for the application Ensure acceptable performance at expected load levels Determine how to scale application for the future In the CNS model above, focus is primarily on Server part. Networking part matters however: CDNs do reduce server load for Internet scenarios. In geographically distributed farms WAN bandwidth and latency affect capacity planning. 38 Theoretical Web Server Model http://cuip.net/~dloquinte/researchfiles/IIT(RET)/reliability/webmodel.pdf 39 Server Under Load: Theoretical Model M/M/1 queue for single web server and MM/c queue for load-balanced servers Poisson Distribution – Memorylessness: knowledge of last occurred event does not have an impact on successive events Little’s Law: Nqueue = SRT * Ratearrival Consequences: Understanding of physical capacity limits Approximate but practical server load function Importance of RPS as a measure of capacity 40 Theoretical Server Response Time Server performance is analyzed together with the server load. From queuing analysis for M/M/1 queue: SRT = SRT(0)/(1 – U) SRT – server response time SRT(0) – server response time at 0 utilization U – utilization, or average percentage of time the server is busy. 41 SRT is a Function of Utilization 42 Load-Balanced Servers 43 SharePoint Farm Capacity Planning Theory explains guidance parameters & helps with rough estimates Rules-of-Thumb, best practices & reference performance tests are used to determine components of the farm Requests per Second (RPS) are used to measure farm capacity Additional tools: SPCP: http://technet.microsoft.com/en-us/library/bb961988.aspx 44 Throughput Targets: Classic Usage Model All SharePoint site users can be classified into 4 groups: 1. 1. 2. 3. 4. Light users – generate 20 RPH or 2 User Ops/Hour Typical users – generate 36 RPH or 3.6 User Ops/Hour Heavy users – generate 60 RPH or 6 User Ops/Hour Extreme users – generate 120 RPH or 12 User Ops/Hour 3. RPH are calculated based on daily average non-401 requests made by distinct users. 4. Given total number of users in each class set percentage of them that is active, i.e. actively using the SharePoint site. This is also known as concurrency. Even at peak usage 10% is a high concurrency, 5% is typical. 5. Weighted sum yields total demand in RPS. Reference: http://technet.microsoft.com/en-us/library/cc261795.aspx 45 Classic Usage Model - Example There are total of 30,000 users of the portal. 25,000 of them are typical users. 4,500 of them are heavy users. 500 of them are extreme users. During the peak hour on average 10% of typical users and 5% of heavy and extreme users are accessing the site. What is the required farm capacity? Capacity = (0.1 * 25,000 * 36 + 0.05 * 4,500 * 60 + 0.05 * 500 * 120)/3600 = 29.6 RPS 46 SharePoint Activities Affect Capacity A farm is serving a number of activities: User operations (web page & file requests) Search indexing Publishing Profile import/sync Variations, workflows, scheduled jobs Backup Office clients requests AJAX calls User activity and number of concurrent users are the primary factors used in capacity planning. The picture is different when backend activities cannot be confined into 12-hour window. Plan for Peak Concurrency! 47 Rules-of-Thumb: Web Front End Portal Collaboration Scenario WSS Collaboration Scenario http://technet.microsoft.com/enus/library/cc261716.aspx 48 Rules-of-Thumb: Web Front End HA prevail over capacity requirements for small and medium installations. Max RPS achieved at 5 WFEs per DB server. More WFEs overload ConfigDB. 1 DC per 3-4 WFEs, if NTLM authentication is used. Set 1 WFE as crawl target, remove it from load balancer. Average WFE CPU utilization should be 30%. 49 Rules-of-Thumb: Storage Sizing Important for performance planning because storage estimates contribute to IOPS requirement for the disk subsystem. 100 GB per content database Use reference installations, or Microsoft estimation guidance: http://technet.microsoft.com/enca/library/cc261716.aspx 50 IOPS Two common measures of disk throughput: IOPS – used for random access to disk, typical for SharePoint workloads. MB/s – used for mostly sequential access, common to serving large files, running large reports on cubes. Use performance counter: Disk Transefers/sec to determine peak IOPS based on RPS. 10K RPM drives give 100-130 IOPS; 15K RPM drives give 150-180 IOPS. Use sqlio.exe utility to determine actual IOPS of a hardware. 51 Rules-of-Thumb: SQL Server Resources on SQL for SharePoint Planning: http://technet.microsoft.com/en-us/library/cc263261.aspx Resources on SQL Mirroring: http://technet.microsoft.com/en-us/library/cc287861.aspx 52 Rules-of-Thumb: SQL Server Disk Latency: Disk sec/transfer Data files < 10ms T-log files < 5ms Disk Capacity: *RAID-5 can be used for static web content. 53 Rules-of-Thumb: SQL Server Typical Deployment Sizes: Metric Small Medium Large Content db size < 50GB 50GB > 50GB # of Content dbs < 20 20 > 20 # of concurrent requests to SQL < 200 200 > 200 # of Users < 1000 1000 > 1000 # of items in regularly accessed list < 2000 2000 > 2000 < 20 20 > 20 # of columns in regularly accessed list 54 Rules-of-Thumb: SQL Server Recommended Capacities: Resource Small Medium Large Recommended DB server memory 8 GB + 16 GB + 32 GB + 2 MB > 2 MB > 2MB Medium High High < 20 < 10 < 10 (data) < 5 (T-log) Gigabit Gigabit Gigabit <1 <1 <1 Processor L2 cache Bus bandwidth Disks latencies (msec) Network Network latency (msec) 55 Capacity Planning Summary 58 SharePoint 2010 Capacity Improvements Large list throttling WFE will return 503 when overloaded Office clients are aware of this, and will in turn throttle server requests Co-authoring of documents; PPT broadcasting. HTTP throttling Blocks robots, search indexing Gives first priority to client traffic Bit rate throttling – used by assets library, implemented in IIS Media Services extension SQL Server 2008 Throttling – Resource Governor can limit use of resources by specific processes Software boundaries improvement 59 SharePoint 2010 Capacity Planning 60 SharePoint 2010 Capacity Management Logging DB Developer Dashboard Load Testing Toolkit (a part of SharePoint Administration Toolkit) There is more to come… 61 Capacity Validation Load Test Performance Counters Performance counters are central in determining all aspects of performance. One example for capacity planning: ASP.NET Applications\Request/sec A comprehensive list of relevant counters is available here: http://blogs.msdn.com/ketaanhs/archive/2010/03/13/mossperformance-counters.aspx 63 Load Testing Tools SharePoint 2010:Load Testing Kit, part of SharePoint Administration Toolkit – reference Web & Load tests. VSTT Useful blog post by Bill Baer lists tools used for stress testing of SharePoint http://blogs.technet.com/wbaer/archive/2007/08/02/stress-testingmicrosoft-office-sharepoint-server-2007-windows-sharepointservices-3-0.aspx 64 Part III – Best Practices Information Architecture Web Front End (WFE) Servers SQL Server 65 Information Architecture: Best Practices Account for software boundaries: http://technet.microsoft.com/en-us/library/cc262787.aspx For large lists, follow performance guidance: http://technet.microsoft.com/en-us/library/cc262813.aspx Separate content with different usage profiles into different site collections Account for authentication performance impact: Anonymous - fastest Kerberos NTLM Basic Forms - slowest 66 WFE Best Practices: Caching Output caching & cache profiles BLOB caching Native to ASP.NET 2.0, individual page level Turned off by default in 2007. Need Publishing Infrastructure Feature on. Enable for read-only users. Never cache search results for authenticated users, alternatively disable search results page. Uses RAM on WFE, adjust ASP.NET private byte limit Used on document libraries only Minimizes round-trips to database for HTML, CSS, image or media files, etc. by creating disk-based cache on WFE Not enabled by default Important to use max-age attribute to instruct clients to cache resources Affects disk I/O of the WFE servers Object caching Benefit for certain page items: navigation data, cross-list query data Uses RAM: default 100MB. Monitor cache hit ratio counters and adjust RAM to have over 90% hits. The only caching turned on by default Office Web Applications Caching (SharePoint 2010) Branch Caching (Windows 2008) More Info: http://technet.microsoft.com/en-us/library/cc298466.aspx 67 WFE Best Practices: IIS Compression Static IIS compression is on by default in IIS 6, 7. Used for *.html, *.htm, *.css, *.txt files by default. Dynamic compression is off by default on both IIS 6 & 7. Used for *.asp, *exe files by default. Using IIS compression increases load on WFE CPU, but it reduces disk I/O, which is much slower, so it can dramatically boost performance. You need to configure compression levels, and add extensions for *.js, *.aspx etc. IIS 7 can be configured to compress items before adding them to cache. This needs to be turned on to reduce load on the CPU. 68 WFE Best Practices: Custom Code Releasing resources for SPSite, SPWeb Avoid thread synchronization issues when caching objects Accessing folders and lists Do not use SPList.Items Use SPList.GetItems(SPQuery) Do not iterate over SPList.Items Use PortalSiteMapProvider to enumerate lists Scalability: avoid code, enumerating OM objects for large # of concurrent users SPQuery objects Do not use unbounded SPQuery objects Use indexed fields in queries Timer jobs Break long-running operations into small pieces to minimize re- do work when restarting a job. 69 WFE Best Practices: Other Load scripts outside of script engine using document.write(<script src=…); SharePoint 2010 does this! Make sure HTTP 1.1 keep-alive header is on. It is used by persistent connections, turned on by default in IIS 6,7 Minimize number of small file downloads. Reason: many small files do not use link capacity fully. Single large file downloads are more efficient. Load scripts on demand where possible. Ex. Core.js script on Internet sites 70 WFE Best Practices: More Info SharePoint Dispose Checker Tool http://code.msdn.microsoft.com/SPDisposeCheck 12 Steps for Faster Web Pages – Jim Pierson: http://msdn.microsoft.com/en-us/magazine/dd188562.aspx Tuning web server performance: http://technet.microsoft.com/en-us/library/cc298550.aspx Andrew Connell on Performance: http://msdn.microsoft.com/en-us/library/ee857096.aspx Common Coding Issues with SharePoint OM: http://msdn.microsoft.com/en-us/library/bb687949.aspx Optimizing Custom WP for the WAN: http://technet.microsoft.com/en-us/library/cc263412.aspx Configuring Caching & Performance – James Petrosky: http://www.microsoft.com/winme/0712/31729/Module5/Local/index.html 71 Search Best Practices (MOSS 2007) Use dedicated server for Indexing when possible. Do not combine search and query roles on the same server. Set one of WFEs as a crawl target, and remove it from load balancer Search query performance is improved when using multiple load-balanced query servers: http://technet.microsoft.com/en us/library/cc262574.aspx 72 SQL Server Best Practices Ensure correct Host Bus Adapter drivers and firmware versions. Configure correct NTFS allocation unit size (done during formatting the drive, as a format.exe option) 64K – best. Equals to SQL Server extent size. Default is 4K, can result in 30% performance hit. Ensure correct Windows sector alignment. Windows 2008 aligns sectors by default (done during partitioning of the drive) Incorrect alignment can result in up to 50% performance hit. More Info: http://msdn.microsoft.com/en-us/library/dd758814.aspx 73 SQL Server Best Practices Database file placement priority among faster disks: tempdb data and T-log files Db T-log files Search database data files Content Database data files Place tempdb, Content db and T-logs on separate LUNs. Use multiple data files for Content and Search db distribute them across disks. # of files should be <= # of processor cores Multiple data files are not supported for other dbs. Place SharePoint Search crawl and query tables on separate spindles. 74 SQL Server Best Practices 100 GB content databases (soft) limit. Break content into content databases by IO profile. Example: store collaboration sites content and publishing portal content in different databases. Use dedicated database for large site collections (> 50 GB) Configure tempdb files = # of processors Configure tempdb to be 25% of content db size. Alternatively either at least 10% or the size of the largest table, whichever is greater. More Info: http://technet.microsoft.com/en-us/library/cc263261.aspx http://technet.microsoft.com/en-us/library/cc298801.aspx 75 Please complete your evaluations to enter in the prize drawing!