Review: Agile Software Testing in Large-Scale Project Talha Majeed COMP 587 Spring 2011 2 Agenda Definitions Introduction Adopting XP and diverging from XP Guidelines in four Development areas Test design and activity execution Working with professional testers Planning Defect management Summery Conclusion 3 Definitions Agile testing is a software testing practice that follows the principles of the overall agile manifesto In agile testing, testing is not a phase. It is integrated into Development & Testing QA is not the last line of defense but involves at the beginning of the project. Agile testing focuses on testing iteratively, as often as stable code base is available, until quality is achieved from end customer’s perspective. 4 Definitions Extreme Programming (XP) is a software development methodology which is intended to improve software quality and responsiveness to changing customer requirements. Frequent "releases" in short development cycles which is intended to improve productivity . XP Approach is that if a little testing can eliminate a few flaws, a lot of testing can eliminate many more flaws. Unit tests determine whether a given feature works as intended. A programmer writes as many automated tests as they can think of that might "break" the code; if all tests run successfully, then the coding is complete. Acceptance tests verify that the requirements as understood by the programmers satisfy the customer's actual requirements. 5 Introduction Israel Air Force (IAF) Enterprise information system Project was risky Critical to daily operations & information security Highly complex and extreme quality size Scope (XP-based method) Risk management Metrics were developed Collected & analyzed data 6 Project overview Data collection Four releases (eight months) Quantitative data Project team used for project tracking & decision making Qualitative data Recording of formal reflection meeting & debriefings 7 Confirming to XP Team implemented standard XP practices Short release Working software two week iteration Customer feedback Goal to achieve measureable testing benefits Short average time for defect fixes Short average defect longevity 8 Confirming to XP Planning game practice To plan project iterations and releases Entire teem met and customer described and prioritized stories Goal was to achieve simple design by refactoring activities XP’s whole-team and site-together practices Two measureable results Defect management overhead was lower Fewer false defects 9 Diverging from XP The project differed from classic XP in three aspects First aspect Second aspect Team was required to produce detailed specification in addition to code and test due to organizational policy. Project’s accepting testing( user testing) was far too complex for customer to fully specify Third aspect Modified test-driven development to focus on the continuous creation of automated acceptance tests rather than unit tests. Never consider a feature “complete before all its tests were written and passed. 10 Test design and execution Each team’s member was responsible for testing In a traditional project, "everyone is responsible for quality,” but in agile project, everyone actually writes testes. Advantages First, It eliminated the team’s reliance's on single tester Second, Developers test-awareness increased Finally, team members wrote specification and code to write test cases for their work. 11 Test design and execution Product size = test size Progress was measured by defining the project’s product size metrics as the number of regression test steps ran at each iteration The above approach was chosen for two reasons First, test size is more correlated with complexity than are lines of code or specification 12 Test design and execution Second, only features with full regression testing at each iteration are counted as delivered product size. Figure 1 plots the product size metric period of time 13 Test design and execution Untested work = no work (major shifting to agile development) Untested work equals to no work in all channels and all under conditions. Written Questionnaire response after two releases 90% team members considered acceptance and system tests “highly important” 10% ranked as “important” 33% were interested in taking a leading role in acceptance testing 60% were “interested” in leading system testing. 14 Working with professional testers Why professional testers? Add value through more testing Developers code more features Professional tester two key challenges for organization Testing bottlenecks Coordinating the testing work among testers and programmers 15 Working with professional testers Easing bottlenecks Developer s simply code less, test more Figure 2. illustrates by comparing the product size metric points attained by tests that the entire team ran to the team’s tester alone 16 Working with professional testers Encourage interaction over isolation Initially testers worked alone due to two reasons Testers must be independent of programmers to prevent implementation details from affecting their test designs. Specification were up-to-date so testers write test according to specification Testers were integrated to project team 17 Activity Planning Planning for quality activities involve Time allocation Challenges in feature testing Regression testing Repair defects 18 Activity Planning Figure 3. Testing and defect repair (a) Net hours devoted to running and maintaining regression tests. (b) Net hours devoted to fixing defects 19 Activity planning Integrate feature testing and coding Testing and coding time are usually equal Regression testing: divide and conquer Alocate bug-fix time globally To estimate the time to fix bug is difficult Reasonably predict the time required to fixing defects in forthcoming iterations 20 Defect management Defect management involves two major challenges Managing workflow Selecting and scheduling the defects to fix Use a team-centered defect-management approach Anyone can open defect Anyone can close a defect, after fixing it Anyone who finds a defect also assigns it to someone to be fixed. 21 Defect management Fix defects as soon as possible Bug fix time was slightly over an hour Bug fix requires more time ( a day or more) should be suspended to prevent bottleneck and to ensure deadlines are met. Advantages Defects require far less time to fix Working on clean highly stable code base makes new development faster Avoids unpleasant customer negotiations over which defects to solve. 22 Defect management Figure 3. Average net time to fix a defect per iteration 23 Summery Presented definitions of Agile testing, XP , Unit test, Acceptance tests etc. How Project adopts/ diverges from XP Identified How agile software testing works on large-scale project Showed the graphs of real data from Israel Air Force enterprise information system. Described the Guidelines in four Development areas 24 Conclusion Project's data and analysis are useful both to practitioners wishing to establish new agile teams and to research work Even on large-scale project, the team achieved full regression testing at each iteration and developer testing There is a need of quality data on large-scale project so, it is highly encouraged for further investigation on long-term agile projects.