Regression Testing - Gerrard Consulting

Regression Testing – What to Automate
and how
Paul Gerrard
Gerrard Consulting
1 Old Forge Close
e: [email protected]
t: 01628 639173
Assurance with Intelligence
In this session, I want to re-trace
our early steps to automation
Is life hard at the end because we
didn’t think it through the beginning?
Assurance with Intelligence
Slide 3
Automating regression tests
• Regression tests are the most likely to be stable
and run repeatedly so:
– Automation promises big savings of time
– Automation ensures reliable execution and results
– Stable tests/software are usually easiest to automate
• Hold on! Paradox ALERT!!!!
– We regression test because things change
– Chaotic, unstable environments need it the most
– When things change automation is HARD!
Slide 4
Assurance with Intelligence
Slide 4
What is the regression testing
thought process?
Assurance with Intelligence
Slide 5
How I think about regression
Establish what we mean by regression testing
Understand why regressions occur
Combating regression – there are choices
Who are the stakeholders for regression testing
and what do they want?
Context of regression testing
Regression test policy
Selecting what to regression test
Selecting what regression tests to automate
Assurance with Intelligence
Slide 6
A common experience?
• Regression testing is
a) Painful
b) Boring
c) A waste of time
“Let’s get a tool in”
Automating 10% of tests is easy
Automating 90% of tests is hard (impossible?)
“Why are we doing this?”
“Let’s do it manually”
Assurance with Intelligence
Slide 7
Let’s look at each thought
Assurance with Intelligence
Slide 8
(What is...) regression testing?
• When a software fault is fixed, it often happens that
'knock-on' effects occur
• We need to check that when changing the faulty code,
the developer has not introduced new faults
• There is a 50% chance of introducing regression faults
• Regression tests tell us whether new faults have been
– i.e. whether the system still works after a change to the
code or environment has been made.
• When environments are changed, we might also
regression test.
Slide 9
Assurance with Intelligence
Slide 9
Regression testing 2
• “Testing to ensure a change has not caused
faults in unchanged parts of the system"
• Not necessarily a separate stage
• Regression testing most important during
maintenance activities
• Regression tests are usually the first to be
considered for automation.
Slide 10
Assurance with Intelligence
Slide 10
How do regressions occur?
• Something changes and
affects ‘working’ software
• An enhancement is
• A bug is fixed
• Two bug fixes interfere
with each other
• One programmer
overwrites another
programmer’s fix
• Something in the
environment setup
• Test data changes in some
• A test changes in some
• Test configuration/setup
• Etc. etc.
• Can you think of others?
Assurance with Intelligence
Slide 11
Code fixes have a 50% chance of
introducing side-effects in
working software
Ancient history – who cares whether
it’s 10% or 90%?
Regressions have a disproportionate
impact on effort, confidence, morale
Assurance with Intelligence
Slide 12
Combating regression - choices
• Several options:
– Don’t change anything!
– Impact analysis (of code/data//test changes)
– Regression testing (to detect unplanned changes)
• Impact analysis is tricky; not 100% reliable but...
Compare source code – see what’s changed
Module dependencies – see what’s affected
Rigour in managing environments and data
Rigour in managing source code
• Regression testing choices
Manual v automated
Test first or test last
All tests or a selection of tests
Assurance with Intelligence
Slide 13
Who are our stakeholders?
• Stakeholders are people interested in the outcome of test
• What do they want? Ideally, nothing!
• Sponsors, customers
– Regressions are an immediate setback/delay just when they
thought you could move the project forward
• Users and testers
– “All our good work is undermined!”
• Developers
– They’ve missed something obvious, embarrassing; unplanned
work – could be trivial or substantial
• Tech support
– “Users are going to be furious!”
– “We feel really let down by dev, maintenance, testers”
Assurance with Intelligence
Slide 14
Context of regression testing
• Test- or behaviour-driven development:
– All tests are automated; these are our safety-net
– Tests allow us to refactor with confidence
• Integration/System/Acceptance testing 1st pass
– RT after main testing done – confirmation that all
tests pass with all fixes in place
• Maintenance phase
– Minor fixes (don’t get impact analysed)
– Enhancements fit with existing system
– Refactoring/conversion/interface changes.
Assurance with Intelligence
Slide 15
Regression test policy
• Common:
– Business critical, high risk, error-prone areas
– FIFA – first in, first to automate?
– LIFA – last in first to automate
– Easily automatable
• But let’s step back a little
– Focus on stakeholders
– What do THEY actually want?
Assurance with Intelligence
Slide 16
What would stakeholders want?
• Sponsors, customers
– Want to avoid delays
– Is the most critical functionality working?
• Tech support
– Are we inconveniencing the users?
– Is the most critical functionality working?
• Users and testers
– Can we trust the results of our early (high priority tests)
– (Maybe we aren’t so worried about the rest?)
– Is the most critical functionality working?
• Developers
– Want instant feedback (so they can fix with least hassle)
– The best case for them doing their own automated tests!
Assurance with Intelligence
Slide 17
If a developer ran your test team...
• Whose problem is regression anyway?
• It’s developers’ problem not the testers’!
• Dev have been asked to fix/enhance/refactor
code without screwing up
• How would a developer address the regression
– Design a harness and code to execute transactions at
the API level
– Easier, more maintainable, less sensitive to cosmetic
• A reusable asset that allows refactoring.
Assurance with Intelligence
Slide 18
The real problem to solve?
• If the problem is identifying change where there
shouldn’t be any...
– With a few caveats, it doesn’t matter what tests we run
as long as we cover the important functionality – and that
it behaves the same way as it did last time
– We still need to demonstrate the user interface isn’t
adversely affected and the end-to-ends still work
– Is it POSSIBLE and BETTER to do this manually?
– Why use a framework on top of a test execution tool on
top of a browser to test server-based code? Too complex!
• Where does that leave automated test execution – on
the shelf? (It’s probably there already!)
Assurance with Intelligence
Slide 19
Assurance with Intelligence
Slide 20
Techniques for Testing GUI
• Eurostar 1997: the most popular paper on my
website is referenced by Wikipedia
• Won’t repeat the whole story here...
• But most of the 1997 slides are still relevant
• Have we made NO progress since 1997?
Assurance with Intelligence
Slide 21
Selecting what regression tests
to automate
Assurance with Intelligence
Slide 22
Manual v automatic scripts - very
rough guideline
Manual Scripts
Automatic Scripts
 Check list (application
 Integration
 Check list (object
states, menus, standard
 Synchronisation
 Navigation
 Client/server
 Integration (for soak
 Low-volume/complex
 State transition
 Simple EP/BV and
decision tables.
What aspects to automate?
• Pareto law:
– go for the easy scripts first
– don’t waste time scripting low volume complex
scripts at the expense of high volume simple ones
• Consider hybrid approach:
– use the tool to do navigation and data entry
– do the checking on-line or at end of test.
Automated test scripts
• Coded scripts
– navigation through the system, some checklist test
cases can be can be coded directly
– relatively easy to maintain
• Recorded scripts
– automated scripts should be documented
– then used to record the automated script
– less easy to maintain
• Integrate all scripts into automated suite.
Migration of manual scripts
• Automated scripts are easier to maintain
– if designed to be maintained
– Means you have to build test automation scaffolding
before you script tests
– Automation projects are software development
projects, not test projects
• Allow for maintenance cost before deciding to
migrate functional scripts
– delay migration until system is stable
– soak and regression tests, as required.
• The regression problem is a developers’
problem that can be solved by:
Developer discipline
Better impact analysis
Automated testing at an API level
Manual/automated testing (critical/E2E level) for
visibility to stakeholders
• Is this what the big product companies do?
Assurance with Intelligence
Slide 27
Regression Testing – What to Automate
and how
Paul Gerrard
Gerrard Consulting
1 Old Forge Close
e: [email protected]
t: 01628 639173
Assurance with Intelligence

similar documents