Agile Testing Explained
The role of testers and testing in Agile development is largely misunderstood. Although there are many ways that testers can help in this maturing area (several books have been written on the subject) the main role of testers is to take user stories and turn them into business-facing tests by way of examples. The majority of these are then automated to provide ongoing regression testing.
Figure 1 illustrates the Agile Testing approach.
Unit Testing and Business-Facing Testing take place in parallel, both creating automated tests that form part of the Continuous Build. There is no waiting for code drops and a close working relationship between developers, business analysts and testers.
Figure 1 Notes
- Testers work closely with Business Analysts (or people with relevant domain knowledge) to take user stories and construct table-based examples that can be shared with developers. For the most part these are then automated and included in the Continuous Build.
The mindset of testers is different to that of business analysts and developers. They're adept at defining both positive and negative test cases. Over time the goal should be to build up extensive test suites that verify not just the happy path that's of most interest to business users, but also negative tests to confirm that the application is sufficiently robust.
- The use of concrete examples acts as a vehicle for confirmation of detailed business needs and provides developers with a clearer basis on which to construct the solution. Business Analysts, Developers and Testers meet regularly to clarify requirements.
One deficiency of Agile development is a lack of detailed functional requirements. Developers traditionally draw knowledge of what's needed from these and other detailed design documents. Also, because of the short development cycles (e.g. a typical "Scrum" Sprint is 4 weeks), there's a tendency to focus on developing and unit testing individual pieces of functionality with less attention paid to the overall business workflow and rules, Sometimes important elements of end-to-end functionality are missed. Use of detailed end-to-end examples helps to fill these gaps. They also add a layer of traceability that gives stakeholders and the business greater confidence in the development process.
- Unit Testing is conducted by developers, typically as a by-product of Test Driven Development (TDD). Tools such as JUnit are used to automate these tests and they are included in the Continuous Build Process for ongoing Regression Testing.
Traditional testers don't normally get involved with Unit Testing. They should however have visibility of the Unit Testing process, confirming that developer testing is in line with an agreed and acceptable approach. Try to put yourself in the shoes of the stakeholder. What assurances will stakeholders be looking for as the project moves forward? Confirmation from Quality Assurance that developers are properly testing what they build is one of these.
- Calculation examples are (often) the simplest and most common form of tests with each one represented by a single table row that includes both input and expected output. The same style can also be used for decision-type tests (e.g. with Yes / No output).
- List examples produce sets of results i.e. multiple rows. They're more complicated than calculation tests with expected outputs varying depending on filtering, duplication etc.
- Process examples allow the execution of steps and an assessment of success/failure along the way. They are typically used to test workflow.
- Mixed examples are the reality of Agile Testing. A way to build up more complex test suites that accurately reflect the execution of complete user stories and epics.
- Fixtures allow communication between test examples and the application under test. How fixtures are developed and used varies depending on the Testing Framework used. In its simplest form a developer extends a base class (these exist for each of the test types: Calculation, List and Process) and codes in the relevant attributes and method names to facilitate linkage.
The development of Fixtures is a potentially onerous task and requires programming skills. There’s a real need out there for testers who have a coding bent!
- The Testing Framework provides the infrastructure for execution of business-facing tests. It scans the tables of test examples, instantiates the relevant fixtures and requests execution against the Application Under Test (AUT). On return it completes a comparison of actual versus expected values and then logs the result.
There are several offerings in the market including:
FIT (Framework for Integrated Testing ... the first of its kind 2002)
Fitnesse (using FIT)
Fitnesse (using SLIM)
- One of the key advantages of combining examples with a Testing Framework is the ability to automate business-facing tests and include them in the Continuous Build. These act as Regression Tests ensuring that desired business functionality continues to operate correctly as changes are made by developers.
- Inclusion of unit and business-facing tests is usually a simple matter e.g. adding a reference to (or the example tables themselves) a directory structure that’s used as input for the build.
Agile Development Testing Context
The context within which Agile Testing occurs varies from project to project depending upon the Agile methodology used and the delivery and quality assurance requirements of individual stakeholders.
Figure 2 illustrates a common Agile Testing situation in the context of Scrum-based Agile Development using Sprints. User Acceptance is conducted following release of significant areas of functionality.
Figure 2 Notes
- Sprint Zero is a preparation sprint, a time to sort out setup of the Testing Framework, test environment, how each sprints will work etc.
- The makeup of a sprint consists of initial planning and then the day to day development and creation of business-facing tests. It culminates with manual regression testing, presentation to the business and refactoring of code and tests.
- Releases typically consist of end to end functionality that marks a significant milestone for the business. Functionality is deemed to be complete in terms of business rules provided.
- User Acceptance Testing (UAT) is checking by the business to verify that they’re getting what they want and have paid for. The value of business-facing tests comes into its own here. As input to UAT the business may even decide that these tests are sufficient on their own to cover off the verification. The bottom line is that the extent of UAT conducted is up to the business.
- The product backlog is a list of user stories requested by the business. It typically gets larger, not smaller, with the business adding more requests as the project progresses. At the end of the project the product backlog often contains undelivered stories.
- A planning session is conducted at the beginning of each sprint to decide which user stories will be developed and the tasks, including resources and priorities, involved. It is normal for user stories to be grouped into epics for this purpose.
- The Sprint Backlog typically consists of the developer tasks to be completed during the Sprint e.g. test preparation, coding, testing.
- A selection of user stories, usually combined into Epics, is assigned for the creation of business-facing tests.
- There will be a significant number of business-facing tests that require too much effort or cost to automate. These should be prepared to assist with development and executed either towards the end of the Sprint or at some time prior to release for UAT.
- A retrospective is an opportunity for the business to see the delivered functionality for the Sprint. It also acts as a UAT primer, giving the business a chance to vet what’s been done and request changes.
- Time is usually allowed towards the end of the Sprint for rationalisation of both programming code and tests.
Frequently Asked Questions
- Who creates the fixtures?
Either a developer or, and ideally, a tester who has programming skills. There's a real niche out there for traditional QA-style testers who can take the load off developers and develop their own fixtures.
- What are the biggest traps?
The biggest trap is trying to do traditional system testing with Agile development i.e. waiting for a code-drop and then testing against it. It doesn't work because traditional system testing relies on the existence of functional specifications ... that typically don't exist in Agile development.
Another trap is trying to automate "all" of the business-facing tests. Some of them will be just too difficult to build fixtures for in which case you should simply allocate time towards the end of each sprint for manual testing.
- What's the reality? Is Agile Testing truly viable?
It's more than viable. Creating examples of business rules and using them in place of functional specifications to aid development, as well as being able to test what's built ongoingly, gets results faster and to higher quality.
There's much for QA-style testers to do both in creating (and agreeing with the business) the business-facing tests and also helping to integrate / automate the tests with the application under test.
Fixture creation is often not straight forward and "simple" tests are rarely the norm. It's common to have a combination of calculations, set verifications and process checking all in a single test. So get ready for some complexity!
- What else should QA-Testers be doing?
There are books written that cover "the whole" of an Agile Tester's lot including areas such as Exploratory Testing and non-functional test areas e.g. performance testing, that Agile development tends to overlook. Avoid getting caught up in these areas if you're just starting out. Putting the business-facing tests together and automating and including them in the continuous build process has to be front and centre.
- What framework should I use?
For me GreenPepper is the way of the future. It's well documented and well connected in terms of associations with recognised vendors (i.e. Atlassian). It's a good fit with tools like Jira. I also like the fact that you have to pay for it (albeit nominally for small businesses) which brings with it support and a better future (possibly).
There are functional reasons for my preference as well e.g. its editing facilities are better and the fixtures are completely hidden from the test-writer which is a good thing if developers are going to create the fixtures.
By all means check out some of the other offerings that are out there including Fitnesse, FIT, SLIM and Concordian.
- How does this measure up to standard System Testing?
I point to things like traceability and practical input and integration with development as key ingredients to both creating quality applications and satisfying stakeholder concerns.
Creating examples for the business rules as part of the development effort creates not only good input material for the developers and ongoing regression testing during (and after) development, these also trace back directly to user stories and, in turn, business requirements. So business-facing tests stack up well in terms of traceability and they also lead in favourably to the UAT process (see below).
- Are business-facing tests the same as UAT?
UAT means different things to different organisations depending on their circumstances. The business-facing tests created by QA-style testers provide a great starting point for the business to do their Acceptance Testing. And if the business are able to use the same table-based standard for the definition of further "UAT" tests, there's no reason why they can't also be automated and included in the build process.
So, to answer the question, the business-facing tests created by QA-style testers "could" indeed be deemed the same as or at least part of UAT if the business is satisfied that they provide sufficient evidence of what they expect the application to do.
Alternatively the business can prepare and execute completely different tests to satisfy themselves that all's well.