Agile Testing is Different

When people think of Agile software testing, they often think of testing in an Agile context. For example, testing on a Scrum team. Alternatively, some people think of Agile testing as being the set of quality practices and principles that have evolved out of the Agile software development movement, such as BDD, TDD, story testing, continuous integration, etc. Others declare: “there is no such thing as Agile testing, there is just Agile development, which is intrinsically quality focused”.

As much as there may be benefit in some of these perspectives, I’d want to suggest they are limiting and that there is indeed such a phenomenon as “agile testing”, which is not only distinctive but also hugely beneficial. A helpful way forward might be to avoid thinking about Agile (with a capital A) testing and remind ourselves that agile is a verb not a noun. A definition of agile testing then (the type of testing I want to talk about anyway) could be:

Testing in such a way that agility is increased.

This could be testing within an Agile project, eg Scrum et al, where undoubtedly it would be easier to implement. However, it could also be extremely beneficial in more traditional contexts.

Similarly, various principles and practices have emerged out of the Agile software development movement, but unthinking use of these techniques just “because we’re ‘Agile’” can lead to greater restriction rather than agility. In contrast, testing techniques associated with traditional approaches may be more appropriate in producing agility.

That leads to the question though: “What is so good about agility anyway?”. Why are we seeing that as something to aspire to? Shouldn’t we be more focused on delivering a valuable product, on time and on budget? Or, particularly as testing professionals, shouldn’t we be more interested in quality as a focus? The response, as the Agile Manifesto signatories concluded in 2001, is that being agile is the best way to enable delivering a high quality, valuable product in our day.

Earlier than 2001 though, plenty of other people had discovered the importance of agility. Way back in 1986, Hirotaka Takeuchi & Ikujiro Nonaka, in their ground-breaking article in the Harvard Business Review, observed:

“In today’s fast-paced, fiercely competitive world of commercial new product development, speed and flexibility are essential.”

Takeuchi and Nonaka made a number of suggestions in their article on how companies could be more successful by being more agile. One of their key suggestions was the need to “take up rugby”, using this metaphor to describe optimal delivery by a team that can adapt to problems and new opportunities as they arise. Takeuchi & Nonaka’s observations were that this agility was essential for success in the fast-paced, fiercely competitive world of the 1980s.

How much more so now, in the 2010s, is our world increasingly fast-paced and competitive? The widespread adoption of Agile software approaches in recent years is indicative of the recognised need for agility in this world.

However, the goal is not to achieve “Agile”, but to achieve our goal by being agile. As much as Agile software development approaches are a great way to do this, an ‘agile mindset’ in any context is far more important in my view. (I’ve certainly worked with ‘Agile’ teams who didn’t have such a mindset, and struggled to deliver anything).

Software testing, then, needs to consider how to support agility as much as all the other development discipline.

So, how does Test encourage agility? Firstly, by understanding that agile testing is different. If we just carry on doing what we’ve always done, “we’ll always get what we’ve always got” (to paraphrase Henry Ford).

Accepting that is the case, I believe there are five key principles for testing in such a way that supports agility:

  1. Encourage collaboration not independence

Independence has always been highly prized in software testing and for good reason. Open University research suggests that if a developer tests their own code they will find less than 50% of defects contained in it. It is common sense that the closer you are to building the code, the less objective you might be – even as much as you try to be. What we might refer to as developer bias. Independence is seen as the way to deal with this bias.

This is the kind of thing taught in foundational testing courses. For example, the ISTQB Certified Tester Foundation Level syllabus includes a section on the benefits of independence, outlining levels of increasing independence from developer testing to external independent testers.

However, independence is inevitably very restrictive. When people work in isolation, there is little scope for innovation and plenty of opportunity for miscommunication. Passing code/test reports/defect information from one professional to another often causes delay and limits the flexibility of the delivery team to respond to changes (particularly if some are based in another location, eg overseas). What independent testers gain in objectivity, they lose far more from being distant from the product. What we really want is for testers to really understand the product and what it is to be used for, but at that same time deal with dangers of bias.

The answer, as many Agile approaches have discovered is collaboration, rather than independence. Instead of getting a separate, independent view of the product and basing tests on that. A collaborative approach aims to encourage familiarity, which is important, but, at the same time, defeat bias – by getting diverse views of the product and what needs testing, from many different people from different disciplines, including, but not exclusive to, Test.

How does this work out in practice? In a project based around multi-disciplinary teams, this should be fairly straightforward. The team can get together regularly to discuss new requirements, thrash out acceptance criteria, specifications and test conditions – as a team. Thus avoiding bias and getting a better understanding of the product at the same time. Behaviour Driven Development, in particular, encourage early collaborative discussion around requirements, in order to drive all development activities, inc testing.

In more traditional approaches, where functional domains are siloed and testing conducted in specific test stages, it is not so straightforward, but it is still possible to encourage more collaborative approaches in this context. For example, testers can simply break down the barriers, by going and discussing code/defects with the code builders face-to-face, rather than communicating via documentation. Testers would be expected to be involved in early reviews of requirements, designs etc. Holding more workshop-style review events rather than formal inspections could be another good way to encourage collaboration.

External, particularly off-shored, test teams are harder to collaborate with, but not impossible. I have worked with a number of companies in recent years who have tried hard to make this work. Regular communication, via collaborative tools and video conferencing, is important, but also real face-to-face communication, even for just social time, seems to be essential to achieving that high level of collaboration we want.


  1. Testing for overall value not for defects

A more collaborative environment is important for improving agility in many ways, but one important aspect is raising the vision of the individual delivery team members to see the bigger picture. It can be easy to focus on individual tasks and ultimately fail to deliver anything valuable to the customer.

This is a particular challenge for those involved in Test. If my responsibility is to complete my list of test activities and I do that to the best of my abilities, then I am happy. If that means I raise a long list of defects and throw them back over the wall for the coders to deal with, then that’s their problem not mine. If I test the product thoroughly according to the requirements, but it’s not what the customer wanted, then I’ve done my job, so why should I care?

Hopefully, all of us do care. We want to do our own job well, but ultimately we want to deliver a high-quality product that the customer actually wants. Testing, whether in an Agile context or not, should always keep the bigger picture in mind. Finding defects is important of course, but sometimes it is important to spend more time with other disciplines to ensure the overall delivery, rather than worry about increasing our bug tally.

For some testers that is a sacrilegious thought, but it stems from a poor understanding of what testing is all about. Those testers who see themselves as a metaphorical ambulance service, dealing with the accidents of others before them, will relish the opportunity to find more and more defects. But this is a naïve and ineffective approach to testing. Instead, the metaphor should be something more like a warning sign. Working collaboratively with other disciplines to ensure a valuable product is delivered, to the appropriate level of quality required.

In practice, this involves prioritisation of testing activities, on the basis of risk and value to the customer (standard practices). However, the emphasis needs to be on understanding what is really important to the customer and not just what is presumed. Collaboration with the customer is therefore essential, but also on a regular basis, not just a one-off at the beginning of the project – the customer may well change their mind regularly in our fast-paced changing world.

It is also involves more end-to-end ‘system thinking’. For example, conducting root cause analysis on defects to identify quality prevention measures.


  1. Focus on testing hypotheses not software

Software testing is obviously a software development discipline. Testing involves the planning, design, execution and reporting of tests run on software. However, when the focus is on the task of testing software, then it can obscure the fact that what we’re doing in testing is actually testing hypotheses. We want to focus on Why we’re testing, not What we’re testing.

This is an important aspect of agility. We don’t want to get bogged down in ‘running tests’ for the sake of it unnecessarily. Maximizing the amount of work not done is an important art for all disciplines, including testing. A focus on testing hypotheses moves the emphasis away from creating vast quantities of test artefacts or vanity coverage metrics (Test Cases Run Against Test Cases Identified et al), to more intelligent testing of what actually is important.

For example, say we decide to provide a particular piece of functionality to enable the user to log on with a username and password to our software. The underlying hypothesis may be that the customer will get benefit from being able to perform this activity. Rather than construct a load of test cases to check for boundary value analysis and equivalence partitioning of the username and password input fields at unit and system level, etc, we might do better to test the hypothesis that this functionality is actually required. Perhaps performing A/B testing with different groups of customers could try out some different ui options before committing to a solution that may be unneeded.

A focus on testing hypotheses also supports the previous two principles of collaboration and considering overall value. When we think of testing narrowly in terms of types of test, then we often assign the responsibility of certain types to certain teams or domains, which leads to isolation and selfish thinking. For example, developers are responsible for ‘White Box’ unit testing and testers are responsible for ‘Black Box’ system testing. As much as this may be broadly appropriate in some cases, it limits agility and could prevent us from delivering effectively. Coders may well get a lot of benefit from having tester input in unit testing for example. Or, we may decide that other quality assurance measures such as pairing or continuous integration will help test our hypothesis better than extensive system testing.


  1. Being a test specialist not the owner of test

As we’ve discussed, a more collaborative environment is conducive for agility. This is easier where we have cross-functional teams, but we can get benefit from more collaboration in traditional, siloed contexts. However, in my experience, it is often the case that the people who are most resistant to collaborative work are testers, unfortunately. Testers often want to keep ownership of testing to ensure ‘it is done right’ and consequently avoid collaboration.

If that is not true for you, then that’s great, but you can see why it can be the case. Partly it stems from the traditional, outdated emphasis on maintaining independence and quality control, but it also stems from an understandable desire to want to promote testing as an important professional discipline (not just a development activity amongst many). Testers may feel that if test becomes more collaborative, with anyone and everyone involved in testing, then it could be dumbed down.

There is certainly a risk of a ‘Jack of all trades – Master of known’ phenomena if software development projects rely on testing being done by everybody, but the answer is not to resort to keeping test activities within strict domain boundaries. This inevitably restricts agility.

Instead we need to move from owning test to providing test specialism. If you consider the rugby analogy, few teams would do well if they didn’t have skilled individuals in particular positions, eg wingers are usually the best to run up the field with the ball, but sometimes another member of the team may need to step in, even if they’re not best suited to it.

The ability to have people with broad skill sets, who can adapt to changing circumstances is essential for agility. But, then, having skilled craftspeople, who are experts in their area and improving their skills is also essential.

Good organisations, whether playing rugby or delivering software, encourage the development of T-shaped people. That is, we develop deep skills in our particular profession and domain knowledge, (think T upright), but at the same time, we aim to broaden our skills into other areas, to enable agility. For example, a skilled coder would try to improve their testing skills and conversely, a tester might try to take on coding skills.

How do organisations encourage the development of T-shaped professionals? Organised training and coaching are always good, but often the key thing is simply to provide time. We are often driven by tight deadlines and taking time out to think about skill development, particularly in areas we are unfamiliar with, is considered a luxury. Taking time out from the day job to learn new skills and knowledge can be far more beneficial in the long term and is vital for building a highly agile workforce.

Much skill development can be done ‘on the job’ of course and pairing between disciplines can be a highly effective way of learning. The benefits of testers pairing with, for example, coders may have a number of benefits. Not just the testers and coders getting a better understanding of each other’s roles, but also potentially better quality work at the same time.


  1. Emphasis on sharing knowledge not documentation

Finally, good agile testing places an emphasis on sharing knowledge, not just passing information around in document form. Documentation is an important form of communication, but anyone who has worked in software development knows that detailed documentation can be restrictive and obscure large numbers of defects. Unfortunately, many projects simply insist on producing documentation for the sake of it, despite it contributing little to the overall delivery.

We can’t do without documentation, but accept that it is not the most effective way to communicate, so how do we emphasise sharing knowledge over just writing documents? One way is to question the production of documentation more. I can certainly remember being asked to produce test documentation for project managers, so that they could tick off an item on their todo list, but with little benefit to anyone else. If I was asked to produce a document, my first question would be “what information are you looking for?” A good rule of thumb is to produce just-in-need documentation not just-in-case.

Avoiding writing unnecessary documentation is one thing, but we also want to improve our knowledge sharing. One of the issues with documentation is not just that it can be nugatory, but that it can also be subject to different interpretations. An emphasis on workshops and face-to-face communication can help people to thrash out a problem, consider expert input and achieve consensus. Plus then when the results are documented, you can write less down because everyone involves already understands what it is all about.

The use of examples is another developing technique for encouraging good knowledge sharing. Rather than talk in abstract terms that might leave some people out of the loop, we can discuss particular concrete examples that help us understand better. This can be done at different stages of development. For example, while we are elaborating a requirement for a bank account functionality, we could talk about what would happen if a person, with a bank balance of +1000, tried to transfer money. We could then look at other examples to highlight assumptions and acceptance criteria, eg a person who is overdrawn.

Good use of test automation can also help share knowledge effectively while creating a high degree of agility. One of the biggest issues with documentation of course is that it often needs ongoing updating, and can get out of sync with the software it describes. Clever use of automation can result in ‘living documentation’, ie writing automated tests (or better described as executable specifications) in such a way that they are understandable by everyone, including business users, means that the ‘documentation’ describing what the functionality does (and whether it is working) is automatically updated every time you run the test.



Charles Darwn is often quoted as saying:

It is not the strongest of the species that survives, but the most adaptable

Although it is unlikely he did say it, it’s still a recognisable truism and it is simply an observable fact that in our current world, agile companies succeed. In software testing, being agile is obviously going to be far easier in an environment that is trying to ascribe to Agile software development principles, but that does not mean that we shouldn’t aim to be more agile in whatever context. The goal is not to achieve “Agile”, but to achieve our goal by being agile. This can, and should, apply to all areas of delivering software, including Test.