An Interview with Lisa Crispin

At a time when many XP people thought they could do without a tester, Lisa Crispin wrote the book on it.

An Interview with Lisa Crispin by Magnus Ljadas, Citerus AB

Lisa Crispin is a tester with a mission – to explore best practices for agile testing – as she calls it. At a time when many XP people thought they could do without a tester, she wrote the book Testing Extreme Programming with co-author Tip House. Today, she says, developers are becoming increasingly more interested in testing. At her presentation about test-driven development at JAOO this year, the majority of the audience consisted of developers.

Magnus: What was your most compelling reason to write a book about testing?

Lisa: When I joined my first XP team in July 2000, I read all the available literature on XP (at the time, that wasn’t all that much). There wasn’t much to guide me as a tester on an XP team. Participating in the extreme programming mailing list, I found that many people doing XP felt they didn’t even need professional testers; they thought customers could write the tests and programmers could automate them. So who needs a tester?

My team and I learned by trial and error many ways that I, as a tester, added value to the team. I shared my experiences at conferences and on the mailing lists. So many other testers asked me, “Is there a book I can read to learn how to be a tester on an XP team?” that I decided if nobody else would write the book, I would try.

Fortunately I had a great co-author. It was a great deal of work to write the book while working a full time job, and I kept hoping some actual expert would start a book so we could stop. But nobody else wrote one, so we kept working. I have been gratified that so many people have told me the book was helpful to them; it made all the hard work worthwhile.

Magnus: You said in your book that one of the most valuable skills you can learn is how to ask for feedback. Why is asking for feedback so important?

Lisa: Just as feedback helps the team make continuous small corrections to keep the project on course, feedback helps me, as a tester, know if I’m helping the team the best way I can. For example, my current team decided to use customer tests (aka acceptance tests) to help drive development, along with the unit tests. The customers and I wrote detailed test cases in FitNesse at the start of the iteration, which we thought would be helpful to the programmers. However, when I asked the programmers if this was helping, we found there were two problems:

1. The tests were too detailed and complex, so the programmers had trouble grasping the basic requirements and the ‘big picture’.
2. Because the tests were in such a detailed level, but we didn’t know yet what logic would be in what layer of the code (eg., the database layer, the ‘black box’ code layer), the tests were not organized correctly and major refactoring was needed before the programmers could write the fixtures. This caused a bit of a testing bottleneck.

If I hadn’t asked for feedback, the programmers might have complained about the tests eventually anyway. Because I was proactive, we were able to have an open and honest discussion. Now we’re trying a new approach: we will write only high level test scenarios or descriptions for each feature to start with, and only write the detailed test cases as the programmers begin work on that particular code. That way the programmers have enough information to start the code, and we don’t waste time having to reorganize and refactor tests so much. As we try this, I’ll ask for feedback on how it is working. Because I’m shy and don’t like to interrupt busy programmers, and also I don’t want to look stupid, it can be hard to ask for feedback. That’s where courage comes in!

Magnus: Asking for feedback implies there must also be somebody giving feedback. As a programmer, how can I give feedback in order to make testing easier?

Lisa: That’s a good question, I had to think about this for a while. The feedback I need from you as a programmer includes:

• Do the test cases contain an appropriate level of detail? Do they help you with writing the code? What might I change to make them more useful?
• How can these test cases be automated? Are there some that are best automated as unit tests? Can we test behind the UI? Are they written in a way that lends itself to the automation tool we are using? Are we using the right tool, should we consider another approach?
• What, in your opinion, are the riskiest areas of the code we are working on and testing? Where should I focus my test cases, and my exploratory testing efforts?
• What pieces of logic are in which layers of the code? Are certain decisions made in the selection from the database, or in the back end logic? Is there logic in the user interface itself?
• Is there anything you can do to make the code easier to test? For example: Can you minimize client-side logic in the UI? Can you mock out the database for some of the testing, to make it faster? Can we write test fixtures that add the data we need for testing, run the test and then clean up the data?
• Is there some issue that isn’t clear? Do we need to get some people together and discuss it? Is there something you want me to check out with the business experts? (It’s best for ‘customers’ and programmers to communicate directly, but if it will help or save time I can be a liaison).

And most importantly: please let me know if my work helps you – or not! I am gratified to hear a programmer say “Wow, I’m glad you found that bug, I missed it” or “As we write the fixtures for these test cases we are finding requirements we missed”. And if what I do isn’t helping, it’s hard to hear, but I need to know it so I can change.

As a tester, I don’t always have something concrete to show for myself at the end of the day, whereas a programmer usually has some lines of code he can be proud of. I might have written an automated test script, or found a bug, but maybe all I did was ask a good question that made everyone think. Each team member wants to know that she has contributed something of value, and a tester is no exception!

Magnus: You collaborate very close with the programmers. But what about organizations that let their programmers throw code over the wall to the testing department rather than have them communicate directly?

Lisa: All throughout my career as a tester, I have gotten myself and my fellow testers involved in development projects from the start. Even in traditional environments, testers can add value by participating in the planning meetings, and ‘testing’ documents such as the requirements documents. Maybe I’m just pushy, but even when we used waterfall process, I went to the programmers and asked “Do you have some little piece of code, or even a prototype, that we can start testing?” I always had good results; developers were always interested in having someone help find the problems. I think I’ve been lucky to work in environments where this was possible (and, as I say, I’m kind of pushy)!

Before I started in my current job, they were definitely ‘over the wall’, but it was such a small organization (only 3 full time programmers), it was fairly easy to show the value of things like specifying tests (at least high level ones) up front, delivering testable chunks of code so testing can start early in the iteration, and collaborating to automate tests. Well, collaborating to automate tests didn’t come so easily, until the programmers were given the job of doing manual regression tests every iteration!

In my opinion, you can’t do agile development if you’re throwing stuff over the wall. If this is happening, I would try to educate both programmers and testers, and show them the value of working together from the start. I would get them to try it for a few iterations and see how they like it.

I once joined an XP team where they forced me to be on a separate test team, but still wanted all the stories tested before the end of the iteration. Since the test team’s tasks and progress were not represented in the standup meetings or the iteration tracking, the programmers didn’t deliver testable code until the last day of the iteration, and we missed our release date. I persuaded the manager to try it “my way” for one release cycle. We testers became an integrated part of the development team, and the programmers all had to help with acceptance testing and test automation. We finished on time and everyone felt the quality was higher, so everyone agreed to keep using this method.

Magnus: I like the idea to have programmers do manual regression tests in order to motivate them to write automated tests. The chapter about manual testing in your book is only one page long, it simply says: “No manual tests”. You stretch this even further by saying “a manual test may be worse than no test at all”. Is it really possible to automate all testing?

Lisa: That chapter created a lot of controversy, because some people just look at that chapter, and they don’t read the next chapter.

There are many good reasons to do manual testing. If you have a very thin UI, it may not be worth automating the tests for it, because it’s unlikely to have bugs and you can manually test it quickly each iteration. Manual exploratory or context-driven testing is essential to find the most serious defects. Usability testing is of course manual.

When I have joined agile teams as a tester, the programmers were naturally focused on writing production code, and not too interested in automating tests beyond their unit tests. If XP hadn’t had a rule that all tests should be automated, and that the team is responsible for doing this, it would have been hard for me to get the programmers to help with automation – and to design for testability.

Test automation is hard. Many testers are fearful of it, and it isn’t usually first priority for programmers. Having a rule that you should automate means:

1. When you automate all the tedious regression testing, you now have time for productive manual exploratory testing. Otherwise, you spend all your time doing manual regression tests, which aren’t likely to find bugs.
2. By starting with the assumption that a test should be automated, and getting experience doing this, you make intelligent decisions whether particular tests should be manual or automated.

Magnus: This interview is coming to its end, but before we say goodbye, I would like to ask what the most important trends in testing are today?

Lisa: As far as the most important trends in testing today: The one big reason I would love to be able to do a second edition of the book is so that we could make use of the great new open source test tools that have come about in the last couple years. Tools such as Fit and FitNesse totally change how teams work with respect to testing. They force collaboration between customers, testers and programmers (or at least customers and programmers, but I think usually you need a tester in there too!) Another tool I really like is Canoo WebTest. These open source tools are constantly enhanced with useful features. They are programmer-friendly, unlike many vendor test tools.

The second big trend, which is related, is that now PROGRAMMERS are interested in all forms of testing. When I conduct workshops or tutorials on agile testing, many participants are programmers – often they are the majority of the audience. I recently presented at JAOO, a conference for developers. The track on test-driven development was well attended. The programmers I work with spend lots of time writing FitNesse fixtures, in addition to JUnit tests, and they see real value from this effort.

While I don’t think testers necessarily have to be technical or have programming skills, I do think the line between testers and programmers will blur in the coming years. Testing is becoming a natural part of software development, not a separate, add-on, ‘do what we have time to do’ task.

Magnus: It has been a pleasure doing this interview with you, thank you!

Lisa: It has been a pleasure thinking about these questions! Sorry to be so wordy. Now you see how I ended up with a book. ;->

Copyright© Citerus. Reprinted with permission.

Related Link Testing Extreme Programming available from Amazon.com