schools of testing

[Cem Kaner]I wrote a blog post on this at http://www.satisfice.com/kaner/?p=15 that you might find helpful

The notion of a “school” has little to do with what techniques people use. Any school can use any technique (unless, for some reason, that group has decided that, in principle, the technique is flawed or unethical).

The notion of a school has more to do with your foundational thinking of why you do what you do. And from that, it drives your strategy, it drives the way you educate yourself further, and it drives your notions of success and failure.

Take a look at the exchanges I’ve had with Don recently. The differences there are not about techniques. They reflect fundamentally different views about the role of testing in a product development effort, the ethics of test management, the definition of success, the perceived professionalism and motivation of the programming team, the definition of quality, and so on. It is easy for us to talk past each other because our basic assumptions are so different that it is hard for us to notice them.

For example,
· I think that testing is primarily about learning and I am deeply interested in ways to make testers better learners.
· Other people think testing is primarily about techniques, more a mechanical activity than a highly cognitive inquiry.
· The implications of this difference are fundamental. For example, think about the recent thread involving automation of “mainstream tests.” One of the first tasks I give a tester new to a project is to use it in ways that help the tester understand what the product would do if it worked really well. James calls this “sympathetic testing.” The goal is not to find bugs, not to regression test, not to smoke test, but to gain an appreciation for the product that helps the tester THEN hunt more effectively for problems. You would only reserve time for this type of work if you thought that it is important for the tester, this new human on the project, to learn this kind of information and that this time-consuming task is a good way to foster that learning. If you were less worried about developing the knowledge and judgment of the tester, you’d probably just automate these tests. From my point of view, that sabotages the effort because the human doing the testing will probably think about the product with less depth and thus less effectively prioritize later work and less effectively interpret and communicate later test results. But from the point of view of someone who thinks testing is about using well-understood techniques to expose classically-defined errors or deviations from a well-written specification, the automation saves time and frees the tester to apply more techniques to more code. Who’s right?
It is hard to get people to articulate this difference, it is often hard for people to articulate it to themselves. They said instead, “testing is testing, and I think testing is about what testers do” and then they talk about the broad mix of activities they do, rather than about the underlying intuition they have about how to prioritize them.

I don’t think that people consciously join schools until they are senior enough to start asking, “why am I doing this?” and “how do I make decisions when decisions are difficult?” In those situations, professionals come up with guiding principles or guiding heuristics or guiding attitudes that influence the rest of their work.

My goal, in abstracting out the “schools” was to see what the common threads were that characterized groups of people who seemed to have reached the same principles. My primary goal, initially, was descriptive.

When I was a student in psychology, the contrasts between “schools” in that field were useful for exam study—we could organize clusters of thought in ways that the professor would grade well. And they were useful for putting some of the research we studied into context. But undergraduates who puffed their chests out and said, “I am a behaviorist!” or “You behaviorists are all stupid! I am a Gestaltist!” were (usually) playing with the ideas. Trying them on for size, experiencing them, but not really committed to them. As a doctoral student I ran what was expected to be a “safe” set of experiments (boring but original in the sense that no one had ever done this particular boring variation on a class of boring surgery-plus-maze experimental situations) (it was like applying a same-old-boring test technique to a part of the program that hadn’t been tested this way before, expecting to find the same types of bugs that this technique usually exposes on new code. Zzzz Zzzz Zzzz) But instead of giving us the expected results, our rats behaved in entirely unexpected ways. At this point, I was nearly fired from the lab for incompetence. Instead of backing off, friends and I set up video cameras, scripted the experimental method more carefully, and showed that the rats really did behave this way. Then I simplified the experimental situation and showed that they STILL behaved that way. That convinced the head of the lab that I was on to something. UP TO THIS POINT, everything I was doing was about technique. To the extent that theory was involved, it was the professor’s, or the senior doctoral students who were coaching my work. But now I had a set of decisions to make and those decisions would determine how I would spend the next almost-a-year or so of my research time in this lab. Time is limited, resources are limited, lots of ideas are possible, lots of experiments can be run, but if you don’t think carefully about why you want to go further and what types of things you want to learn and why, then in research, you are likely to drift in a sea of not-very-meaningful techniques. A lot of research papers come out that way, from immature researchers who spend their time doing the experiments and not much time on why or what they’ll learn or why they really care, but at some point, choices and constraints hit at the same time. I asked, “How would Edward Tolman have analyzed this situation and how would he have prioritized the next work?” (Tolman, see http://psychclassics.yorku.ca/Tolman/formula.htm, for an example of his foundational thinking) and from there, I was operating within the mental framework of one of the dominant schools of psychology, as it applied to animal learning theory. Had I adopted that school as my own? Not exactly. Was it like my religion? No. But I used it to structure my approach to the problems, to help me choose what to do next and how (what questions, what techniques), and to help me interpret my results. If I had adopted a different worldview, I would have done entirely different experiments from the same starting point. If I had simply followed a pragmatic course, I would have done what several colleagues (other grad students, a post-doctoral researcher, and one professor) advised me to do—put this line of work on the shelf and go do something else that was safer and more likely to yield easy publications in a reasonable period of time. I adopted a theoretically-driven approach and had the good luck to get some really interesting data. (You can see details of that experiment in my video lecture on scripted testing at www.testingeducation.org/BBST).

People can drift through their work for a long time without facing really tough choices. They don’t have to adopt a theoretical framework, or develop one for themselves, because the next decision and the next one and the next one seem pretty obvious. Not easy. But obvious. In practice, these folk pick up a smattering of ideas from the different competing viewpoints in our field. Some of the things that people like Don say sound interesting. Some of the things that James or I say sound interesting. We all (well, most of us) describe our techniques and some of those seem useful, and so, in practice, most testers I have interviewed and most test groups I have worked with have operated in a way that they considered largely atheoretical, pragmatic, and (to varying degrees) eclectic (willing to adopt a variety of different techniques and approaches to developing test ideas).

At some point, as a manager, you might hit a set of situations that are difficult enough, and feel important enough, that you start thinking at a more strategic level. Why are we in this situation? Whose opinion counts about what services we are supposed to provide, what information we are supposed to report, what information we are supposed to NOT report? How hard should we fight, over what issues? What should we do next and how can we explain it? How much more time and money do we REALLY need to do an adequate job, and what would “adequate” really mean in our situation, and how can we explain it?

Imagine you finally come up with some answers. If you’re running a group, say 6 testers (or 60), some of those people will be doing things that don’t seem like the most important things to do under the strategy you have finally thought through. Some of them can’t explain their decisions in ways that make sense to you. Some of them seem determined to do stuff that you consider wrong-headed, unlikely to yield any value at all.

At that point, whether you realize it or not, you have probably adopted a viewpoint, a way of thinking about your project, your tasks, your staff, your priorities and your values, that maps to one of the schools. Maybe not. But it’s worth considering. What we tried to do was to bring them out into the open. So that in some critical situation, you could say, “I’m not sure what to do next, but in principle, most of my decision making is more consistent with this way of thinking than that way of thinking. So in my copious spare time (3 hours) that I can spare to do some reading about how other people approach problems like mine, I think I should look first at this person’s writing, at this group’s ideas.” At that point, whether you have fully bought into it, you are operating within the context of a school.

Now, let’s consider your questions:

1.What happens when people of different schools work together?
2.Can we cross-fertilize between schools?
3.Do I have to pick a specific school?

1.What happens when people of different schools work together?

It really depends on the people involved. Doug Hoffman and I started a long way apart from each other. We decided to teach courses together and consult together, partially to gain insight into each other’s views. We learned a lot from each other. In some of our courses, our students had a fun time because one of us would lecture for a while and then the other one would butt in and contradict many of the ideas presented. They got a richer course, and in long debrief meetings after class, Doug and I worked through a lot of stuff. In this particular case, the experience (which continued over several years) was transformational. Both of us ended up in a different place. I abandoned some of my old ideas; so did he.

In small test groups, it is harder to manage testers who operate from fundamentally different perspectives. This often turns into what looks like a personality conflict, as one person described what the other person does as stupid and wasteful. (Some people say this more politely, but it can be very hard.) The problem is not just that they are doing different tasks or using different techniques or even pursuing different priorities. The problem is that their ethical analyses are different, their ideas about who is to blame, what it means to improve, who has to be retrained and why, who should be told what about the project, what autonomy individual testers should have, what control the project manager should have, what documentation is sufficient, what constitutes professionally competent work and competent recording of the work—when people have fundamentally different visions of the work they are doing:
· Sometimes (probably the most common case), one group quits
· Sometimes (not an uncommon case), the manager adopts one view or supports one faction and makes the other folks conform (and eventually, people quit)
· Sometimes people move into competing workgroups and we have bitter corporate politics
· Sometimes a particularly charismatic manager identifies the foundational differences, brings everyone together, lays them on the table, gets people to articulate them as their own, and then persuades the groups to tolerate each other:
We don’t know which is the right way, we have an impossibly complex problem/project in front of us, let’s agree to approach it from different directions and see what happens. In my limited experience, success here doesn’t just come from charisma (whatever that is), it comes from humility and the ability to convey that humility. This really IS a problem bigger than what we know how to solve, we really SHOULD accept the idea of our own fallibility, and agree to do the best we can while suspending judgment on an approach that looks less promising to us but, with our imperfectness of knowledge, might be more useful than we realize.
I have seen this, I think, but not often.

I am sure there are other cases, but they are outside of my personal experience.


2. Can we cross-fertilize between schools?

Yes. I’ve learned a lot from people whose approaches are fundamentally different from mine. In a few cases, other people have learned a lot from me, despite their fundamentally different view.

More often, I think people learn the superficial things and miss the underlying ones—learning very little. Transfer of techniques is pretty trivial. Transfer of ways of thinking, and transformation of them, is harder and much slower.

I think most of the cross-fertilization that I have seen has come from intentional, explicit effort. Often it happens at the individual level – two friends, or people who become friends, decide to take the time to work together closely enough to really appreciate the other one’s art.

3. Do I have to pick a specific school?

To the extent that you adopt any foundational thinking about what you do, you’re in the territory of people who do foundational thinking.

You can read about lots of different viewpoints—Bret’s goal in publishing his talk was to articulate those differences in a way that would be useful for educators, to help their students appreciate the diversity of the field. (Bret’s original presentation was to the Workshop on Teaching Software Testing.) Coming to understand their similarities and differences is useful.

At some point, if you are to become a mature professional in the field, you have to adopt your own foundational views. I’ve given enough examples of foundational questions above that I won’t repeat them here. At some point, you have to come up with a set of answers, and approach reflected within that set of answers, that makes sense to you and that has some coherency.

Probably, at that point, whether you have intentionally adopted one of the schools or not, probably you have adopted the point of view of one of the schools.

The goal of a descriptive taxonomy (Bret’s, mine, others to come) is to capture the essence of most of the leading, rational, competing viewpoints. If it’s a good taxonomy, and you end up doing a good job developing your own viewpoint, your view should fit somewhere on the taxonomy.

As I mentioned in the blog post, my first shot at a taxonomy was technique-focused and it succeeded in the sense of leading to some interesting ideas, but it failed as a way of describing schools of thought. I like the new one better, and will continue to like it until we find a better replacement.

Not everyone belongs to a school:
· Newcomers don’t have enough experience or insight. Some people never develop enough experience or insight to develop or adopt a foundational view of their work.
· If you mean, do you have to take sides like voting in an election or choosing to murder or be murdered in the latest religious crusade, no. Absolutely not. I (me, personally, my choice) happen to think that the way of thinking that I have adopted is a really good one, and I talk favorably about it a lot. I also happen to think that some other viewpoints are wrongheaded, and I explain why. But that’s my choice, and it reflects my personality.
· Many people in our field read so little, or read so superficially, that they never understand what any of the schools is really trying to say. If you spend most of your reading time on mailing lists or qaforums and on magazines like stqe / better software that boil things down into short, simplified, presentations then you might never be exposed to enough serious professional writing to make sense of the schools anyway. That doesn’t mean you won’t adopt your own foundational views. It will mean that you’ll reinvent the wheel, probably with the same mistakes that the last N generations of people who invented this wheel made. In testing, I think that usually means a naïve version of the factory school or the control school.
· In some fields, some people are professionally eclectic. Different professionals have different reference examples for this. For me, a good example field is psychotherapy. Psychotherapists operate within rich intellectual traditions (not all of them—for example, some hacks are mainly focused on the business of psychotherapy than the service—but the ones I know have studied the intellectual history of their field and have their own viewpoint, and that viewpoint profoundly influences their work.)
o Some very senior people are professionally eclectic, and brilliant. The ability to say, “How would Freud have analyzed this situation?” in one case and “How would Jacque Lacan think about this?” for a different case – and really understand each one and really follow in the mental shoes of one for one case and the other for the other case – that ability is very special, and I think very rare.
o Some people say they are eclectic, but it really means that they haven’t found themselves yet, or that they’re not really good at anything.
o Some people say they are eclectic, but it really means that they are still thinking about techniques (and learning how to manage their practice or work as a therapist) and haven’t faced the life crises that make them ask themselves what they really believe. (I guess this is another way of saying they haven’t found themselves yet.)

=========
Schools are not about technical skill sets. They are about the foundational visions of the field within which people develop and deploy their skill sets.

Similarly, schools are not about specific contexts (such as types of applications). It is a damning criticism of a school, a way of demonstrating the superficiality of its thinking, to say that it applies onto to one type of software or one type of customer. A school that tells you to treat all situations in much the same way TACTICALLY (same techniques, same “best” practices) across different categories of product, customer or risk—well, to me, that’s a poor way of thinking. In other fields, that kind of intellectual narrowness relegated a lot of schools to history books. But then again, what answer would you expect from someone in the context-driven school?

-- Cem Kaner