|
|
|
|
|
James Bach,
co-author of Lessons Learned in
Software Testing
, is a respected name
in the world of testing because
of his new and unconventional ideas. You can visit his
website at http://www.satisfice.com.
WhatIsTesting interviewed him
recently. We hope that this interview will be useful for
you. |
| Q1.
James, please tell us something about you, your background and experience.
What are your interests other than testing and what do you do
with your free time? |
I was born in
the state of Iowa, USA. I spent my formative years mainly living in
Vermont, a quiet, rural place.
I had a terrible time in school, after fifth grade or so. I had read
in the U.S. Constitution that "Neither slavery nor involuntary servitude,
except as a punishment for crime whereof the party shall have been
duly convicted, shall exist within the United States, or any place
subject to their jurisdiction." So, I decided that meant I was not
required to do homework. My rejection of homework grew into a general
rejection of all educational bureacracy and presumptuous authority.
It all seemed absurd to me. When I was 16, in 1982, my father (who
wrote a book called Jonathan Livingston Seagull about the power of
going your own way in life) urged me to drop out of school entirely.
So, I did, and continued education in my own fashion.
I got a job at
a computer store-- where in six months I believe I did not successfully
sell a single thing to anybody-- but soon was hired away to write
video games on Apples and Commodore 64's. A few years later I was
hired at Apple Computer to be a test manager. I believe I was the
youngest manager at Apple, at the time. I had no management or testing
background, but that didn't turn out to be much of a problem. Few
people there knew much about testing, anyway. I decided to dedicate
myself to becoming a testing expert, and spent many hours in a coffee
shop across the street from work, reading textbook after textbook,
trying to discern the secret of testing.
So, I have an anti-establishment
pedigree. I don't feel obligated to remain confined to any one discipline.
This has advantages and disadvantages. I was never part of the system
that indoctrinates people into the belief that some guru somewhere
has found a simple recipe for developing software. I believe there
are no such gurus. There are no simple recipes, unless you count the
simple recipes that require great skill to fulfill, such as "use your
mind."
I have reverence for two things, the scientific method and
the happiness of innocent people. In everything else I am an energetic
skeptic. |
|
Q2. Your Web Site is named
satisfice. What is the significance of this name as well as
the words 'Epistemology for the rest of us' used on your
site?
|
To satisfice
is to find a solution that is good enough. The word was coined by
Herbert Simon, who won a Nobel prize for his investigations into how
organizations make real decisions. Complex decisions, such as whether
a product is ready to ship, are not made with perfect rationality,
but through that foggy process Simon called bounded rationality. My
business and all my work is founded on that idea: How can I make an
excellent decision, even though I don't have all the facts and even
though I'm imperfect as a thinker? That's what interests me and my
clients.
Epistemology is the philosophy of how we know what we know.
Science is founded on epistemology. "Epistemology for the rest of
us" is a play on Apple Computer's old slogan "computers for the rest
of us." Like Apple, I am trying to bring something out of the realm
of specialists and into common everyday use. That's where I'm going
with my philosophy of testing and software development. |
|
Q3. James, this may be a
short question with a long answer. Can you describe your association
with context driven school of software testing. How did it all start?
How did it mature? Please tell us something about the history and
the future that you see. |
"Context-driven testing" is a term coined over dinner, on November
21, 1999, by Cem Kaner, Brian Marick, and I, while we were attending
the eighth meeting of the Los Altos Workshops on Software Testing.
Two weeks later, we established the context-driven testing discussion
list on Yahoogroups. In 2001, Cem, Bret Pettichord, and I formulated
seven principles of context-driven testing and published the first
self-described context-driven testing book, Lessons Learned in Software
Testing.
Although we didn't coin the term until '99, what we call
the context-driven testing community emerged as an identifiable phenomenon
with the advent of the LAWST meetings in 1997. That was a watershed
event. LAWST stands for the Los Altos Workshops on Software Testing.
They are non-commercial, facilitated, weekend-long discussions where
small groups of peers come together to share testing experience reports.
Cem Kaner and Brian Lawrence founded the first LAWST. There have been
almost forty meetings of LAWST or LAWST-like spinoffs with acronyms
too numerous to list, here.
In a LAWST meeting, no one who presents
an idea is protected from cross-examination and criticism. The critical
process of LAWST requires each attendee to learn how to discuss the
underlying reasons for their choices, and the context-specific variables
involved. People who came to LAWST looking for "best practices" were
often disappointed to find more questions than answers, and more controversy
than consensus. Many people attended a few meetings and then dropped
out, but the ones who continued to meet gained skill in the art of
test methodology. So, LAWST and its sibling peer conferences are largely
responsible for the rise a new sensibility that says any practice
might be a good practice, and any practice might be a bad practice,
depending on the context-- a context that is always dominated by the
people who populate it.
Context-driven testing, then, is not a technique,
it is a professional attitude toward techniques. To "do" context-driven
testing is to establish a conscious, explainable, self-critical relationship
between your test process and the context in which you are testing.
A context-driven tester desires to be fully responsible for his own
process of testing.
I see the future of context-driven testing developing slowly
as like-minded scientific thinkers come together
in small peer conferences and try to write insightful articles about about how to think about testing.
I don't think there will ever be more than a small
percentage of testers in the world who seriously consider themselves
context-driven, just as there are very few ham radio enthusiasts compared
to casual cell phone users. In the far future (30 years or so) I believe
the values of context-driven testing will supplant the other schools
of testing and be recognized as simply "skilled testing." The context-driven
label will become unnecessary, because educated testers will consider
it obviously true that context dominates practice. |
| Q4.
There is another piece of history that we would like you to shed some
light on. Please tell us something about exploratory testing and your
association with it. Where do you think exploratory testing is heading
towards? How does context driven school embrace exploratory testing?
|
Exploratory testing
is a particular family of test practices defined as "simultaneous
learning, test design, and test execution." It can be thought of as
a mental martial art. The term was coined (in the software field,
anyway) by Cem Kaner in the late eighties, but he didn't write very much about
it, and few people used the term. I picked up the term in 1994 after
trying and failing to convince people that ad hoc testing is a teachable,
sophisticated form of testing. Ad hoc does not mean "sloppy", but
most people seem to think it does, so I figured exploratory testing
would be a better term for my purposes. Today it's a relatively well
known term, and Cem and I are known for our work in this area.
Elisabeth Hendrickson has also done some striking work
with it, as has James Whittaker. Not many other people are actively
working on expanding and describing the practice, which seems to mirror the
situation with exploratory work in other fields.
Exploration is by
nature difficult to turn into a simple concrete recipe, so many managers
are frightened of it. I think exploratory testing is a foundational
approach to testing. In other words, I can't accept that a tester
is skilled unless he can do exploratory testing well, and it's the
first skill that I teach to a new tester who works with me. The basic
phenomenon of exploratory thinking has been studied in the contexts
of medicine, engineering, philosophy of science, economics, theorem
proving, game theory, creative thinking, art, navigation, education,
and artificial intelligence. And that list is just from the books I happen
to have read.
Exploration is a core part of agile software development. No person
can claim to be educated about testing, in my opinion, and yet maintain
the belief that exploration is just a luxury to be done only when
we finish all our scripted tests. So I find it hard to take seriously
those authors and consultants to continue to talk as if exploratory
testing is an undisciplined, unteachable, unmanageable process. I
hope they will realize that it's not 1985 any more and the craft has
moved on. |
|
Q5.
What is the place of scripted testing in context driven school? How
have your views on scripted testing changed over time?
|
The context-driven
school has absolutely no preferences for any practice over any other
practice, except inasmuch as a practice violates one of the seven
principles of the school as given at http://www.context-driven-testing.com
That means scripted testing has the same place every other family
of practices has: a context-driven tester strives to understand the
dynamics of the scripted testing and how it might be the solution
to some interesting problem.
A context-driven tester applies those
forms of testing that fit the situation at hand. As a context-driven tester,
for instance, I'm aware of nine specific reasons why I might want
to repeat a test, and that implies some form of scripting. Regardless of repetition,
there are other possible motivations for scripting. Still,
I would say that most testing on a typical complex test project should
not be very scripted if your goal is to find important bugs. Exploratory testing tends to find more problems
more quickly.
Scripted testing means you create the test before you
perform it, and you don't change it while performing it or as a consequence
of performing it. This is sort of the opposite of exploratory testing,
but I find that most testing includes some elements of scripting and
exploration at the same time-- in fact, the realization that most
testing combines both styles is the biggest way my thinking about
it has changed over the last few years. |
| Q6. What is your advice to the companies that
are stuck with the 'old-world-model' of testing? What should they
do that results in better software?
|
My advice is
pay attention. Learn to watch what is really going on. Be self-critical.
So many companies pursue test approaches based on theories of productivity
and quality that fall apart the moment they open their eyes and notice
that, for instance, 85% of bugs (a popular estimate in my experiences
with clients) are found through exploratory means. Or the moment they
notice that the test documents they create aren't actually helping
to improve testing, and rather draw a lot of time and energy away
from the actual business of testing. Many people assume that new testers
will benefit from reading old test documentation, but watch new testers
and you find this is almost completely untrue. Testers learn through
exploration and by being involved in projects with other people.
My
other quick advice is to read books by Gerald M. Weinberg. Particularly
his Quality Software Management series. |
| Q7. How did 'lessons learned in software testing' happen?
|
Cem Kaner and
I were at a peer conference dedicated to creating "patterns" of software
testing, and we became frustrated with the stilted and unhelpful formats
we were being asked to use to communicate wisdom. So, Cem had the idea
of writing a simple set of ideas, each with a compelling and memorable
label, that would comprise our pattern set. We drafted Bret Pettichord
into the project and got to work.
At first, like all writing projects
I've been a part of, not much happened. But then Cem negotiated a
publishing contract and soon lit a fire under Bret and I to get it
done. We got together at my lab and brainstormed a list of 522 testing
ideas we wished someone had shared with us when we were new to the
craft. We divided these into chapters and assigned them to each other.
Writing the book was a lot like writing a bunch of emails to ourselves.
The design concept of the book is to provide a busy tester with a
book he can benefit from without having to read the whole thing in
a particular order. |
| Q8. What are the most important lessons that you think
every tester should learn, either from other's experience or own experience?
|
Here it is:
Testing Is In Your Head. That is, everything important about testing
is a consequence of how you think, what you know, what you see, and
what you remember. The human part of testing is fundamental, and
cannot be automated out of the process.
Most testing books, I now believe, are
written by people who don't know very much about testing. They have
experience, and so they think they know what it is. But the problem
is that testing is a very subtle craft. Like psychology, it evades
simple descriptions and analyses. So, I guess another lesson is
Most Testing Books Don't Tell The Truth About Testing.
Therefore, learn
these skills:
- How to observe
what's going on
- How to use the
scientific method
- How to think
systemically
- How to deal
with people
- How to be a
part of a self-critical community
|
| Q9. Are you working on any other book?
|
I periodically
contribute to Kaner's next edition of Testing Computer Software, and
I'm idly working toward a book on how to give yourself an unconventional
education. |
| Q10. What do you mean by "Sleeping Giant" when you
refer to Indian Testers in your blog? How do you think they can '"wake
up"? |
I mean that I see
no significant obstacle to India becoming an internationally recognized
powerhouse of skilled testing, whenever people in India decide they
want to do that.
Before I visited India, I thought there were educational
or cultural barriers to becoming skilled testers. I was wrong about
that. Please, testers of India, accept my apology.
I don't know what it takes to wake people up. I wonder how it's possible
that I wake up each morning. One moment I'm asleep, the next I'm awake.
All I can say is that the people I dealt with at the two Bangalore
companies I taught started my class with what seemed to be a dull
and limited view of testing. But as the class progressed, I noticed
a remarkable improvement in their ability to solve the puzzles I set
for them. I also noticed, in those particular classes, a lot of energy
and enthusiasm to learn about testing. I guess what it may take is
for influential people within the Indian technical community to see
the possibility of a competitive advantage in becoming viewed as leaders
in the worldwide testing community, instead of followers. Right now,
I know of no one who speaks of India as leading in any aspect of the
intellectual practices of testing. That will continue to be true until
at least of few of you start showing up online and participating in
the various communities, not as silent lurkers, and not as people pushing outdated ideas from
the eighties (there are already lots of people doing that) but as
practitioners contributing to the cutting edge of testing thinking.
That cutting edge, I believe, has to do with agility, skill-based process
imnprovement, testing heuristics and psychologically sophisticated
theories of testing. |
| If you were your own interviewer what questions would
you like to ask yourself? |
They
are: |
| 1. What is it you spend most of
your time doing? |
I am on the road,
most of the time, teaching rapid software testing and risk-based software
testing classes and attending peer conferences. Occasionally I work
on a court case as an expert witness or trial consultant . |
| 2. What would you most like to be doing?
|
I wish I was
doing more testing. I get tired talking about it all the time. The
most fun I've had on a project was when I was hired to test Windows
XP Embedded as part of the Microsoft antitrust case, in 2002. It's
a crying shame that I am not allowed to write about what I discovered
during that testing process, or what methods I used. Maybe someday
I'll be released from my confidentiality agreement. |
|
