|
|
|
|
|
Michael Bolton is an independent test
consultant. He provides testing related training and mentoring,
frequently contributes to discussions on various mailing lists, writes
a newsletter on testing and plays mandolin. WhatIsTesting interviewed
him recently to discuss various testing topics under the sun including
testing education, testing certification, CMM and things
that he
is currently working on. You can visit his site http://www.developsense.com.
If you want
to ask him more questions or you want to send any feedback on this
interview, please send it to webmaster @ whatistesting.com.
|
| 1. Michael, please tell us something about your self,
the path your career has taken and things that you do in your spare time.
|
Testing
and computer
stuff in
general are my second career, actually. In high school, I was on
a math-and-science track, but in my last couple of years I had a couple
of great English teachers and one math teacher that really didn't
inspire me at all, so I switched focus. At University of Toronto,
I worked on English major and a Theatre (that's the way we spell it
here in Canada) minor. After the kind of post-post-secondary jobs
that you'd expect from a liberal arts major--washing dishes at a comedy
club was one--I became a theatre stage manager, specializing in touring
shows for kids. You wouldn't believe how that prepares one for
dealing with software projects.
I
worked for Quarterdeck
from 1990 to 1998, the first four years in
Toronto and the latter four as a program manager in Los Angeles.
(I was born in New York City to Canadian parents, so I'm a citizen of
both countries and relocating was easier for me than it would be for
most people.) Quarterdeck made DESQview, a multitasker for DOS,
and QEMM, a memory manager. Those products were pretty
successful, and since they were technically excellent and interacted
with so many other products, Quarterdeck was a fantastically
educational place to work. As a friend said, if you didn't learn
five really interesting new things, it was a slow day. On top of
DESQview and QEMM, in the early 90s we had DESQview/X, the only
complete X Window implementation for DOS, where we turned DOS and
Windows into X Clients. Most people didn't understand the advantages of
that sort of thing then (and the rest have forgotten about them now),
and it took a lot of horsepower to run effectively--you needed 64MB of
RAM and one of those newfangled Pentium chips. So DESQview/X
didn't do well commercially, but it was brilliant technically, and it
did lead Quarterdeck into the Web very early on. That was seen to
be a Good Thing, because as Windows got memory management, QEMM's
market niche eroded. Quarterdeck was the first company to licence
Mosaic for commercial release. Our developers rewrote it into a
wonderful product, but we released it too late to make a difference in
the Netscape/Explorer space. We released a bunch of other
Internet tools soon after that, but things were pretty noisy in the
marketplace by then. I was managing QEMM and CleanSweep and some
other utility software at the time, but Quarterdeck was a small enough
company that we could all get exposed to everything--the strengths,
which were mostly technical and the weaknesses, which were mostly
business problems. Quarterdeck grew far too quickly by buying
other companies with Internet-based paper, which led to the company's
demise. I point out with some pride that Quarterdeck, as an
Internet company, flamed out in 1998, fully three years before it
became fashionable to do that. Quarterdeck was always ahead of
its time. (laughs)
One of the best things about Quarterdeck, for me, was that it
introduced me to Cem Kaner. Program managers at Quarterdeck were
essentially the chief quality advocates for each product. I was
promoted from testing to be the second program manager there, and
everyone after that came out of the testing group too. Cem came
to teach his Black Box Testing course after I had moved on from the
testing department, but I was lucky enough to be able to sit in on the
course. He was able to take all of the things that I had been
thinking about in terms of testing and quality and put them into
contexts that made sense to me. Cem was--and still
is--extraordinarily generous with advice and mentorship, and he also
pointed me towards Jerry Weinberg and James Bach, who are also huge
influences on my work and my thinking.
So now I'm an independent
consultant, and have been for six
years. I test software, and I teach other people how to test
software, and I write about doing both. Most of my writing
eventually makes it to my Web site, www.developsense.com.
That was the career
path part. For fun, I play mandolin at Irish
sessions around Toronto (and in other cities when I visit), I work on
trying to keep up with repairs and reconstruction of our family home,
Pandora's House, and I like bicycling--Toronto's a wonderful city for
that. I enjoy cooking, and reading about food and cooking--those
things have taught me a lot about testing, too. But the most fun
and the most educational thing in my life these days is my first
daughter Ariel, three weeks old today.
|
2.You publish a
testing newsletter. Can you please tell us something about it, why did
you start it and what are your plans regarding its future?
|
I
started writing the
newsletter mostly because I wanted to spread around some ideas about
testing and about the world in general that I think are
important. Although I do get the occasional flash of insight
myself, most of the ideas I'd like to spread are from other people--so
I'm really trying to be a curator of things that I've found to be
interesting or useful. Other people write blogs for the same
purpose. I wanted to do something a little more formal than that;
a schedule drives me to write in a more regular way than a blog
would. It's also a way of marketing my services, and, I hope,
adding to my credibility. I'll keep doing it as long as I get
positive feedback, and as long as I keep learning things as part of the
exercise. If you want to learn about something, in fact, I
recommend writing about it.
|
3.
To
which school of testing such as Process oriented, exploratory testing
etc. do you belong? Or are you unhappy with this classification?
|
I'm
perfectly happy--and
proud--to be a member of the Context-Driven School of Software Testing;
our manifesto is at www.context-driven-testing.com. The principal tenet
is that there are no practises or techniques that can be called "best";
there are things that might be more or less useful or appropriate
depending on the context in which they're used. The whole idea is
to do things that serve the testing mission and that serve the project.
We testers already have plenty to do; we don't have time for practises
that /don't/ serve the project.
I'm a reformed process-oholic. Quarterdeck was pretty typical of
hot, publicly-held technology companies in the mid-to-late nineties;
product release schedules driven by the need for quarterly results,
lots of mergers and acquisitions, lots of rapid expansion and
catastrophic layoffs. A bunch of us hoped that formal processes
would help to foster a bit of coherence. Some processes did help us in
the development department to get organized to some degree, which
wasn't a bad thing. But general systems thinking will tell you
what came next: the processes helped us to deal with an increasingly
absurd number of products and projects to the point where, as soon as
we were able to cope, the business piled even more projects on.
So we tried to get the business people to follow processes too.
We wanted them to follow /our/ processes. Robert Heinlein, I
think, said "Never try to teach a pig to sing; it wastes your time and
annoys the pig." No, I'm not calling the business people
pigs--far from it--but I am saying that some of them had a set of
priorities and imperatives that baffled and frustrated a lot of the
technologists, and vice versa. To the extent that we could get
good work done--and we did do some very good work, under heavy pressure
and tight deadlines--it always came down to negotiation and
collaboration between people on both sides with skills, understanding,
and good will. Rules or guidelines that got in the way would get
pitched if we could collaborate to make the project happen. The
big lessons there were that people, not processes, saved projects on a
regular basis; that processes might be great or terrible, depending on
the context in which we chose to use them; that processes and
documentation have to serve the needs of project and the business; and
that neither people nor processes can save a company that's too deeply
in the mire. In order to get something done, you need motivated
people. If you need to dig your way out of a hole, your processes
won't ever pick up the shovel.
|
| 4.
What is your take on quality certifications such as ISO and CMM? How do
you see them being used or abused, especially in relation to testing? |
I mostly see them as being abused, again
mostly because some believe that processes, rather than people, are
most important to the success of an organization. I recently worked on
a project that incorporated software from a company that claimed CMM
Level 5. I suspect that at one stage, one project or one
department managed to garner that assessment from somebody, and now the
whole company claims it. The software was terrible, just
rotten. Fixes were shoddy in all kinds of ways; half or more of
them failed.
In the early days, the
ISO 9000 series of standards were designed for manufacturing
contexts--repeated production of large volumes of tangible stuff--and I
think they're more appropriate there. Software development is not
that orderly; it's a creative, intellectual process, not a
deterministic, manufacturing process. As for the CMM, its
original context was for defense-related development projects, where
human lives are at stake, and where time, money, and manpower are all
available in quantity.
Are CMM practises
appropriate for every software development context? Certainly not; no
way. No single process model can fit every organization, and it's
inappropriate to think so, in my view. To the extent that the CMM
is ever appropriate, it's tailored for patient, affluent kinds of
environments; I think CMM processes would ruin most companies that had
to respond quickly to conditions in the commercial software market or
the Web world. The trouble is that, to some influential people
who are working in a chaotic circumstance, the CMM can have a seductive
appeal. Those people will try to impose processes on their
organizations--they'll teach many pigs to sing--but what if you need a
critter to help you dig up truffles? The singing would be
interesting, but it wouldn't help to find more truffles.
Jerry Weinberg defines
quality as "value to some person", and he goes on to note that the next
question is "who is that person"? The next step from there is to
note that decisions about quality are political decisions. The
CMM clearly has value to some people. So some questions, if
you're interested in the CMM for your organization: Would it
bring value to your organization? At what cost? Do you have
the political power to see it through?
Let me give a thought
experiment: there's a restaurant in Toronto called Susur.
The chef, Susur Lee, is a brilliant, creative artist, a world-famous
chef. The meals are very expensive, but they're exquisite and
innovative. Pretty much the menu changes every night, so the cooks have
to be well-trained, knowledgeable, and flexible. There's a much
smaller and less expensive restaurant a few doors down--not as
exciting, but still really nice and a lot cheaper. They have a
standard menu and some specials every night. Just down the street
from there, there's a McDonald's. Where are you going to
eat? What factors go into your decision? Now ask: of
the three, which restaurant uses processes that are closest to the CMM
style?
|
5.
What
is your opinion on various tester certification schemes and
examinations?
|
Do any of
the
certification
schemes or examinations involve sitting a tester in front of a piece of
software, watching while she actively investigates it, and making
assessments about her thought processes? Or do the exams ask you
to fill in the blanks or check boxes, using terms that appear in
someone's Body of Knowledge? That is, do the certifications test
your ability to think and to perform well as a tester, or do they
merely test your memory? I have a very good memory, so I can
assure you that having a good memory can make you appear to be much
more clever than you really are.
I've spoken to a bunch of certified testers who tell me things like,
"Well, I sat the test, and it was mostly multiple choice; I knew the
answers that they wanted, so I put those down even when I didn't agree
with them." That kind of conversation led me to describe
certification recently: You go to a room in a hotel, you pay a
couple hundred dollars to someone you've never met, you stay with them
for a couple of hours, you say a bunch of things you don't mean, and at
the end they tell you "Thanks--you were great." That a business
model with a long pedigree, don't you think?
As a hiring manager, I would want to certify my employees by my own
criteria--resumes, letters of reference, interviews, auditions--and not
by criteria set down by someone else. On the other hand, if
you're applying for a job, some certification or another might be
important to the hiring manager. If the manager wanted a
certification that I didn't find credible, I don't think I'd want to
work for that kind of manager anyway.
|
6. Can
you please suggest how testers should educate themselves in all matters
pertaining to testing?
|
I'd
say that the first
thing is to educate yourself in all matters that interest you, whether
they pertain to testing or not. I don't think that testing is a
body of knowledge so much as a point of view, or a way of
thinking: that is, thinking critically. I should point out
that I mean thinking as a literary critic would, trying to comprehend
something, rather than merely dissing it. A tester at some point
should learn to think scientifically, too. I have some video of
Richard Feynman summing it up: "When we want to discover a new law of
nature, we form a hypothesis or a /guess/. Then we make a
prediction and perform an experiment. If the hypothesis doesn't
fit with the results of the experiment, then the hypothesis is
/wrong/--it doesn't matter how beautiful it is, or how elegant, or who
wrote it up: if it doesn't fit with observation, then the
hypothesis is wrong." If you're tester, people will regularly
approach you with a piece of software, and a theory that it works
correctly in all circumstances. If you're a tester, a huge part
of the job is to come up with experiments that challenge that theory,
and to use observations to drive new theories.
I mentioned that I was
interested in cooking. Testers who like to
cook would do well to read Cook's Illustrated Magazine, which takes a
very scientific, testing-oriented approach to cooking. In each
article, the authors tell you what they were striving for in terms of
taste and texture and presentation, and if they say that they table
cream is better than the whipping cream in a given recipe, it's because
they tested it.
I think testing and
humour something important in common. Jonathan Miller (a British
fellow who has been a comedian and doctor and TV presenter and stage
director) once gave a lecture that I saw, in which he said that humour
allows us to change our perspective on things, "to alter our
categories", as he put it. Comedians are iconoclasts, they rely
on jarring conventional views of things, and they also listen closely
to the way people express themselves. Those are important skills
for testers. When we see or hear things in a new light, we laugh,
and I think we can learn from that new perspective. A lot of the
people I admire are funny. George Carlin, the American standup
comedian, is a great example of someone who observes things and relates
them to us in new ways. The Europeans would love him; it's a
shame he's not better known outside North America. (You can find
out more about him at http://www.georgecarlin.com; I should give a
strong caution to those who don't like strong language, though.)
Richard Feynman, one of the great physicists of the 20th century, was a
very funny guy. On the other hand, Newton was famously
humourless, so you never know.
Similarly, testing is
all about looking at things from as many
perspectives as you can. That's a perfect place for me to pass on
some advice from Jerry Weinberg, which is to think of at least three
possible explanations for something that you've observed. If you
can do that reliably, you're on your way to becoming a first-class
tester.
Talk a lot with
colleagues and peers. Swap stories, exchange
techniques. Sign up for my newsletter (send mail to
addme@developsense.com). Go to conferences and local testing
groups; start one if you can't find one. For reading--read Kaner,
Bach, and especially Jerry Weinberg. Oh, and Feynman.
|
9.
What's
your favourite testing joke?
|
Someone once posted a sign on
a laser at CalTech that said, "Please do not look directly into laser
with remaining eye." |
7.
Have
you evevr been involved with testing in the XP world? Would you like to
share
your experiences with us?
|
I haven't been involved with
much testing in the XP world--except that at Quarterdeck, our best
developers were doing many XP-like things a full decade before Kent
Beck codified XP. I loved that environment. I sometimes see
some religiosity in the XP mailing lists, which worries me. But
XP's values--collaboration, communication, courage, refactoring,
simplicity, rapid and continuous feedback--those things are
great. I seek them out, whether they're called XP or not.
|
8.
What are the
things you are currently working on?
|
Heh, heh.
Marketing. For me, marketing is much harder than testing. I
need people to introduce me to people and organizations that find
testing harder than marketing. I think there are more of them
than there are of us.
(laughs) That
means writing the newsletter, working on the Web
site, and so forth. Doing interviews for whatistesting.com.
(laughs)
I'm preparing and
offering presentations to the testing and development
trade shows. I'm working with James Bach on a couple of workshops
for Jerry Weinberg's Amplifying Your Effectiveness (AYE) conference
(which is a fantastic and unusual conferences; check it out at
http://www.ayeconference.com), and I'm trying to contribute to James'
Rapid Software Testing course. Continuing to develop really good
exercises seems to be the biggest challenge there. I'm also in
the process of qualifying to teach Elisabeth Hendrickson's Creative
Software Testing course.
I try to write at least
a little every day, mostly on testing and
quality issues. Since software is so pervasive, I think we need a
wider awareness of testing's role, so I'd like to publish articles in
places that don't usually talk about such issues.
And then there's that
aforementioned daughter.
|
10.
Any big ideas?
|
Some
things seem to be percolating
these days. One is that
we're trying to automate large
parts of
our
world,
particularly the business world, with the goal of making things
more efficient (note that efficient is usually just a code word for
"cheaper" in modern-day business parlance). We're trying to
replace
expensive people with cheap machines, or with cheaper people aided by
cheap machines. Most customers have needs that are "exceptional"
to some degree; customers are individuals, and they have individual
needs and preferences and circumstances. However, computers are
pretty stupid and have no imagination, and processes are rarely
designed to respond well to exceptional circumstances. When we
force humans to follow business processes that are being modelled and
controlled by poorly-designed, buggy software, will that really serve
the customers? Are customers going to enjoy the experience?
The testing world contains an instance of that trend. There's a
drive to push automated testing, but I'm not sure that the people
behind that push really understand testing OR automation. As I
said earlier, testing isn't an activity as much as it's a way of
thinking. A human can observe all kinds of problems that a
machine cannot. A human can change her behaviour based on
observation and judgement; a machine cannot. Machines can do
repetitive, deterministic tasks really quickly and accurately; so let's
make sure we put automation to those tasks and those tasks only.
XP and Agile Processes emphasize automation for unit testing, and that
seems to be a good thing, as long as we continue to refine and develop
the tests to reflect what the software has to do. But I think
that automation shows its limits as it gets closer to modelling the
end-user's task--for one thing, people rarely choose to automate the
kinds of mistakes that people can quite reasonably be expected to make
from time to time. So let's automate the low-level and
high-volume stuff, but let's emphasize the value of the human capacity
for thinking and inventing and observing and reporting things that
matter.
That may be tough, because I frankly don't think businesses are very
good at measuring or even considering cost and value if there's
not a direct cash figure that can be calculated easily.
Outsourcing is a great example: outsourcing is usually less expensive
if you look only at the cost of labour. However, many companies
today have no assets to speak of, other than intellectual ones--the
real assets are in the minds and experience of their staff.
Economics suggests that we can't put an accurate value on intellectual
capital; there aren't good ways to measure its worth. That's not
the same thing as intellectual capital having /no/ value, but without a
yardstick, nobody bothers measuring, and the value implicitly gets set
to zero. And yet it seems pretty clear to me that software
companies that don't place value on knowledge, or that don't treat
their people well are vulnerable losing their most valuable
assets. Companies that work hard to retain their employees, to
keep their minds engaged, to keep their skills sharp, and to keep
knowledge in-house will do correspondingly better.
That's why, although I offer testing services, I prefer to provide
training or mentorship. I can do expert testing for an
organization, but in that case, a lot of what the organization /could/
learn leaves with me. If I train people, the knowledge stays with
the organization, and what the people learn from their own subsequent
testing stays with them and with the organization. Quarterdeck in
its best days used and valued the intellects of its employees, and I
can assure you that there's nothing more stimulating than that in the
workplace. I hope I can pass on ways for other people to have
that experience.
|
|
