|
|
|
|
Scott
Barber is the person behind www.perftestplus.com, co-founder of WOPR
(the Workshop On Performance and Reliability) and a
prolific writer. In this
interview he explains what performance testing is, how one should go
about learning it,
various resources that can help you and many more things.
If
you want to ask him more questions or you want to send any feedback on
this interview, please send it to webmaster @ whatistesting.com.
|
| 1. Scott, please tell us something about
yourself. |
I was born in Woodbury, NJ USA in 1971
and
grew up in a typical American small town. As a child I was very active
in school, sports and Boy Scouts. Growing up, the goals I had for
myself in adulthood included:
- Getting to travel and “see the world”
- Get a good college education
- Get out of small-town America
- Be recognized as a success in my career
(as with most young people in the USA
my specific career choice changed many times as I grew up)
- Be a father who is active in his
children’s lives and activities.
- Design and build my own home.
I
left “small-town America”
to attend Virginia Tech on a U.S. Army ROTC Scholarship to pursue a
degree in Civil Engineering. After four years, I completed a Bachelor’s
of Science in Civil Engineering and started my
4-year active duty commitment in the U.S. Army as a Second Lieutenant
(2LT). I later went back to school at night to
earn a Master’s Degree in Information Technology. My
career has taken a rather wandering, but in
retrospect an extremely
beneficial, path to finally land in the area of Software Testing.
I spent 4 successful, but difficult years in the U.S. Army culminating
as
a Company Commander for the 101st Airborne Division, Air Assault
Headquarters and Headquarters Company approximately 8 years before I
was technically
qualified to hold the position. As a junior Captain at the end of my 4
years, I
was recruited into a government consulting company to be an Information
Engineer on a team that was designing and building a system to replace
the computer
system I most often complained about while I was on active duty. I held
this
position for about 1½ years while
becoming increasingly more technical before I went to a small private
software development company with the intent of becoming an Oracle
Database
Administrator, but actually ended up spending 6 months as a
Configuration Manager. During that time, I learned a
ton about Databases, Configuration Management, development and
networking. Most importantly, I learned
that none of those were what I wanted to do
day in and day out. 6 months later, that
company was bought by a huge, publicly traded conglomerate that
resulted in a significant reduction in pay, leading me to seek a
new job. This is when I was recruited into a Performance Test
Consulting
job by an old college friend where I spent the next
4 years. This time as a consultant is where most of my performance
testing experience comes from. After 4 years of consulting, my family
and I
decided it was time to move. As it turned out, we moved to Florida
about the same time the consulting company went through a major
re-organization leading me to
re-think what I wanted to do next in my career. After a long and
difficult 4 months of soul searching, I ended up taking a wonderful
position as a Systems Test Engineer and Software Test
Manager at my current company AuthenTec, Inc.
As far as my childhood
goals are concerned, I have had a chance to travel more than I ever
thought I would, but have plenty left to do. I achieved my educational
goals, but have a passive interest in earning a Doctorate degree
someday.
After living in a wide variety of places, I have finally settled in a
place that feel like “home” and am very content with my career
progression so far.
I have two young boys
who are the greatest joy in my life. As I write this, it is with
thoughts of my older boy starting school tomorrow morning and my
youngest learning to
talk. I guess that means that the only childhood goal I have left is to
design and build that house. Luckily, I continue adding goals, so I’m
far from bored. |
|
2. You
specialize in performance testing. What drew you to it? How did
you get started, and what led you to
start writing about it?
|
One Sunday evening I got a call
from my manager…
“Scott, be at the Marriott Hotel Conference Room at 8am tomorrow.
Our CEO has an announcement to make.”
Being a good employee, I did as I was told and went to the meeting. As
it turns out, our software development company was “merging” with a
huge media company. We were going to become their “development branch”.
As a result, we were giving up our bonuses, overtime, and our base pay
was being reduced, but we were going to get stock options.
As one might imagine, I was less than enthusiastic about this. I called
my friend of 10 years before I even left the parking lot of the hotel.
I told him the circumstances and asked him if he’d help me update my
resume, to which he responded:
“Dude, that sucks. Send me your resume I’ll take a look, but, we
need performance engineers. You’d be a perfect fit.”
I said “Performance engineer? What’s that?!?”
He replied “Don’t worry, you’ll like it.”
He was right, I’ve loved every minute of it! Since then I have
dedicated my professional career to this field. It has been a
wonderful ride so far.
After a few years, people started approaching me to write about what I
do. I never though of myself as much of a writer, so I hesitated and
stalled. (Mind you, I was the Civil Engineer in college who avoided
writing classes at ALL COSTS. When they finally told me that if I
didn’t take a writing course I wouldn’t graduate, I went to the
smallest, least reputable, community college I could find, took the
course and transferred it!) Eventually, I agreed to create the
Performance Engineering section of my company’s Best Practices. Some
folks thought it was good and started selling me to other companies to
write their procedures. Finally, my boss talked me into trying my hand
at writing articles. I really didn’t think anyone would read them, but
I wrote a few anyway. As it turns out, people did read them, and liked
them.
|
3. What fascinates
you
in performance testing?
|
Most simply, I am fascinated by the
diversity of it. As my meandering career path let me to this
field, I had a lot of “false starts” in other branches of software
development, I learned that regardless of my degree of success in those
more
narrow branches, I found myself becoming easily bored. As a
technical tester, specifically as a performance tester of multi-user
distributed systems, I get to apply aspects of all of those branches
and more in new and unique ways on every project I do. This
diversity from project to project is also what led me to embrace the
Context-Driven School of software Testing and ultimately become friends
with such software testing thought leaders as Cem Kaner and James Bach.
|
|
4. If one wants to gain
expertise in
performance testing
what all should one do? What are the tools, books and sites that can be
helpful?
|
There is very little consolidated
information out there about the type of performance testing that most
organizations need. There is no doubt that my site
(http://www.perfetsplus.com) and articles are one of the most complete
references for “Practical Performance Testing”. However, it is by
no means complete. Some of the other places I have learned
valuable lessons that I have applied to performance testing are:
- Everything I can find written by
Alberto Savoia
- Lessons Learned in Software Testing,
John Wiley & Sons, 2002 by Cem Kaner, James Bach and Brett
Pettichord.
- Testing Computer Software (2nd Ed.),
International Thomson Computer Press, 1993 by Cem Kaner, Jack Falk,
& Hung Quoc Nguyen.
- Select Chapters from The Web Testing
Handbook, STQE Publishers, 2001 by Steve Splaine and Stefan P. Jaskiel.
- Research published by performance tool
vendors.
- Research published by testing and
research organizations such as Nielsen//NetRatings, Keynote Systems,
Inc., and IEEE
- Online forums and magazines such as
StickyMinds.com, QAForums.com and stpmag.com and performancetester.com.
- Performance Engineering of Software
Systems, Addison-Wesley, 1990 by Connie U. Smith, Ph.D.
- Scaling for E-Business: technologies,
models, performance, and capacity planning, Prentice Hall, 2000 by
Daniel A. Menascé, Ph.D.
- Speed Up Your Site, New Riders, 2003 by
Andrew B. King
- Anything by Cem Kaner, James Bach,
Jerry Weinburg and/or Edward Tufte.
These
days, I continue learning through networking with experts, being a
technical editor for new performance related publications, beta testing
tools, hosting and/or attending workshops and conferences and assisting
with research being led by other experts and students in the field.
To really grow your career as a performance tester, I believe there are
several things you
can do that will gain you largest benefit.
- Become a “Mid-Level
Everything” –
Developer, DBA, Network Admin, Systems Admin, Architect, Business
Analyst, etc.
- Become an expert in the
following –
your tools, performance analysis, relevant aspects of operational
research and statistics, consolidating, interpreting, sorting, and
graphically presenting complex information in an intuitive format.
- Become skilled/knowledgeable in the
following areas – group facilitation, product management and training.
- Find a mentor. There are several
exceptional performance testers out there who believe in Jerry
Weinburg’s consulting principle “Give your best work away for
free.” Befriending one (or more) of them to ask questions of,
share ideas and brainstorm with is amazingly useful… to everyone
involved.
|
|
5. What is
your opinion about the state of performance testing tools? Are there
some open
source or free tools available?
|
Most tools on the market aren’t
actually Performance Testing tools, they are Load Generation
tools. No matter what the tool vendors claim, they do not make
performance testing simple, they do not analyze results, and they do
not pinpoint performance issues. They are virtually all grossly
overpriced, and never believe the demo. That said, I have a
favorite tool that I support heavily… with a long list of
caveats. My favorite is my favorite because it allows me to
create realistic loads by writing custom functions and procedures in C
– rather than trying to guess how to make the silly GUI buttons
duplicate the workload I have constructed.
The best OpenSource tool for general web based load generation is
OpenSTA. As long as you are only testing websites, it is
competitive with the major (pay) tools in terms of actual ability to
generate load and collect data. There are plenty of other free
niche tools for specific platforms and protocols, but they are only
useful to small groups of people doing very specific types of testing.
|
| 6. How
do
you combine performance testing with load and stress testing |
Actually,
I see Performance Testing as a
superset of testing that includes load and stress testing. I
often refer to it as “Performance Related Testing”. In general,
load testing is testing your application under expected load conditions
to determine the overall performance. Stress testing is testing
your application under unexpected or extreme conditions to determine
failure modes so you can protect against them. In my presentation
“Introduction to Performance Testing; The Who, What, Where, When and
Why.” (available here)
I describe various components of Performance Testing, and the process
in general.
|
|
7. What is
WOPR? How did it happen? Who all are involved in it and what is your
role?
|
WOPR is the
Workshop On Performance
and Reliability (http://www.performance-workshop.org).
Ross Collard and I co-founded
this workshop
about a year ago with the purpose of bringing together both Performance
Testing
Experts and “Enthusiastic Beginners” in a forum where we can share our
experiences together in a format that encourages questions, critical
thinking
and learning. So far, we have had three
very successful workshops and plan to continue having them in the
spring and
fall of every year. You can find out
more from the website. The workshop is
limited to 20 people per meeting and is by invitation only, but you can
apply
for an invitation by filling out the form on the website.
|
|
8. How is
performance testing for consumer applications different from
enterprise/server
based applications? |
There
is actually less difference
between consumer
applications and enterprise/server based applications than there can be
between
any two individual applications in the same category.
The real difference centers around
performance expectations. One of the
keys to effective performance testing is the ability to collect and
classify
requirements and then develop tests that verify those requirements.
|
|
9. Where do
you think performance testing fits in the testing life cycle? Should it
be done
early or should it be done late? What are the pros and cons and how do
you
decide?
|
There are actually at least 3
separate, but
highly related, disciplines related to performance that should ideally
be included in the Software Development Lifecycle – Software
Performance Engineering (SPE), Performance Testing (what we have been
talking about) and Capacity Planning. The only book I am aware of
that addresses these three discipline as part of an integrated process
is Improving .NET Application Performance and Scalability, Microsoft
Press, 2004. In this book, Senior Author J.D. Meier does a
fantastic job of taking the reader through the entire process.
The only drawback is that much of the book is extremely Microsoft/.NET
specific.
Performance Test planning should actually begin before the architecture
has been determined or any code has been written. It is
especially important to finalize performance testing planning before
any machines have been configured for testing and the plan should
continue to be revised as necessary until the application has reached
it’s target load in production several times. Very few
organizations do this. Most organizations create a plan shortly
before load generation is scheduled to commence and stick to the letter
of the plan regardless of the actual results or detected issues.
By working this way, much of the value that can be gained through
performance testing is lost.
The bottom line is this, Performance Testing late in the process alone,
as most organizations do, can be highly effective, but only if all
identified issues are relatively easy to resolve. Saving the
testing for the end of the development cycle can be disastrous if the
identified issues point back to poor architectural or design decisions.
|
| 10. You have
written a lot of papers on performance testing. Do you plan to write a
book on
this subject? |
I
have written quite a number of papers, presentations and tried to tie
them all
together on my website. When I started
doing performance testing, I found very little written on the topic and
had to learn
through trial and error, seeking out mentors and applying principles
from other
disciplines. I became frustrated with
the lack of valuable information available and found many others who
were
looking to improve their performance testing. As
a result I tried to share what I learned through
these mediums so
that others will have an easier time getting started than I did.
I have tentative plans for two books. The first is simply a
consolidation and expansion of my existing articles. There are
two main reasons that I would like to do this. First, I’d like to
retrofit the articles to be even more generally applicable, include
examples and code samples for various tools since they were initially
written on contract for IBM Rational. The second reason is
because many readers have requested hard copies, which is unfortunately
prohibitively expensive on a case-by-case basis.
While there are some very good books on performance tuning, and other
performance related topics and some very good chapters in various
testing books about performance testing, there is no single “good” book
for practical performance testing lessons. Writing a book is much
more difficult than writing a book’s worth of articles, but I keep
hoping that I’ll find the time, energy, resources and support to get it
done in the not too distant future. While I certainly don’t think
I know everything there is to know about performance testing, I do
think sharing what I do know would be helpful to many performance
testers both now and in the future.
|
|
11.
What
are the things you are currently working on?
|
Outside
of the books and some “clean-up” articles, I have actually moved
in a new direction with my new job. I am now in charge of testing
fingerprint sensors and their associated software (API, drivers, etc)
on both PC and embedded platforms. This is extremely challenging
as the testing is a blend of four very different industries (Hardware,
Software, Embedded/RealTime and Biometrics) that, to the best of my
knowledge, has never been written about and as far as I can tell has
rarely, if ever, been tried. The knowledge I have gained as a
Performance Tester/Analyst has been extremely helpful in the endeavor,
as has my relationships with various industry experts such as Cem
Kaner, James Bach, and Alan Jorgensen, not to mention my Embedded
Software Development Instructor friends at MIT.
Additionally, I am trying to take my User Community Modeling Language
(UCML™) to the next level. It was created specifically to model
Performance Workloads and the data associated to those workloads, but I
have found it to be personally useful in modeling other application
characteristics by simply adding a few symbols and thinking about it
from a slightly different perspective. Soon, I hope to publish
UCML™ v2.0 to see what other folks think.
|
|
