Test Jam Session

Desktop of the future

2011-09-19T12:15:00.001+01:00

ver aqui: http://limusine.livejournal.com/tag/future

Developers, find a creative hobby (as everyone)

2011-04-12T12:59:00.001+01:00

Many times, as a software tester, I find problems and end up with this comment: "Why are they/we re-inventing the wheel?"...

I think it is a fair assumption that most of the problems in a software project arise from new or changed features. Maybe not so obvious is hte fact that most of the design/implementation work could be saved, if developers searched for waht is done, and how it is done, instead of just creating things as what is the "perfect way" to them. For instance, designing a UI for an email client.

Most of the creative work from the developers comes from a natural human instinct, which is to create, or use imagination, intelligence, association, abstraction, brain capacities, to deliver something to the world. Something that hoppefully will be recognized as a great master piece, aesthetically safisfying to the developer and to the stakeholder, or to the artist and to the "connoisseur" of art.

The problem is that what we expect from software is not what we expect from art. In a very concise way, we expect software functions to be recognizable (unlike art which should be original), emotionally we want software to be innocuous, and not to awake strong emotions, like art.
So what I am saying is that developers should find some creative hobby to express themselves, so they could focus on the plain and simple ways to get bug free software!

PS: Of course, this advice is valid for eveyone, not just developers: "Find a creative hobby"

Defect quantity estimation

2010-10-27T11:33:00.000+01:00

There are some indicators of the quantity of defects a certain areea of the product might have.

Some say that testing is asking the product the important questions. Well I think that when a certain area raises too many questions about the expected product behaviour, it will probably have proportionally many defects. I guess this is based on these principles:

what's hard for me (user) to understand, is also hard for the developer to implement.
the more complex an area is, the more dificult it is to be self-explanatory, the more questions it will raise for the user, and themore defects it might have (because it is complex)
bug quantity grows exponentioally with product complexity, or, which is the same,
bug quantity grows exponentially with product size, if we count for product size areas of the product are connected

Other things that make me suspect of defects abundance:

when same operation/feature/information is presented on different places in different ways. More generally, usually Redundance leads either to incorrectness or mis-information, because when a user sees apparently same information on different places, he might think it is not the same.

testar é...

2010-09-17T12:22:00.000+01:00

Reina um grande desconhecimento e equívoco sobre o que é testar. Vulgarmente confundido com "verificar", é considerado uma tarefa monótona, que não exige criatividade ou raciocínio e tomadas de decisões na hora. A definição que actualmente adopto é a seguinte: "testing is this: helping our clients to ask and answer their most important questions. (M. Bolton)". O cliente pode ser quem faz o código, quem diz as specs, quem "manda" na empresa, ou o cliente própriamente dito. Testar é essencialmente aprender, aprender o que faz o produto, o que devia fazer, o que querem que ele faça, o que pode eventualmente fazer inadvertidamente. Testar é experimentação científica, colecção de estatísticas, investigação criminal (encontrar um bug, persegui-lo, determinar as provas necessárias ou circunstanciais, etc) e não é de todo monótono. A parte da verificação, ou confirmação propriamente dita é apenas uma pequena percentagem da nossa actividade, e tendencialmente candidata a automatização. Testar é também comunicar, advogar, intermediar.

Isto vem ao encontro do meu segundo comentário: A engenharia de software, dum modo geral, é uma ciência social. Ao contrário do que se pensou durante décadas, não é uma ciência exacta, nem sequer é comparável a outras engenharias como a mecânica ou construção civil. É um campo em que o factor humano está presente ao longo de todo o processo: nos requisitos e na constante adaptação ás transformações, no compromisso entre funcionalidade idealizada, funcionalidade real e funcionalidade execuível. Se analisarmos a ocupação do nosso tempo, chegamos à conclusão que apenas uma parte é utilizada na "produção" de software, propriamente dita. A negociação, a aprendizagem, a experimentação, o debate, a passagem de conhecimentos, etc, ditam que este domínio em que nós trabalhamos seja, como todas as ciencias sociais, imprevísivel. Não existem projectos sem imprevistos de toda a espécie, e só o mais ingénuo dos estudantes pode acreditar que os requisitos não vão mudar, que o código vai funcionar à primeira, que as tecnologias não vão mudar...

Não é "defeito" das empresas, é "feitio" da área onde se movem!

Tester job is to make things work, not the opposite!

2009-10-09T11:33:00.000+01:00

Tester job is to make things work, not the opposite!

In my last post I forgot to mention the main idea I'd like to retain, which is, the tester Mission is to make the software work.

What we want, in the End, is to have working and satisfying software, with few relevant bugs. Let's not forget it. The more I get the software to work, the more complex scenarios I will build, and eventually the more interesting and relevant defects I will find.

If a tester gets stuck with basic defects, often superficial, often interface, but sometimes really blocking ones, he won't be able to advance in its mission, wich is to have working software. That's why I advised in my last post to start with simple and common, expected scenarios.

Corner cases are often interesting and exciting from an academic point of view, but in the long term will not yield so efficiently to working software. Although usually testing is seen as a way to find defects, I found out that on complex systems, our mindset should be focused on software success, not on software failures.

What should first tests address?

2009-10-08T10:45:00.000+01:00

When planning a testing effort, or if you wish, when designing a Test Plan, some considerations may turn out very useful.

First tests on a given area are decisive. If we consider testing is Learning, or asking valuable questions to the product, then first questions are the most important ones, because we are getting a first contact with the product (first impressions tend to stick in our mind more easily).

When addressing these questions, we should expect answers like: "Yes, this feature is working exactly as I expected", or "Not at all, there is no way this could work". Answers like "in this particular situations, this might not work", or "I think this is working, but I am not sure if this is very useful" are discardable at an early stage, because we are trying to gather basic information, not exploring detailed scenarios.

We should seek the most common scenario, the main functionality, and try to setup ideal conditions so the software succeeds. At this point, we may say if the feature is working, or on the other hand, if it is not working at all.

Even after pass the very first test, after we say the feature is working, we will increase complexity of testing, but always trying to go for the next simpler and most common scenario that someone might want. This way we achieve:

-Continuous and steady knowledge growth, easier and logical learning. Our mental model of the product is built from the basis to the upper
structure, not the other way.

-Efficient Testing effort, every results are probably matched to our mental model, which is growing continuously, thus minimizing gaps in the complexity levels of our knowledge. Suppose we go for a complex scenario at a very early stage, and we find it is not working. My experience tells me that the test is often discarded, or at least it will have to be repeated later, after do some more simple tests. Often what we see as a product failure, is a (tester) knowledge failue, because we are exploring conditions and scenarios for which it was not built.

Because our knowledge in this early stage is extremely dynamic, Exploratory Testing (ET) may be very efficient, due to itts adaptability and dynamism (see literature on ET)

Status vs State

2009-10-06T15:48:00.001+01:00

Ok, I'll tell you my two cents: a state is any condition which is durable or lasting (Zustand, Istzustand) whereas a status is a classification of state among several well-defined possibilities. An immigrant's status is either legal or illegal, and if he is the latter, he may be in an anxious emotional state.

The best in testing is not the Tester: the domain problem

2009-07-13T09:34:00.000+01:00

While reading some critics about a book about Lester Young, on the ever-best-blog-of-the-world , thebadplus , I got to the paradox that often the best work is done by people from other domains, and that work is often not recognized by the specialists due to lack of versatility, and interdisciplinarity.

There is this book about a musician, written by an historian specialized in Black Studies. Most people who will read the book are musicians (like me) and are not sensible, or informed , or even interested into historic details, or to litterary value. However that book is very well written, because it was written by a writer, (not a musician), and has insightful information, because it was written by an historian, (not a musician).

Nevertheless, most critics are negative, because they are done by the majority of musician-readers, and a minority of social studies people.

Sometimes in testing, and of course, in life, we are victims of our domain knowledge, and our lack of cross domain knowledge. I have seen psycologists or lawyers, do great work on the field of testing.

The cross domain exercize can be really refreshing, and is surely one of the brain storming unblocking techniques.

tester skills: domain (business) knowledge vs. "technical" knowledge

2009-06-22T11:35:00.000+01:00

here is my response to a post, followed by a (much better) argumentation from Cem aner:

>Having the ability to understand the implementation gives you the confidence and knowledge to make sounds decisions on what to test and where to focus your time and energy. It allows you to challenge the system design and test pieces of the system as they are developed.

On the other hand, you will be thinking in the same biased way as the developer, only (hopefully) with a deeper caution and inspection level. If you don’t have any idea about the implementation, you might stumble across totally different issues. Certainly everyone has the experience of looking at a product from a new point of view, for instance when the product is new, or when the tester is new, and how productive that might be (in terms of number of reported defects).

>However we agree that technology understanding is more expensive or more difficult to master. We don't believe that domain knowledge is not important but that it is easier to acquire.

With all respect, I don’t see how you agree on that. When a tester is hired, the academical skills are required, but in my experience, there is no specific need for a programming language, or a specific technology. The academical skills indicate that the tester is able to learn, and has already learned, technical disciplines, and has the ability to think as an engineer, solve problems, etc, etc. Usually, it is not a guarantee that he will master any particular technology. The professional will learn to master a new technology as the work demands it.
But to acquire business expertise, it usually takes time and money. While the technical knowledge is assumed when hiring, you will have to pay a salary to the tester so he learns the domain particularities. So it is not cheap!

In my field (contact center software) there are visible differences between someone that has 1 year of work, or 2, or 5, or 10! We lately hire people from other departments within the company to do the testing, because they start producing results much faster.
You might think that is because my case is a complex one (contact centers, client databases, telephony switches, etc), but even considering a simple form to input a client name might have lots of problems. Let me give an example: recently I bought an airplane ticket on a well known company using the internet. Can you imagine a simpler issue than requesting the name of the traveler? Well, there were problems on the check-in because it was not clear that only first and last name were requested, so the ticket only caught middle names!

Having said this to defend the “business” side, I should say I always try to model (pseudo-code) the implementation, and try to know the technology that is being used, because it actually helps me to find out “what’s happening” when something unexpected happens (and that’s all the time).

Me

-------------

> First our argument about domain was insurance. I argue that it does not take several
> years to know what you need about insurance to be an effective developer tester or
> system architect. This is why people in banking and insurance can move easily.

Funny thing about insurance.

I too have seen incompetent programmers, architects, project managers – and testers – at insurance companies, and banks and government agencies and universities and telephone companies (and even at game companies). I too have worked with people who have no relevant education and who serve as their corporation’s advocates of excess paperwork, invalid metrics, and the politics of blame. (And who take exams so they can put on their resume that they are Internationally Certified as Professionally Incompetent .)

However, I’ve also seen really good people in these kinds of organizations. After all, SOMEONE has to get the stuff done and working.

I don’t know why incompetents survive in these organizations, but when I think about knowledge, skills and attitudes needed to do a job well, my first point of reference is that I need people who can do the job well, not that they should not be aggressively incompetent.

Insurance companies create or acquire a lot of different types of software.

For example, testing software that does risk estimation (a core issue for insurers) takes deep subject matter knowledge. You CAN do the superficial testing (OK, it doesn’t take letters in number fields), but that doesn’t tell you whether the program is processing the probabilities appropriately – and that’s what the company is betting its future on.

Here’s another example: the software that salespeople use has to get them the information they need, quickly (quickly by their standards, not programmers’ standards), accurate in every potentially relevant respect (where relevance and accuracy are evaluated through the eyes of the salesperson and corporate counsel, not by the “good enough” filter of the programmer who has never done this job), in a format they can understand (not a format that we tell them they should be able to learn if they’re smart enough and work hard enough), displayed in a way that is not going to cause difficulties if someone looks over their shoulder at their screen, and that facilitates closing the deal. Having sat on the other side of sales software (as a user, rather than developer), my impression is that very few programmers understand what salespeople actually need to do their job well and even fewer have enough respect for the work salespeople do to CARE what the salespeople need. Sales is difficult work. Tools that support salespeople well are difficult to design and evaluate. There is such a thing as genuine domain expertise in this field, it is rare, and it takes time and work (including a lot of visits to the field) to develop.

Let’s pretend that insurance companies operate with 10 different types of software (actuarial being one example, sales support a second, etc.) I think that means that a central test organization has to develop subject matter expertise in each of these areas.

Unit tests, and most other types of glass-box tests, are verification-oriented. The question is whether the program does what the programmer intended. It’s an interesting question whether verification should be done by the programmers (whether this is a task for people designated “tester” at all), but whoever does it, that doesn’t take us to the other classes of issues that I think testers SHOULD be spending time on. Not whether the programmers implemented a model correctly but whether it was the right model. Not whether the program meets specified performance criteria but whether performance is fast enough to meet the needs of the client. Not whether the program presents the intended menus and graphics but whether they are natural and useful to the people for whom the software was written. Not whether the software works on the specified platforms but whether it works on the platforms in use. Whether it interoperates effectively with other software, etc.

Verification is the handmaiden of validation and evaluation. It is easier to evaluate a program that operates in a consistent and predictable way. Someone has to do verification, it is important work, and the test group can help with this, but if the program doesn’t solve the problems that its users need solved, if it doesn’t provide the benefits that they need provided, then it doesn’t matter how well the program doesn’t do it. Programming-focused testers can help a lot with verification, but testers need a lot of other knowledge to do the rest.

Cem Kaner, J.D., Ph.D.

Mcluhan for testers (reminder)

2009-04-02T16:20:00.000+01:00

The 4 questions to be raised in front of a new tool/technique/heuristic/etc:
(replace /medium/ with /tool/ or whatever)

1- What capabilities does the medium extend? (E)
2- What medium or media do the new medium obsolesce? (O)
3- How might the message of the medium eventually become reversed from its original message? (R)
4- How did things change when the retrieved medium appeared as a new figure in its corresponding ground? (R)

Mnemonic EORR

Michael Bolton pointed out this great definition:

a medium is something that causes change to happen.

Creativity in Testing

2009-03-27T16:15:00.000Z

I would go even more general, and say that engineering is a creative activity. I see engineering as the craft of solving problems, which require deep domain knowledge, along with technical skills, and an open, creative and critical mind.

For testers, I think creativity comes up in things like:
-Test design aimed at exposing specific product features
-Finding workarounds for known problems
-Finding probable flaws or defects, based on our knowledge of the product history, and on the human nature.
-Finding potential ways to break a fix
-Finding faster/smarter ways to accomplish my objectives
-Find out what would be the best/ideal tool for specific task
-Finding tools
-Creating tools
-Imagine how the tool I am using might threaten the value of my work, for instance giving me confidence on certain area by false or biased results
-Generically, imagine how my concepts might be deceived. For instance: am I pretending to advocate the customer, when in fact she doesn't value my values?
-At last, but most important, finding or previewing defects just by conducting our though to the conclusion. Imagine a chess game: if a developer thinks 3 moves in advance, and the tester thinks 4, then it is possible for the tester to anticipate defects that the developer didn't. I often think of this as /Imagination/, rather than creativity, in the sense that we let our mind go to all the way to the end

---, Lisa Crispin wrote:
>
> IMO, software development in general is a creative activity, testing
> included. Testers add the most value when they strive to hone their craft,
> collaborating with team members to find good ways to ensure delivering high
> quality software and delighting customers. Exploratory testing requires
> creativity as well as critical thinking. Testers need lots of creativity,
> imagination and empathy to come up with good test cases and
> scenarios.Testers on my team come up with creative solutions to
> coding/development problems, not only testing problems (and the reverse is
> true too).
> -- Lisa
>
> On Fri, Mar 27, 2009 at 9:17 AM, Strg Pune wrote:
>
> > Hi All,
> >
> > Recently I watched a video on YouTube and fallen in love with the thought
> > expressed by Sir Robinson.
> > Sir Ken Robinson: Do schools kill creativity?
> > *
> > *
> > The question is how can we as testers' community adopt the notion of
> > creativity?

-----------

Michael Bolton added a plus 1 :) and:

>If one believes in the theory of multiple intelligence of Howard Gardner,
what is that the tester has to possess to be a creative tester?

Among other things...

- independence of mind
- diversity of experience, approaches, models, tools, education...
- freedom and encouragement to innovate, explore, and discover
- freedom from arbitrary constraints
- responsibility to explain and justify his/her work

---Michael B.

Another add on:
The creative process is said to be of five stages.
1. Preparation
2. Incubation
3. Illumination
4. Elaboration
5. Verification

Failure Improvement vs crime investigation

2008-07-24T10:45:00.000+01:00

Brian Marick said:
I'd probably say the most important skill is
"failure improvement":

<http://www.testingcraft.com/failure-improvement.html>

It, at least, is the skill I see testers paying attention to that I
almost never see anyone else paying attention to.

---
As I was reading the link, I thought this part of a tester work is what I usuallly compare to detective work (specially when someone says testing is boring):
-As the crime (failure) is found, I will try to gather all the information from the crime scene (context)
- I will try to rebuild the story (remember what I did exactly), using all the evidences I can gather (see traces and logs)
- I will try to preview where the criminal will attack again (or what are the precise conditions for the failure to occur)
- Then I will try to catch him on the crime spot (replicate the failure)

In fact, all six items that Brian refers on his article map quite well on this crime metaphor:
(The purpose of failure improvement, or defect isolation)

1-Split defects - maybe there is more than one criminal; possibly even more than one crime
2-Simplify defect - When there is a simple explanation for a crime, it has great chances of being the true. When it is complex, it might as well be the true, if there is no simpler one (as Sherlock Holmes said :)
3-Replicate - the criminal always comes back to the crime scene.
4-Find the "uggliest side" of a defect - What is the most damage this criminal can do if he is not caught? I believe the authorities often do this question in order to evaluate the effort to put on the investigation (or the risk of "not fixing the defect")
5-Describe accuratelly the failure boundaries - or defining precisely the pattern of the crimes
6-Find new failures - find new crimes - preferably before they happen!!

Software Artist

2008-06-05T10:06:00.000+01:00

Indeed [Chris said that the Sowftware Artist is *not* a metaphor] . Metaphor (in this context) is a conceptual tool to trigger creativity through unexpected relations between apparently different subjects. It is used both in software world and in traditionally pure arts world.
---
[Shrini asked about performance vs. productivity]
Both terms go with both. I will try to give counter examples:

-On musical creation, we talk a lot about productivity. For example, you have modern jazz artist Anthony Braxton who releases dozens of albums a year. In classical music you have famous people who were extremely *productive*, despite living only 30 years, and other that compose very few pieces, but got famous anyway.
I remember earing this kind of conversations back when I studied in the conservatory of music.

The value of productivity is arguable, some may do more with less. This is true for software world and for arts. Of course when we talk purely about money, we always want "more" (and that is also true for both worlds.)

-When designing a piece of software, sometimes there are problems, like a feature we don't know exactly how to implement. On my company, there is a guy which is known for always finding great solutions. He is known for his *creativity* in working around hard problems.

Maybe there is a problem with the words. In a previous post Chris suggests using "create" instead of "write" or "code", or "build". This seems a great idea.

.....
Sorry I used "creativity" on my previous post instead of "performance", but the reasoning is still simple.
Word performance is used in engineering fields very often: note how it was used in marketing language for describing new car models ("modelxxx has a great performance off-road") so that people get a fast intuitive idea of an overall caracteristic of that vehicle.

We could equally say "John has a great performance as an exploratory tester", or "the new kernel has a great performace on that scenario"

-----------------

Michael Bolton describes metaphor even better than I do:
Actually, "is like" denotes simile, not metaphor. What you'resaying--something like "software IS art" is metaphor, quite distinct from"software IS LIKE art".

>I thoroughly do not intend for The Software Artists to be interpreted as anunsupported metaphor.

That's too bad, because metaphors rule. In addition, metaphors don'trequire support or references or citations. Metaphors are models that (aslong as we're paying attention and participating in the exercise) highlightboth the similarities and the differences. Metaphors get their powerprincipally from the cognitive friction that gets generated from equating(not merely comparing) something with something else that is manifestlydifferent AND manifestly the same.
---Michael B.

Software artists explained

2008-06-04T11:00:00.000+01:00

Shrini, thanks for your always critic thoughts.You put some phrasing that is not my own, if you read carefully my post you see I say:

"engineering is also a form of art, in what relates to creativity, beauty and emotion drive"
and
"Every piece of software we create [...], has a beauty that will echo [...]"
which is not the same as saying:
"all engineernig is art" (typo kept)

but I get your idea, so I will try to explain a little better.
I think there is a continuum between the two extremes: the complete mechanical process (call it automated, or automatic) and the human or artistic process.

Take some examples:
On a painting, for example, there is a large amount of work that is technical, like the materials, inks, brushes and the painting technique itself. There is also a lot that is conceptual, emotional, artistic.
On a musical instrument same applies: technique, sound engineering, scales, arpegios, etc are ideally automated "under the fingers", so that our mind can search more musical ideas.
So these activities, although traditionally from the "art" field, often consist of 90% of mechanics and 10% of art, if you know what I mean.

Now for the field of engineering:
-Suppose my job is to test a piece of software.I will get to know the business context, so I can explore some useful scenarios - just a painter will focus on a theme and explore it;I will place myself on a end-user position, try to see what means value to this product - just as an artist who is constantly evaluating his workSometimes I will complain because the GUI does not communicate the idea I think it should, thus misleading the user - sometimes music fails to communicate, doesn't "click", as well.

So, just to resume my point, I don't think "everything is art", but instead that we can learn a lot if we see the processes that are traditionally not artistic, through an "art lens". Particularly through the use of metaphors, that are everywhere in art, but also in software (for example, a button is a metaphore for an action. Sometimes the button is there only so we can notice we can do something)

This said, I would like to add that we cannot separate the artistic individual from the practical individual inside each one of us, although we use in different moments one or the other (some say it is located in different zones of the brain...)

Software Artists

2008-06-03T12:08:00.000+01:00

I think this theme is definitely important, and I believe in the present and near future people are realizing this reality. So I congratulate you for this article.

Now for the critic...When you split between engineering and art, I tend to disagree. I think engineering is also a form of art, in what relates to creativity, beauty and emotion drive. For example, the beauty of a construction: one may argue that the architect is an artist, but the engineering itself can be beautiful, for instance, on the simplicity of the structures, the way the light enters in the inside (this is really an engineering problem). The effort of the engineer was drived by his feelings, his ideas, and the emotions he placed on the future of the users for the house.

More generally, I think there is no world where we can drop the art, as we cannot drop the feelings. Every piece of software we create, every test we design, every chair we build, has a beauty that will echo when someone else will see our mind through that work (of art)

--- "Chris McMahon" wrote:
>
> A couple of times on this list I've mentioned starting what I call the
> "Artistic School" of software development and testing. I think it's
> critically important to have intellectually rigorous descriptions of
> software practice that are *not* based in the language of
> manufacturing or engineering.
>
> To that end, I have published The Software Artists:
> http://chrismcmahonsblog.blogspot.com/2008/05/software-artists-index-of-links-to-all.html.> This is an attempt to describe software practice in the language of
> art and performance instead of manufacturing and engineering.
>
> I"m interested in comments and criticism on the paper. I vacillate
> between thinking it's either the coolest thing I've ever written, or
> completely and totally irrelevant. It might also be just a mildly
> interesting failure.
>
> Anyway, if you read it, let me know what you think.
> -Chris

Funny Quotes

2008-05-16T12:08:00.000+01:00

Bill Hetzel, 1988, The Complete Guide to Software Testing
"The only exhaustive testing there is is so much testing that the tester is exhausted! "

The variation of this that is my wording is "Programs aren't released. They escape!" The variation of this that is uniquely James' is the notion that there are some testers who won't release a product--"you have to pry it from their cold, dead hands." -- cem kaner

Delightful engineering

2008-03-18T11:17:00.001Z

I usually relate delight operating a software with usability or functionality.

A Delightful software is more easily sold. A Demo or a talk featuring nice usability features is much more successful, like:
-the software "guessing" what the user would do next and provide shortcuts; remove redundancy often cuts several useless steps in wizards, for example; according to context, do not provide useless options:
-a software that is self-explained, and makes the user feel a genius, instead of making him feel stupid;
-Functionalities presented as bonus; the user is surprised by capabilities that were not intended at first, but tooltips are telling him they exist and are easily accessible.

The problem here is that delightful features often become standards. If they are good and useful features, people will expect for them, and maybe submit defects when the "delightful features" are not present. Then the term "delight" often evolves to "good design".

Just trying to resume, I think good "engineering" has the implicit capability of creating delight, even if it is not it's purpose. The world is full of this, just like good furniture, cars, engines, planes, architecture, music, art...

The sooner you discover a problem, the easier it is to fix it?

2008-02-29T16:01:00.000Z

The last sentence (“it is better to think of the "cost" of the fix more that the ease to fix it.”) made me think of different concepts that are often confused leading to over simplified heuristics. Heuristics tend to become more far from truth as they become more popular and generic.Here are some concepts I could filter:

1-Cost of a defect: this is the sum of several other costs (2+3+4+5+6+7)

2-Cost of finding a defect: seems that usually the later, the easiest to find a defect. Finding it on specs may be hard, on working software may be easier but requires testing, on a client the cost is zero.

3-Cost of debug/fine tuning the defect: for instance, after a tester finds a defect, he will try to reproduce, see logs, etc to find out more about it before report it. I think when found on the design, this cost is zero; when found on integration testing the cost is low; when found by a beta tester or a client this cost may be higher (developers will have to investigate more)

4-Cost of reporting

5-Cost of the /harm/ of a defect: If found in the specs, it does no harm; if found during integration or system testing it may delay the tests are cause come entropy/confusion; if found by a client it may cause more harm to software factory image, or to the client’s service.

6-Cost of fixing the defect: the minimum cost is at the design phase; depending on the language it may cost more or less to fix it in the code; depending on the complexity of the system the cost may be quite high due to integration issues, for instance.

7-Applying/propagating a fix: may be zero for an in-house tool; may be high for a product sold around the world; But: note that even in this case the cost of propagating a specific fix may tend to zero, because there are already scheduled Service Packs (SP), introducing a fix in a SP may be automatic, SP distribution may be automatic as well, so the cost of propagating a single fix is almost zero!

8-Cost of verifying the fix: seems not depending on the time it is done
As sort of conclusion:-I would not use the word “easier” but the word “cost” as Paul Holland suggests-I would not stick to the over simplified heuristic “the sooner the better” without paying attention to the context, as James Bach said.

Developer quotes

2008-02-13T09:21:00.000Z

Question: How many developers it takes to fix a light bulb?
Answer: what's the problem? It lights in my desk!

schools of testing

2008-02-12T09:29:00.001Z

[Cem Kaner]I wrote a blog post on this at http://www.satisfice.com/kaner/?p=15 that you might find helpful

The notion of a “school” has little to do with what techniques people use. Any school can use any technique (unless, for some reason, that group has decided that, in principle, the technique is flawed or unethical).

The notion of a school has more to do with your foundational thinking of why you do what you do. And from that, it drives your strategy, it drives the way you educate yourself further, and it drives your notions of success and failure.

Take a look at the exchanges I’ve had with Don recently. The differences there are not about techniques. They reflect fundamentally different views about the role of testing in a product development effort, the ethics of test management, the definition of success, the perceived professionalism and motivation of the programming team, the definition of quality, and so on. It is easy for us to talk past each other because our basic assumptions are so different that it is hard for us to notice them.

For example,
· I think that testing is primarily about learning and I am deeply interested in ways to make testers better learners.
· Other people think testing is primarily about techniques, more a mechanical activity than a highly cognitive inquiry.
· The implications of this difference are fundamental. For example, think about the recent thread involving automation of “mainstream tests.” One of the first tasks I give a tester new to a project is to use it in ways that help the tester understand what the product would do if it worked really well. James calls this “sympathetic testing.” The goal is not to find bugs, not to regression test, not to smoke test, but to gain an appreciation for the product that helps the tester THEN hunt more effectively for problems. You would only reserve time for this type of work if you thought that it is important for the tester, this new human on the project, to learn this kind of information and that this time-consuming task is a good way to foster that learning. If you were less worried about developing the knowledge and judgment of the tester, you’d probably just automate these tests. From my point of view, that sabotages the effort because the human doing the testing will probably think about the product with less depth and thus less effectively prioritize later work and less effectively interpret and communicate later test results. But from the point of view of someone who thinks testing is about using well-understood techniques to expose classically-defined errors or deviations from a well-written specification, the automation saves time and frees the tester to apply more techniques to more code. Who’s right?
It is hard to get people to articulate this difference, it is often hard for people to articulate it to themselves. They said instead, “testing is testing, and I think testing is about what testers do” and then they talk about the broad mix of activities they do, rather than about the underlying intuition they have about how to prioritize them.

I don’t think that people consciously join schools until they are senior enough to start asking, “why am I doing this?” and “how do I make decisions when decisions are difficult?” In those situations, professionals come up with guiding principles or guiding heuristics or guiding attitudes that influence the rest of their work.

My goal, in abstracting out the “schools” was to see what the common threads were that characterized groups of people who seemed to have reached the same principles. My primary goal, initially, was descriptive.

When I was a student in psychology, the contrasts between “schools” in that field were useful for exam study—we could organize clusters of thought in ways that the professor would grade well. And they were useful for putting some of the research we studied into context. But undergraduates who puffed their chests out and said, “I am a behaviorist!” or “You behaviorists are all stupid! I am a Gestaltist!” were (usually) playing with the ideas. Trying them on for size, experiencing them, but not really committed to them. As a doctoral student I ran what was expected to be a “safe” set of experiments (boring but original in the sense that no one had ever done this particular boring variation on a class of boring surgery-plus-maze experimental situations) (it was like applying a same-old-boring test technique to a part of the program that hadn’t been tested this way before, expecting to find the same types of bugs that this technique usually exposes on new code. Zzzz Zzzz Zzzz) But instead of giving us the expected results, our rats behaved in entirely unexpected ways. At this point, I was nearly fired from the lab for incompetence. Instead of backing off, friends and I set up video cameras, scripted the experimental method more carefully, and showed that the rats really did behave this way. Then I simplified the experimental situation and showed that they STILL behaved that way. That convinced the head of the lab that I was on to something. UP TO THIS POINT, everything I was doing was about technique. To the extent that theory was involved, it was the professor’s, or the senior doctoral students who were coaching my work. But now I had a set of decisions to make and those decisions would determine how I would spend the next almost-a-year or so of my research time in this lab. Time is limited, resources are limited, lots of ideas are possible, lots of experiments can be run, but if you don’t think carefully about why you want to go further and what types of things you want to learn and why, then in research, you are likely to drift in a sea of not-very-meaningful techniques. A lot of research papers come out that way, from immature researchers who spend their time doing the experiments and not much time on why or what they’ll learn or why they really care, but at some point, choices and constraints hit at the same time. I asked, “How would Edward Tolman have analyzed this situation and how would he have prioritized the next work?” (Tolman, see http://psychclassics.yorku.ca/Tolman/formula.htm, for an example of his foundational thinking) and from there, I was operating within the mental framework of one of the dominant schools of psychology, as it applied to animal learning theory. Had I adopted that school as my own? Not exactly. Was it like my religion? No. But I used it to structure my approach to the problems, to help me choose what to do next and how (what questions, what techniques), and to help me interpret my results. If I had adopted a different worldview, I would have done entirely different experiments from the same starting point. If I had simply followed a pragmatic course, I would have done what several colleagues (other grad students, a post-doctoral researcher, and one professor) advised me to do—put this line of work on the shelf and go do something else that was safer and more likely to yield easy publications in a reasonable period of time. I adopted a theoretically-driven approach and had the good luck to get some really interesting data. (You can see details of that experiment in my video lecture on scripted testing at www.testingeducation.org/BBST).

People can drift through their work for a long time without facing really tough choices. They don’t have to adopt a theoretical framework, or develop one for themselves, because the next decision and the next one and the next one seem pretty obvious. Not easy. But obvious. In practice, these folk pick up a smattering of ideas from the different competing viewpoints in our field. Some of the things that people like Don say sound interesting. Some of the things that James or I say sound interesting. We all (well, most of us) describe our techniques and some of those seem useful, and so, in practice, most testers I have interviewed and most test groups I have worked with have operated in a way that they considered largely atheoretical, pragmatic, and (to varying degrees) eclectic (willing to adopt a variety of different techniques and approaches to developing test ideas).

At some point, as a manager, you might hit a set of situations that are difficult enough, and feel important enough, that you start thinking at a more strategic level. Why are we in this situation? Whose opinion counts about what services we are supposed to provide, what information we are supposed to report, what information we are supposed to NOT report? How hard should we fight, over what issues? What should we do next and how can we explain it? How much more time and money do we REALLY need to do an adequate job, and what would “adequate” really mean in our situation, and how can we explain it?

Imagine you finally come up with some answers. If you’re running a group, say 6 testers (or 60), some of those people will be doing things that don’t seem like the most important things to do under the strategy you have finally thought through. Some of them can’t explain their decisions in ways that make sense to you. Some of them seem determined to do stuff that you consider wrong-headed, unlikely to yield any value at all.

At that point, whether you realize it or not, you have probably adopted a viewpoint, a way of thinking about your project, your tasks, your staff, your priorities and your values, that maps to one of the schools. Maybe not. But it’s worth considering. What we tried to do was to bring them out into the open. So that in some critical situation, you could say, “I’m not sure what to do next, but in principle, most of my decision making is more consistent with this way of thinking than that way of thinking. So in my copious spare time (3 hours) that I can spare to do some reading about how other people approach problems like mine, I think I should look first at this person’s writing, at this group’s ideas.” At that point, whether you have fully bought into it, you are operating within the context of a school.

Now, let’s consider your questions:

1.What happens when people of different schools work together?
2.Can we cross-fertilize between schools?
3.Do I have to pick a specific school?

1.What happens when people of different schools work together?

It really depends on the people involved. Doug Hoffman and I started a long way apart from each other. We decided to teach courses together and consult together, partially to gain insight into each other’s views. We learned a lot from each other. In some of our courses, our students had a fun time because one of us would lecture for a while and then the other one would butt in and contradict many of the ideas presented. They got a richer course, and in long debrief meetings after class, Doug and I worked through a lot of stuff. In this particular case, the experience (which continued over several years) was transformational. Both of us ended up in a different place. I abandoned some of my old ideas; so did he.

In small test groups, it is harder to manage testers who operate from fundamentally different perspectives. This often turns into what looks like a personality conflict, as one person described what the other person does as stupid and wasteful. (Some people say this more politely, but it can be very hard.) The problem is not just that they are doing different tasks or using different techniques or even pursuing different priorities. The problem is that their ethical analyses are different, their ideas about who is to blame, what it means to improve, who has to be retrained and why, who should be told what about the project, what autonomy individual testers should have, what control the project manager should have, what documentation is sufficient, what constitutes professionally competent work and competent recording of the work—when people have fundamentally different visions of the work they are doing:
· Sometimes (probably the most common case), one group quits
· Sometimes (not an uncommon case), the manager adopts one view or supports one faction and makes the other folks conform (and eventually, people quit)
· Sometimes people move into competing workgroups and we have bitter corporate politics
· Sometimes a particularly charismatic manager identifies the foundational differences, brings everyone together, lays them on the table, gets people to articulate them as their own, and then persuades the groups to tolerate each other:
We don’t know which is the right way, we have an impossibly complex problem/project in front of us, let’s agree to approach it from different directions and see what happens. In my limited experience, success here doesn’t just come from charisma (whatever that is), it comes from humility and the ability to convey that humility. This really IS a problem bigger than what we know how to solve, we really SHOULD accept the idea of our own fallibility, and agree to do the best we can while suspending judgment on an approach that looks less promising to us but, with our imperfectness of knowledge, might be more useful than we realize.
I have seen this, I think, but not often.

I am sure there are other cases, but they are outside of my personal experience.

2. Can we cross-fertilize between schools?

Yes. I’ve learned a lot from people whose approaches are fundamentally different from mine. In a few cases, other people have learned a lot from me, despite their fundamentally different view.

More often, I think people learn the superficial things and miss the underlying ones—learning very little. Transfer of techniques is pretty trivial. Transfer of ways of thinking, and transformation of them, is harder and much slower.

I think most of the cross-fertilization that I have seen has come from intentional, explicit effort. Often it happens at the individual level – two friends, or people who become friends, decide to take the time to work together closely enough to really appreciate the other one’s art.

3. Do I have to pick a specific school?

To the extent that you adopt any foundational thinking about what you do, you’re in the territory of people who do foundational thinking.

You can read about lots of different viewpoints—Bret’s goal in publishing his talk was to articulate those differences in a way that would be useful for educators, to help their students appreciate the diversity of the field. (Bret’s original presentation was to the Workshop on Teaching Software Testing.) Coming to understand their similarities and differences is useful.

At some point, if you are to become a mature professional in the field, you have to adopt your own foundational views. I’ve given enough examples of foundational questions above that I won’t repeat them here. At some point, you have to come up with a set of answers, and approach reflected within that set of answers, that makes sense to you and that has some coherency.

Probably, at that point, whether you have intentionally adopted one of the schools or not, probably you have adopted the point of view of one of the schools.

The goal of a descriptive taxonomy (Bret’s, mine, others to come) is to capture the essence of most of the leading, rational, competing viewpoints. If it’s a good taxonomy, and you end up doing a good job developing your own viewpoint, your view should fit somewhere on the taxonomy.

As I mentioned in the blog post, my first shot at a taxonomy was technique-focused and it succeeded in the sense of leading to some interesting ideas, but it failed as a way of describing schools of thought. I like the new one better, and will continue to like it until we find a better replacement.

Not everyone belongs to a school:
· Newcomers don’t have enough experience or insight. Some people never develop enough experience or insight to develop or adopt a foundational view of their work.
· If you mean, do you have to take sides like voting in an election or choosing to murder or be murdered in the latest religious crusade, no. Absolutely not. I (me, personally, my choice) happen to think that the way of thinking that I have adopted is a really good one, and I talk favorably about it a lot. I also happen to think that some other viewpoints are wrongheaded, and I explain why. But that’s my choice, and it reflects my personality.
· Many people in our field read so little, or read so superficially, that they never understand what any of the schools is really trying to say. If you spend most of your reading time on mailing lists or qaforums and on magazines like stqe / better software that boil things down into short, simplified, presentations then you might never be exposed to enough serious professional writing to make sense of the schools anyway. That doesn’t mean you won’t adopt your own foundational views. It will mean that you’ll reinvent the wheel, probably with the same mistakes that the last N generations of people who invented this wheel made. In testing, I think that usually means a naïve version of the factory school or the control school.
· In some fields, some people are professionally eclectic. Different professionals have different reference examples for this. For me, a good example field is psychotherapy. Psychotherapists operate within rich intellectual traditions (not all of them—for example, some hacks are mainly focused on the business of psychotherapy than the service—but the ones I know have studied the intellectual history of their field and have their own viewpoint, and that viewpoint profoundly influences their work.)
o Some very senior people are professionally eclectic, and brilliant. The ability to say, “How would Freud have analyzed this situation?” in one case and “How would Jacque Lacan think about this?” for a different case – and really understand each one and really follow in the mental shoes of one for one case and the other for the other case – that ability is very special, and I think very rare.
o Some people say they are eclectic, but it really means that they haven’t found themselves yet, or that they’re not really good at anything.
o Some people say they are eclectic, but it really means that they are still thinking about techniques (and learning how to manage their practice or work as a therapist) and haven’t faced the life crises that make them ask themselves what they really believe. (I guess this is another way of saying they haven’t found themselves yet.)

=========
Schools are not about technical skill sets. They are about the foundational visions of the field within which people develop and deploy their skill sets.

Similarly, schools are not about specific contexts (such as types of applications). It is a damning criticism of a school, a way of demonstrating the superficiality of its thinking, to say that it applies onto to one type of software or one type of customer. A school that tells you to treat all situations in much the same way TACTICALLY (same techniques, same “best” practices) across different categories of product, customer or risk—well, to me, that’s a poor way of thinking. In other fields, that kind of intellectual narrowness relegated a lot of schools to history books. But then again, what answer would you expect from someone in the context-driven school?

-- Cem Kaner

Heuristic is....

2008-02-11T17:00:00.000Z

"A fallible method of solving a problem or making a decision."

There are only two key ideas that make a heuristic: 1. It may help solve problemor make a decision. 2. It may not.

I consider a test idea to be any idea related to a test you might want to perform.

A heuristic (as a noun) is "a(fallible) method for solving a problem or making a decision". Some peoplealso say "guideline" or "rule of thumb". When used as an adjective (e.g. "aheuristic approach"), "heuristic" means "conducive to learning (albeitfallible)". Contrast "heuristics" with "algorithms", which I'll define hereas "a step-by-step procedures or recipes for solving one particular class ofproblem".

Heuristics are
- not to be confused with rules or edicts
- context
-dependent, situational
- potentially contradicted by other heuristics
- to be used well, need to be used by someone with the appropriate level ofjudgement and skill to choose and use them appropriately- not to be considered perfect, but good enough.
- typically "fast and frugal"-
-much faster than more rigourous and complexforms of analysis, plus
- according to Gigerenzer, often more accurate than other rigourous andcomplex forms of analysis

The hidden secret is that /all/ of our methods of solving problems or making decisions are heuristic--good-enough approximations. Even mathematics orphysics are fallible outside of a context in which (There's a wonderful linefrom "An Introduction To General Systems Thinking", by Weinberg, that goessomething like: "Mechanics, then, is the study of things for which theapproximations of mechanics work sufficiently well.")

If you want to understand heuristics, I STRONGLY recommend GerdGigernenzer's recent book "Gut Feelings", or George Polya'snot-at-all-recent "How to Solve It", or Billy Vaughan Koen's recent butnow-hard-to-find book "Discussions of the Method." They're all verydifferent books, in fact, so maybe that "or" should be an "and".

(ideas stolen from James Bach, Cem Kaner and Michael bolton)

Product Maps (M. Bolton)

2008-01-29T11:16:00.000Z

>1) How do you empirically or quanitatively measure "dept of coverage", doyou have an alogorithm or formula ?

I empirically and heuristically (but not quantitatively) assess (but notmeasure) depth of coverage, thus I have no algorithm or formula. Somepeople claim to have reliable algorithms or formulae, but I've never seenone that, on its own, stands up to critical scrutiny. Code coverage toolsare wonderful examples: someone can show me that they've exercised everyline of code in the product, but that doesn't really tell me anything usefulabout what they observed and evaluated.

2) Would you please provide me with a real example of what is being referredto as a "Map of the product" - and how do you build one? And how is "Mapcompleteness measured?"
To me, a map is anything that models some space such that it links ideas andreality. A map may exist only in the mind; it might not even hit paper orpixels, but a representation of it might in some form. For me, those formscan include diagrams, lists, mind maps, tables, narratives, checklists,flowcharts, matrices (all of which I have used effectively), cause andeffect diagrams (which I haven't)--you name it. These are all typicallysupplemented by description, often in the form of conversation. I'm usuallybiased towards maps and lists that are fast and frugal and that developretrospectively--that is, maps that I fill in through testing--rather thanmaps that attempt to be comprehensive from the get-go, since we can bepretty sure they won't be.

Examples abound:

Test coverage outlines and risk lists:http://www.satisfice.com/rst-appendices.pdf has several, in several forms.See pages 59-94, especially, for examples of this in list and tabularformats.Mind maps: Rob Sabourin's feature article in StickyMinds from November2006.

Test matrices: http://www.developsense.com/testing/TestMatrices.htmlAnnotated structural diagrams: Older versions of Rapid Software--and we mayrestore them in the future; plus an upcoming presentation at STAR East.

Checklists: Elisabeth Hendrickson's Test Heuristics Cheat Sheet; MichaelHunter's You Are Not Done Yet list.

Guideword heuristics:
a) James Bach's Heuristic Test Strategy Model(http://www.satisfice.com). Cem has a very interesting way of using this toanalyze a requirements document; use exactly one of the guidewords to tageach statement in a requirements document, then assess which areas areover-represented (and thereby, which ones are underrepresented).

b) the list of heuristics that NASA used for geological surveys for the Apollomissions, also in http://www.satisfice.com/rst-appendices.pdf.

---Michael B.

Big Up-Front Design (still...)

2008-01-29T10:29:00.000Z

BUFD / BUFP = Big Up-Front Design / [ditto] Process.

Matt Heuser wrote: "I believe this is what Weinberg calls the primary management sin: 'If it ain't working, do more of it.'"

Also reminds me of my guitar teacher: "It is not because you practice your mistake a thousand times that it will become correct"

Customer collaboration over contract negotiation

2008-01-29T10:00:00.000Z

Don wrote:
“a requirement specification was supposed to detail the problem space: business context, business goals, business data, business rules, and business constraints: but usually the problem was insufficiently well described for an adequate product to be developed to reliably resolve it. As a result, lots of time was wasted designing and building inappropriate products, which failed to incorporate or correctly implement fundamental business rules regarding data control, transformation, and information generation (roughly, input-process-output). These requirements were nothing to do with software, but everything to do with achieving business success.”

As a law student, I heard story after story from commercial law specialists of lawyers who insisted that their contracts lock down all the details. It was an interesting controversy:

· On the one hand, there was the risk of inadequacy of the contract, because unanticipated details would cause later disputes.
· On the other hand, negotiation after negotiation failed because the legal costs were so high and the negotiating delays were so long.

Later, as a lawyer, I got to see some of these disasters myself and, through my work on computer-related legislation, I got to discuss contracting with people who had negotiated Big Deals and participated in Big Expensive Failures. It was a second education.

In my own negotiations as a lawyer, I adopted two key heuristics:

1. In deals of any complexity, there will always be holes in the understanding of the parties. Even if a problem has been solved by many other people, this problem probably hasn’t been solved this way before by these people in this deal. Their education will come to some degree as they negotiate the deal, but to a larger degree as they work through the project (do the work together) and gain life experience.
2. A contract is a symbol of an underlying agreement. No amount of paper will compensate for bad faith or protect the parties against it. Between parties of good faith, it is more valuable to specify a dispute resolution process that helps the project move forward when unexpected things happen than to try to anticipate all of the future contingencies.

I heard lots of preaching in favor of locking down every last detail. And we certainly studied lots of horror stories (that is, after all, what lawsuits are all about). But if our client didn’t have infinite time and an infinite budget for “transaction costs,” and if the other party(s) didn’t just agree to whatever we asked for, then negotiations had to stop and life had to go on long, long before all the details (what you are calling the project requirements) could be specified.

Rather than locking all of those requirements down—or blaming people for not being willing to achieve this impossibly expensive task—I saw my task as helping people find the smallest set of things that had to be locked down and finding a process for resolving the rest later.

I was delighted when the agile methods started gaining traction because they finally took seriously and constructively the same issue, presenting a technical-process solution to the need for ongoing requirements evolution, rather than legal-process (mid-project mediation, mid-project arbitration and other structures for supervising negotiation of the evolving deal).

Let me suggest a different pair of interpretations of what you are writing. When you say that lots of time was wasted creating the wrong things in the wrong ways, I suspect that:

· Lots of time was wasted because the vendor had no incentive to meet regularly with the client to steer the project to success, and the client was locked into a contracting process that gave them little authority to steer without driving their costs through the roof. The contract replaced the requirement of ongoing mutual-satisfaction-oriented communication. That is the essential difference between BUFD (big up front design) and iterative development (read Tom Gilb on evolutionary development—he wrote about using it with the air force long before XP was a formalized methodology).
· Lots of time was wasted because the vendors who play in this space are the ones who have learned how to profit from a grossly inefficient development process. There was lots to dislike about Donald Rumsfeld (our previous Secretary of Defense), but this was one of the issues he raised time and again. Rather than failing more expensively with larger warehouses full of paper, Rumsfeld wanted a more agile contracting and development process. I’m not sure how successful this has been, but I am sure that a big part of the problem has been inertia on the part of the vendors.

During my legislative work, I spent a lot of time working with lawyers who represented (or worked for) insurance companies, banks, and other big non-government institutions. One of the most striking things was the extent to which they mistrusted (sometimes, hated) software vendors (especially custom software development companies) compared to their attitudes toward other vendors in other industries they dealt with. I think BFUD (oops, I mean BUFD) has played a big role in the evolution of that profound dissatisfaction.

- Cem Kaner

Testers find bugs that users don't find

2008-01-14T15:26:00.000Z

It is hard to admit it, but many times we testers find bugs that a user most probably won't find.

For instance, I am testing piece A that is designed to work with piece B.
They seem completely independant, but in real life they are used always toghether.
However I, as a tester, set up piece A and test it immediatelly (unit testing, remember?).

It fails. I find later it only works when opiece B is correctly set up and working.

A system administrator would not go across this, because he would set up piece A and pice B then would try the system...

Encourage Software Testing as a career (or not)

2007-11-15T15:28:00.000Z

Cem Kaner's thoughts about this:

I don't generally encourage my students to pursue software testing AS A CAREER. They can make that decision later, after they have more experience.

I prefer to encourage them to try SOFTWARE DEVELOPMENT as a career -- to me, development includes testing. And that they take a job doing serious, skilled testing as PART of that career.

Most of the best testers I know have significant experience outside of testing and apply that experience to what they do as testers or test managers.

Most (all?) of the hidebound process-pushers that I know in the field have never done serious professional work outside of testing. From their narrow perspective, they think they know more about how to manage a development project than the people who retain their testing services. Instead of trying out their ideas as project managers (where they will be accountable if they fail) these process advocates undermine the projects they work on by trying to control things they don't understand with rigid policies and procedures, standards and superstitions, whose costs and impacts are beyond their imagination. We have too many of these people in our field. We need more people who have a broader view of the tremendous value that testing can offer--within its limited role--and are glad to embrace a service-provider role that provides that value.

What makes testing worth spending time on--as a job and maybe as a career?

We are professional investigators. Rather than building things, we find ways to answer difficult questions about the quality of the products or services we test. Our job--if we choose to do it well--requires us to constantly learn new things, about the product, its market, its implementation, its risks, its usability, etc. To learn these, we are constantly developing new skills and new cognitive structures in a diversity of fields. It also requires us to communicate well to a diverse group of people. We ALSO get to build things (test tools), but very often, we build to our own designs, which can be more satisfying than building an application that does something we'll never personally do (or want to do). Learning to do good software testing requires learning to do critical thinking well, and to back it up with empirical research.

Not everyone will like to do testing. Not every engineer or programmer will have the skills or the interest to do professional-level testing. But for those of us who enjoy critical thinking, experimentation, and keeping the human relevance of what we do always in mind, there is nothing else like it in software development (except, for some people on some projects, requirements analysis backed with rapid prototyping and prototype-based research).

-- cem

Should major defect be fixed earlier than usability/interface defects?

2007-11-14T12:18:00.000Z

Usually there is a bug tracking system in which the reporter (tester) assignes a severity to the defect, and then a developer leader assigns a priority. Usually the usability issues are not as severe as crashes, for instance, and so are assigned more priority and fixed earlier. On the other hand, the usability/interface issues are postponed to a later phase "when the product is more stable"

Now something about the context. I am talking about a big size commercial application, and I am refering to an alfa version. The product is not officially released, it was already delivered to testing team, but there is a long way to go until it is beta ready (let's say about 6 months.

I think that in this scenario, interface and usability issues should be prioritized above other problems, like crashes or errors, provided that these are not blocking tests and development evolution.

Here are the reasons for this:
-Interface changes like uniformizations, tend to spread to many areas of the aplication, and the cost tends to be high. By fixing this earlier we are reducing the cost of the fix.

-Interface changes are likely to introduce unstability and break the code on many areas, so the sooner these major changes are done, the sooner these new defects can be fixed.

-Incorrect Labels cause confusion: testers and tech writers will have to talk to developers to understand the software, developers will have to do extra work to find, understand and fix things, etc. So this "minor" defect can increase the cost of software while fixing this would be very cheap. (I would include here the names used in the code, whose importance tends to be underestimated by the developers. I which they could count the defects caused by naming issues, and the time spent finding badly named variables, etc...)

Does anyone agree with me on this?
If so, any tips on how can we explain to leaders that "label is confuse" minor defect should be fixed earlier than that uggly "application crash" defect?

Creativity in software design: greatest dangers

2007-10-31T09:08:00.000Z

Creativity has always been pointed as a desirable characteristic for a developer, but there are some drawbacks that we should be aware as testers. For instance, consider interface issues.

When: when you find something that you don't understand at first. Then you see it is a completely new way of doing something.

Beware with your repulse: because you might tend to reject it just because it is new. People are emotionally attached to well known and predictable software, thus fearing everything different from standards.
Beware with developers defense: people are emotionally attached to their own creations, for a variety of reasons:
- Because it took a lot of writing work (coding)
- Because it took a lot of creative work
- Because people need creativity to fulfil their lives
- because they really think it is a great idea

When is it acceptable (the new solution)?
-It is the best way to do the task (usability criteria)
-It is easy to find out

Installation testing: should it be so hard?

2007-10-12T10:07:00.000+01:00

I share your concern.
It is indeed a huge task to verify setups, because if you need to repeat the test you tipically must:
1-Uninstall
2-Check that everything was removed (like registry stuff, etc) – Probably you have to verify the uninstall feature as well…
3-Install
4-Perform the verifications

In my company we have patches every week, and you cannot imagine the immense diversity of problems such as:-This file is missing because we forgot to include it-This file is missing because we did not know we needed to include it-This other team did not tell us this shared file was changed, so we did not include it in our module.-There were “compilation problems” (corrupted files)

It seems to me that the way to resolve this problem is more on their side. I mean, they should find procedures and methods that are more reliable, more automated (in the sense that they don’t have to think every time about what files they should include, etc). The more tasks they have to do “by hand”, the more likely they are to miss some. They should get check lists or any other tools to help them.I ought to say that if a file is missing from a release, the responsibility is from who released the product.I don’t see our function (tester function) as a parachute to developers work, because I guess it is impossible to verify everything.
Quality is not responsibility of the quality department (QA, Test team, whatever you call it); it is responsibility of the whole R&D team.(I read this on paper “classic testing mistakes”, I think)
I’m afraid I did not help much to achieve your task, but I hope you have more arguments to reduce it to more “reasonable” levels 

Joao Pedro

________________________________________From: software-testing@yahoogroups.com [mailto:software-testing@yahoogroups.com] On Behalf Of John McCondaSent: quinta-feira, 11 de Outubro de 2007 17:33To: software-testing@yahoogroups.comSubject: [software-testing] Installation Testing
Hi everyone.
I'm perplexed by a problem that seems fairly simple on the outside,but has proved to be a frustrating part of testing on a particularproject. I will try to include as much detail as I can here withoutbeing overwhelming.
The basic problem is installation testing. We test a C++ client/serverand Web app that comes with an installer built with Install Shield.The app has existed since 1998 and has gradually grown very large andcomplex, with multiple services and modules needing to be installed inthe folder structure.
There is one developer who does nothing but installers for theproject. The other developers create a build folder on a server thatthe install developer uses to create the installation exe.
Currently, it is the test team's job to verify each installation invarious ways. This includes running every new installer that iscreated and then using a tool called Beyond Compare to compare thefiles that are installed with the build folder to ensure all files areidentical. Secondly, for each build, the testers must compare allfiles of a new installed build with an installed build of the lastversion and note that all changes are expected.
We often find problems when we do these checks, and it will come backthat the install developer left something out or had the wrong versionof a file and we will need to do the whole process over again.
This seems horribly inefficient to me, and takes much time away frommy team doing actual testing. This is my first time testing somethingwith such a complicated installer though.
Has anyone else found a better way to deal with this type of problem?Are there any good tools for creating installers I can suggest to theinstall developer that will take some of the burden off of the test team?

-------------Here is another response, this time from the setups developer point of view:

John,
In my last company I was also on the other side of the fence, that is,I was the one who had the responsibility of creating the installerbesides testing them. To add to the complexity, the product I createdinstaller for, was supported on windows platform plus 3 flavours ofUNIX [HP, AIX & Solaris].
Below is the approach I took:

1. Created a single Installation script file (we used Install Anywhereso that helped) for all platforms.
2. Created a single ANT script that did the following:
a) First, it would check the build date of all the files that are tobe packaged and compare them to the last check-in date (since allfiles were sourced from a Version Control System)
b) Second check was to make sure that none of the file is of zero size.
C) Then created installer using Install Anywhere's command line utility.
d) Once the installer was created, we executed the installer in silentmode and did a complete installation for each platform.
e) Once the installation was complete, we again check that all thefiles are copied in appropriate directory structures and the values inproperties files are changed appropriately (compared to baseline files).
f) Then brought up the product for Testing team to continue working.
f) Once all the checks passed, the installer was copied to a centrallocation.
If any of the check failed, a mail was sent to appropriate people withall the details. Plus if the post-installation check failed, then thescript used to remove the new installation and restore the backup ofworking setup.
All the installation were done on the setups that were used daily bytesting team.
Now since you are not the one who is creating the installers, you'llneed to have development's buy-in for setting up such kind of process.But if you can translate this in to scenario specific to you, I'm surethe value derived would be too great for development to look other side.

Hope this helps.
ThanksNeeraj

"Once we're off the map, we don't know where we're going next"

2007-10-08T10:46:00.000+01:00

By Michael Bolton.

A reply to a testing enginneer that looked advive on how to persuade developers to fix "small defects"

>I have the following question: what should I tell to the developer(s)when an error occurs with a low severity, occurring in specificsituations with a low probability of happening in real world, when heasks me where or how this error can affect the safety of the system?Every time the developer says that, I can't seem to find a logicalexplanation on how the defect can affect the safety.

The answer that I'd give to this question is that any defect for which wedon't have a good, clear explanation is a potential security threat. Oneway to deal with the problem is to ask the developer, "Can you be sure thatthis problem /doesn't/ represent a threat to the safety of the system? Whatmakes you /sure/? And if you're sure, why don't we handle this condition ina way that makes our certainty evident?"

One thing that tends to make us "sure" that a small symptom isn't a bigproblem is something called "representativeness bias". We have anapparently natural tendency to associated the significance of the symptomand the significance of the underlying problem. In complex systems, thefirst sign of a problem may not be a big sign.

One prominent and memorable example of this is the Challenger disaster.O-ring seals, intended to keep gasses from coming together, degraded.Instead of remaining intact, they burned partially through. NASA managersused the fact that they didn't burn ALL the way through as evidence ofsafety, but (as is now obvious) the opposite was true. As Richard Feynmansaid in his Appendix to the Rogers Commission Report on the Space ShuttleChallenger Accident (I've added emphasis below),

[quote]
There are several references to flights that had gone before. The acceptanceand success of these flights is taken as evidence of safety. /But erosionand blow-by are not what the design expected. They are warnings thatsomething is wrong. The equipment is not operating as expected, andtherefore there is a danger that it can operate with even wider deviationsin this unexpected and not thoroughly understood way./ The fact that thisdanger did not lead to a catastrophe before is no guarantee that it will notthe next time, unless it is completely understood. When playing Russianroulette the fact that the first shot got off safely is little comfort forthe next. The origin and consequences of the erosion and blow-by were notunderstood. They did not occur equally on all flights and all joints;sometimes more, and sometimes less. Why not sometime, when whateverconditions determined it were right, still more leading to catastrophe?
[/quote]

When the system is in an unpredicted state, it's in an unpredictable state.Once we're off the map, we don't know where we're going next.

---Michael B.

(Kaner)

perhaps take a look at the video at http://www.testingeducation.org/BBST/Bugs1.html

and my contribution:

Sometimes it is not easy to give up on a defect after spending several hours trying to figure it out. But sometimes it is the wiser thing to do.

The story usually goes like this:
-We find something that behaves in an unexpected way.
-We try to find out the causes with further testing, perhaps reading documentation, searching the web, talking to the developer.
-Eventually we find out that: it is not noticeable to the user, it does not compromise safety for sure, the cost of fixing clearly does not cover the benefit of having it fixed, or it is just a matter of logical/mathematical consistency.

Sometimes we like to keep everything in the right shelves, even if we’re never going to need them. It kind of gives us “peace of mind”. If we recognize that the reason for wanting the defects fixed is only psychological, it will be easier to let these defects alone.

Anyway, all these defects should be reported formally, and it is not our job or responsibility to discard them. But it is our responsibility not allowing them to be discarded when we have some reason to believe that the consequences might not be completely studied (or more serious than what is acceptable)

Joao Pedro

Is a bug in the code necessarily a bug in the process?

2007-08-09T10:45:00.000+01:00

(pasted from agile forum)
>As Jim Shore says in his forthcoming book, a bug found in released code is
not only a bug in the code, but a bug in the development process. What
caused this bug to be produced? What caused it to not be caught prior to
release? How can we prevent this error in the future. The causes may be
anywhere from a coding error to a product concept that's not clear.

In traditional models of testing, I rarely see any credence given to the
idea that we can (even hope to) know everything before we start. This is
one of the reasons that I like agile--and sometimes Agile--processes, and
it's a reason why I'm enjoying "Artful Making" (Austin). I'd argue that a
bug is not necessarily linked to a bug in the development process if we
learn something from it. Austin goes so far as to say that it's not
necessarily a problem if we fail in the same way ten times, because we might
need ten iterations before we can get insight into what the problem really
is.

---Michael B.

Creating a more open team

2007-07-10T11:10:00.000+01:00

There is an ongoing thread on software-testing forum about how you may turn your team into a great team, in what concern knowledge sharing, interest for investigating new ideas and state-of-the-art testing practices. It all started with this email:

"I work in a team and I'd like to look at moving the team away from what often seems like a team of individuals and create a more open culture where people willingly talk about experiences gained within projects, techniques being used outside of our company and other conversations like simply mentioning an article they've read regarding testing. Has anyone out there had a similar culture-shift they've tried to achieve and therefore has a suggestion on ways to go about this?"

Here are some suggestions:

"A wiki is the ideal tool to do this sort of thing. If your team is not experienced using wikis, you might consider designating some experienced team members as "content creators" at the beginning, and make it part of their job to find and post interesting and important things to the wiki. Less experienced team members might take a more passive role, but be sure that the less-experienced are encouraged to read and comment on the content. Make it so that everyone feels that contributing to the community on the wiki is an important activity. "

"Creating forums is a good idea. You would need to have one dedicated team to drive the whole activities. You can also have brain storming sessions, solution boards.

You can also initiate Knowledge Exchange Session by bring prominent personalities across different areas (it could be testing, architects, domain experts etc)

Reward programs for sharing knowledge will be a good motivational tool to bring about the willingness in sharing experiences."

"I've found the wiki we use at my place of work to be a good place to put processes and how-to guides, but I honestly think that it won't encourage your team to further interact with each other. You can fill it full of useful knowledge, but that's not what makes a team great.
Try a practical demonstrations of the value of working more closely together. I recently reproduced an exercise from the 'rapid software testing' course I attended recently (thanks James :) ), whereby I had the team look at a completely unfamiliar gui and write down what they thought it did and how they'd test it. After a few minutes, I had them read their answers, pressed them a bit for more. Every single person in the room (8 of them) said something different (and useful). That five minute exercise alone was enough to show them that while they have a common set of skills, they also have different approaches and ways of thinking about a problem. They are now much more apt to ask each other questions or opinions than they previously were. I am encouraging the team to come up with ideas and exercises and questions that they have thought of or discovered, and present them to the rest of the group.

Maybe you should be asking them what sort of forum they would enjoy using to communicate their ideas. That question alone might be a good one to get intra-team discussion happening. You could have your test leads present a test post-mortem for a given project, have them identify (and defend) key decisions they made, what risks they identified, how they reacted to adversity and so on - then open it up to the rest of the group for comments and questions. You may need to encourage the quieter ones to speak up, but that will happen regardless. I find that people want to feel useful. If you provide a forum that encourages people to interact, to share their knowledge in a way that is useful to others, then you will probably get more buy-in than giving them a screen to regurgitate their thoughts into."

"Utilize the stickyminds editorials, find some older ones perhaps that are relevant, with some interesting comments as well, send out the link and discuss it over lunch. That will start the process going.
Another possibility now is just watching some of the google testing videos presented by testing luminaries over lunch, then having a discussion. At the google page, select Video then "Google techtalks testing", which finds 66 videos!
It has to be within the scope of people's work commitments though: I often push for people to have an hour or two a week (on average) for private study/research as part of their job role, and encourage them to present their discoveries etc at brown bag lunches."

"A nice idea (the open team one that is) but be careful not to "create" anything artificial or cringe making.
I can't count the number of times I've been on the receiving end of horrific artificial "team building" exercises.
I've watched these things alienate teams rather than build them or open them.
I suppose they do give the team an opportunity to bitch about something they have in common, but is that what you are after? ;-)

The wiki idea actually seems counter productive to your goals to me i.e. just another excuses for technical folk NOT to talk to each other.

The only thing I think every member of your team will have as a common goal is "better work/life balance".
For me, better planning and use of human resources i.e. going home on time AND having lunch (by myself or with friends), is what floats my boat.
I moved from permanent work to contract/consulting work because it actually encouraged my management to send me home on time i.e. every extra hour I spent in a day it affected their budget."

TestLink after one year...

2007-07-06T17:51:00.001+01:00

We used Word documents and Excel worksheets as our documentation tool for scripted testing, and at one point I was given the task to evaluate and choose a new tool.

TestLink is a browser based tool to document Test scripts (planning) and execution results. It was chosen because it has some interesting features:
-there is an area for the tests design and another for the tests execution. Same test Plan (design) may be instantiated several times, for example for different platforms. It also supports Builds. Every result is easily stored and consulted.
-Main metrics are built-in. For example we may see at any point how many tests have passed, failed, blocked, not run
-Roles are supported: a tester has different privileges than a team leader, an administrator or a test designer
-Through Access rights, test executions are easily assigned to testers. Each tester only sees what matters.
-It is possible to import test cases from excel worksheet (though we had to make a macro to convert word to excel…)
-It is possible to connect to a bug tracking system

Note that this is not a tool for automation, or for calling automated scripts. It is a tool to store test documentation.

Now what is not so good:
-it does not link smoothly with exploratory testing; so one should think what is the real value of scripted testing for the System Under Test; in our case we do have to guarantee some basic features recurrently, so we do need some detailed test execution record. If the tests are not to be re-utilized, the usefulness of TestLink should be questioned.

Brilliant post from Michael Bolton, related with context-driven

2007-07-06T17:25:00.000+01:00

Imagine reading this message:

"I'm looking to take a drug, but don't have the budget for the name brand pharaceuticals, and want more than is provided by folk remedies. I'm looking at ampicillin and ritalin, but don't know anyone who has taken these drugs. Does anyone have any experience with these drugs, or other drugs that they would recommend?"

Wouldn't you be curious to know what malady the writer was interested in relieving?

---Michael B.

From: agile-testing@yahoogroups.com [mailto:agile-testing@yahoogroups.com] On Behalf Of gmel74Sent: July 5, 2007 10:17 AMTo: agile-testing@yahoogroups.comSubject: [agile-testing] test management tools

I'm looking to implement a test management tool, but don't have the budget for the name brand solutions and want more than is provided with the open source solutions. I'm looking at TCM by Seapine and Devtest from techexcel, but don't know anyone that has used these tools. Does anyone have any experience with these tools or other test management tools that they would recommend?

4 probes to a medium

2007-06-28T18:53:00.000+01:00

Here is an enlightened post from Michael Bolton (as usual):

Marshall McLuhan talked a lot about media. A lot of people don't realize that his ideas were different from the ones that we usually associate with the words that he used. To McLuhan, a medium is anything that effects a change--a new tool or technology, an idea, a programming language, a hat... anything that appears as a new figure somewhere in an existing ground.

In order to investigate the impact or significance of a medium (or as McLuhan called it, the medium's "message"), McLuhan proposed four probes, or questions, that one could ask about a medium:
1) To what extent does this medium enhance or extend or intensify or expand or enable some human capability?
2) What previously obsolete medium (perhaps from a vastly different field or from a long time ago) does this new medium retrieve, or remind us of?
3) What existing (and perhaps ubiquitous) medium does this new medium make obsolete?
4) How might this medium reverse its intended or expected or beneficial effect when it exceeds its capacity, or when it becomes the norm? Every medium, said McLuhan, exhibits the general characteristics that the probes ask about--extension, retrieval, obsolescence, reversal. The intention is to try to look at a new or anticipated medium using these probes.

(Mark Federman (http://whatisthemessage.blogspot.com ) explains Marshall McLuhan's Laws of Media very well and very clearly; "Creating a Culture of Innovation" is the paper to look for on the right of the blog page. Even better is his book, co-authored by Derrick deKerckhove, "McLuhan for Managers", which is hard but still possible to get.)

So: the test team enhances the ability of the product team to find bugs, to report problems, to understand the product. It retrieves the idea of the test pilots, the food taster, the editor, the lab researcher. It obsolesces, at least to some degree, the field technician, the support centre, the unhappy customer. And it reverses into the developers who won't check their own code, blaming and fingerpointing, mindless confirmation of trivial test ideas, automated behaviour, product managers passing responsibility for release decisions on to the testers...

That list is after a minute or two of brainstorming; it could be much longer. What would happen if you spent 15 minutes on the with your test team, or your development team, or your managers? You might be able to set clearer goals (or limits) about what you want to extend, how you want to model the team, what or who the team might make redundant, and (at least) to anticipate and control for some of the reversal effects of the test team.

One more thing: rather than thinking about metrics, try thinking about observations that you might make. If it makes sense simply to describe the observation, do that. If it makes sense to measure the observation, remember that metrics are media too. That is, each metric you use will extend, retrieve, obsolesce, and reverse some aspect of your observation.

---Michael B.

See the sticky minds article here.

tester skills

2007-06-27T12:40:00.001+01:00

I wouldn't say better, so here it goes:

>A tester with no programming experience at all will not be able to keep up.

Keep up with what?

James Bach, Becky Fiedler and I are having an amusing time this week wrestling with bibliographic software. EndNote (Thomson) is the leading package, but it has bugs. Thomson bought out EndNote's leading two competitors a few years ago. Not surprisingly, progress in advancing the quality of the product has not been so quick.

From a bibliography management viewpoint, I think the EndNote people understand their market's needs well, but they have a multi-year history of corrupting files (including too many of my files). I suspect that their product might benefit -- A LOT -- from adding a few testers with stronger programming and debugging skills. As stands, Becky, James and I are each ready (enthusiastically ready) for a change.

So I was initially glad to see Microsoft bringing some competition to this market by adding bibliographic support to MS Word. I suspect that their software engineers in test did a fine job checking for file-corrupting bugs. On the other hand, I think this product would have benefitted enormously from serious evaluation by people who really understand how bibliography managers are used by graduate students, researchers and technical writers. In its current form, I think the product is useless for us as writers and I would not recommend it to my undergraduates when they are writing their first essays.

A tester with no programming experience will not be able to keep up with programming-related issues, but a tester with no understanding of the non-programming domains will not be able to keep up with those many aspects of the product's quality (quality = value to some person) that have nothing to do with the technical quality of the code but everything to do with whether it is worth using.

Rather than insisting that all testers be programmers or all testers be domain experts (as if any product involves just one domain), I far prefer a staffing model in which people have a variety of capabilities. Some people know code. Some people know peripherals. Some people know accounting (if that's relevant), etc. They cross-train, they pair-test, and they keep up as a team. Better a group of strong individuals than a group of mediocre generalists. And better a group of mediocre generalists than a group of programmers who know a lot of test automation and don't understand what the people who will (not) use the product will (not) use it for.

-- Cem Kaner

Pair-programming

2007-06-15T17:46:00.000+01:00

I only tried pair-programming when I was in college, but now we do something like pair-testing, for example when someone more skilled helps another; or when we reproduce defects so the developer “watches” them happen.

Besides everything else, I think pair-working helps to keep the focus, thus saving a lot of time. I bet we won’t check new email, handle phone calls, surf the web, reply to forums like these… while we are pair-working. So we should count that as time saving, to balance the cost of having 2 workers on the same task. Due to this “’forced’ focus, we also save a lot of brain effort in context-switching.

The drawback is that, as in any focused, concentrated, demanding activity, our mind will not use the self-resting and balancing mechanisms that would work if we were alone. I would dare to say that 8 hours a day on pair-programming must harm our mental health.

Does the words meditation and silence ring any bells in this context?