>1) How do you empirically or quanitatively measure "dept of coverage", doyou have an alogorithm or formula ?
I empirically and heuristically (but not quantitatively) assess (but notmeasure) depth of coverage, thus I have no algorithm or formula. Somepeople claim to have reliable algorithms or formulae, but I've never seenone that, on its own, stands up to critical scrutiny. Code coverage toolsare wonderful examples: someone can show me that they've exercised everyline of code in the product, but that doesn't really tell me anything usefulabout what they observed and evaluated.
2) Would you please provide me with a real example of what is being referredto as a "Map of the product" - and how do you build one? And how is "Mapcompleteness measured?"
To me, a map is anything that models some space such that it links ideas andreality. A map may exist only in the mind; it might not even hit paper orpixels, but a representation of it might in some form. For me, those formscan include diagrams, lists, mind maps, tables, narratives, checklists,flowcharts, matrices (all of which I have used effectively), cause andeffect diagrams (which I haven't)--you name it. These are all typicallysupplemented by description, often in the form of conversation. I'm usuallybiased towards maps and lists that are fast and frugal and that developretrospectively--that is, maps that I fill in through testing--rather thanmaps that attempt to be comprehensive from the get-go, since we can bepretty sure they won't be.
Examples abound:
Test coverage outlines and risk lists:http://www.satisfice.com/rst-appendices.pdf has several, in several forms.See pages 59-94, especially, for examples of this in list and tabularformats.Mind maps: Rob Sabourin's feature article in StickyMinds from November2006.
Test matrices: http://www.developsense.com/testing/TestMatrices.htmlAnnotated structural diagrams: Older versions of Rapid Software--and we mayrestore them in the future; plus an upcoming presentation at STAR East.
Checklists: Elisabeth Hendrickson's Test Heuristics Cheat Sheet; MichaelHunter's You Are Not Done Yet list.
Guideword heuristics:
a) James Bach's Heuristic Test Strategy Model(http://www.satisfice.com). Cem has a very interesting way of using this toanalyze a requirements document; use exactly one of the guidewords to tageach statement in a requirements document, then assess which areas areover-represented (and thereby, which ones are underrepresented).
b) the list of heuristics that NASA used for geological surveys for the Apollomissions, also in http://www.satisfice.com/rst-appendices.pdf.
---Michael B.
Tuesday, January 29, 2008
Big Up-Front Design (still...)
BUFD / BUFP = Big Up-Front Design / [ditto] Process.
Matt Heuser wrote: "I believe this is what Weinberg calls the primary management sin: 'If it ain't working, do more of it.'"
Also reminds me of my guitar teacher: "It is not because you practice your mistake a thousand times that it will become correct"
Matt Heuser wrote: "I believe this is what Weinberg calls the primary management sin: 'If it ain't working, do more of it.'"
Also reminds me of my guitar teacher: "It is not because you practice your mistake a thousand times that it will become correct"
Customer collaboration over contract negotiation
Don wrote:
“a requirement specification was supposed to detail the problem space: business context, business goals, business data, business rules, and business constraints: but usually the problem was insufficiently well described for an adequate product to be developed to reliably resolve it. As a result, lots of time was wasted designing and building inappropriate products, which failed to incorporate or correctly implement fundamental business rules regarding data control, transformation, and information generation (roughly, input-process-output). These requirements were nothing to do with software, but everything to do with achieving business success.”
As a law student, I heard story after story from commercial law specialists of lawyers who insisted that their contracts lock down all the details. It was an interesting controversy:
· On the one hand, there was the risk of inadequacy of the contract, because unanticipated details would cause later disputes.
· On the other hand, negotiation after negotiation failed because the legal costs were so high and the negotiating delays were so long.
Later, as a lawyer, I got to see some of these disasters myself and, through my work on computer-related legislation, I got to discuss contracting with people who had negotiated Big Deals and participated in Big Expensive Failures. It was a second education.
In my own negotiations as a lawyer, I adopted two key heuristics:
1. In deals of any complexity, there will always be holes in the understanding of the parties. Even if a problem has been solved by many other people, this problem probably hasn’t been solved this way before by these people in this deal. Their education will come to some degree as they negotiate the deal, but to a larger degree as they work through the project (do the work together) and gain life experience.
2. A contract is a symbol of an underlying agreement. No amount of paper will compensate for bad faith or protect the parties against it. Between parties of good faith, it is more valuable to specify a dispute resolution process that helps the project move forward when unexpected things happen than to try to anticipate all of the future contingencies.
I heard lots of preaching in favor of locking down every last detail. And we certainly studied lots of horror stories (that is, after all, what lawsuits are all about). But if our client didn’t have infinite time and an infinite budget for “transaction costs,” and if the other party(s) didn’t just agree to whatever we asked for, then negotiations had to stop and life had to go on long, long before all the details (what you are calling the project requirements) could be specified.
Rather than locking all of those requirements down—or blaming people for not being willing to achieve this impossibly expensive task—I saw my task as helping people find the smallest set of things that had to be locked down and finding a process for resolving the rest later.
I was delighted when the agile methods started gaining traction because they finally took seriously and constructively the same issue, presenting a technical-process solution to the need for ongoing requirements evolution, rather than legal-process (mid-project mediation, mid-project arbitration and other structures for supervising negotiation of the evolving deal).
Let me suggest a different pair of interpretations of what you are writing. When you say that lots of time was wasted creating the wrong things in the wrong ways, I suspect that:
· Lots of time was wasted because the vendor had no incentive to meet regularly with the client to steer the project to success, and the client was locked into a contracting process that gave them little authority to steer without driving their costs through the roof. The contract replaced the requirement of ongoing mutual-satisfaction-oriented communication. That is the essential difference between BUFD (big up front design) and iterative development (read Tom Gilb on evolutionary development—he wrote about using it with the air force long before XP was a formalized methodology).
· Lots of time was wasted because the vendors who play in this space are the ones who have learned how to profit from a grossly inefficient development process. There was lots to dislike about Donald Rumsfeld (our previous Secretary of Defense), but this was one of the issues he raised time and again. Rather than failing more expensively with larger warehouses full of paper, Rumsfeld wanted a more agile contracting and development process. I’m not sure how successful this has been, but I am sure that a big part of the problem has been inertia on the part of the vendors.
During my legislative work, I spent a lot of time working with lawyers who represented (or worked for) insurance companies, banks, and other big non-government institutions. One of the most striking things was the extent to which they mistrusted (sometimes, hated) software vendors (especially custom software development companies) compared to their attitudes toward other vendors in other industries they dealt with. I think BFUD (oops, I mean BUFD) has played a big role in the evolution of that profound dissatisfaction.
- Cem Kaner
“a requirement specification was supposed to detail the problem space: business context, business goals, business data, business rules, and business constraints: but usually the problem was insufficiently well described for an adequate product to be developed to reliably resolve it. As a result, lots of time was wasted designing and building inappropriate products, which failed to incorporate or correctly implement fundamental business rules regarding data control, transformation, and information generation (roughly, input-process-output). These requirements were nothing to do with software, but everything to do with achieving business success.”
As a law student, I heard story after story from commercial law specialists of lawyers who insisted that their contracts lock down all the details. It was an interesting controversy:
· On the one hand, there was the risk of inadequacy of the contract, because unanticipated details would cause later disputes.
· On the other hand, negotiation after negotiation failed because the legal costs were so high and the negotiating delays were so long.
Later, as a lawyer, I got to see some of these disasters myself and, through my work on computer-related legislation, I got to discuss contracting with people who had negotiated Big Deals and participated in Big Expensive Failures. It was a second education.
In my own negotiations as a lawyer, I adopted two key heuristics:
1. In deals of any complexity, there will always be holes in the understanding of the parties. Even if a problem has been solved by many other people, this problem probably hasn’t been solved this way before by these people in this deal. Their education will come to some degree as they negotiate the deal, but to a larger degree as they work through the project (do the work together) and gain life experience.
2. A contract is a symbol of an underlying agreement. No amount of paper will compensate for bad faith or protect the parties against it. Between parties of good faith, it is more valuable to specify a dispute resolution process that helps the project move forward when unexpected things happen than to try to anticipate all of the future contingencies.
I heard lots of preaching in favor of locking down every last detail. And we certainly studied lots of horror stories (that is, after all, what lawsuits are all about). But if our client didn’t have infinite time and an infinite budget for “transaction costs,” and if the other party(s) didn’t just agree to whatever we asked for, then negotiations had to stop and life had to go on long, long before all the details (what you are calling the project requirements) could be specified.
Rather than locking all of those requirements down—or blaming people for not being willing to achieve this impossibly expensive task—I saw my task as helping people find the smallest set of things that had to be locked down and finding a process for resolving the rest later.
I was delighted when the agile methods started gaining traction because they finally took seriously and constructively the same issue, presenting a technical-process solution to the need for ongoing requirements evolution, rather than legal-process (mid-project mediation, mid-project arbitration and other structures for supervising negotiation of the evolving deal).
Let me suggest a different pair of interpretations of what you are writing. When you say that lots of time was wasted creating the wrong things in the wrong ways, I suspect that:
· Lots of time was wasted because the vendor had no incentive to meet regularly with the client to steer the project to success, and the client was locked into a contracting process that gave them little authority to steer without driving their costs through the roof. The contract replaced the requirement of ongoing mutual-satisfaction-oriented communication. That is the essential difference between BUFD (big up front design) and iterative development (read Tom Gilb on evolutionary development—he wrote about using it with the air force long before XP was a formalized methodology).
· Lots of time was wasted because the vendors who play in this space are the ones who have learned how to profit from a grossly inefficient development process. There was lots to dislike about Donald Rumsfeld (our previous Secretary of Defense), but this was one of the issues he raised time and again. Rather than failing more expensively with larger warehouses full of paper, Rumsfeld wanted a more agile contracting and development process. I’m not sure how successful this has been, but I am sure that a big part of the problem has been inertia on the part of the vendors.
During my legislative work, I spent a lot of time working with lawyers who represented (or worked for) insurance companies, banks, and other big non-government institutions. One of the most striking things was the extent to which they mistrusted (sometimes, hated) software vendors (especially custom software development companies) compared to their attitudes toward other vendors in other industries they dealt with. I think BFUD (oops, I mean BUFD) has played a big role in the evolution of that profound dissatisfaction.
- Cem Kaner
Monday, January 14, 2008
Testers find bugs that users don't find
It is hard to admit it, but many times we testers find bugs that a user most probably won't find.
For instance, I am testing piece A that is designed to work with piece B.
They seem completely independant, but in real life they are used always toghether.
However I, as a tester, set up piece A and test it immediatelly (unit testing, remember?).
It fails. I find later it only works when opiece B is correctly set up and working.
A system administrator would not go across this, because he would set up piece A and pice B then would try the system...
For instance, I am testing piece A that is designed to work with piece B.
They seem completely independant, but in real life they are used always toghether.
However I, as a tester, set up piece A and test it immediatelly (unit testing, remember?).
It fails. I find later it only works when opiece B is correctly set up and working.
A system administrator would not go across this, because he would set up piece A and pice B then would try the system...
Subscribe to:
Posts (Atom)