Right now one of the "louder voices in the room" for test automation is the "Agile" test automation voice. And by that I mean something very specific - that you create a large series of clerical tests that go from acceptance level all the way to unit; that you can run those tests at the click of a button, and get a green or redbar (pass/fail) very, very quickly. If any component in the system is slow, you can mock or stub it out to get that fast feedback. A few people who talk about this idea admit that you may want to actually run the system end-to-end, for example by running the full, slow test suite overnight.
This kind of test automation can be very valuable, especially for software developers, but it isn't the only kind.
Earlier this week, Ben Simo and I were kicking ideas around and we came up with:
1 Any tool that eliminates manual, clerical work is test automation (diff, loadrunner, tasker)
2 Finite-State-Machine, Random-Number Generating Tests are test automation (In the Harry Robinson sense, or, for that matter, the Ben Simo sense)
3 "Big FOR loop with an external, trusted Oracle" test automation
4 Develop it twice, run it on both systems, and compare the results test automation (Create two, not-very-trustworthy oracles)
5 Monkey Test Automation. Press Random buttons, record the order, wait for a crash, and then dump the log.
6 Pairwise Test Generation tools are test automation.
7 Any test case generator (perlclip) is test automation
This is a quick, sloppy list. What are some other forms of test automation? What am I missing?
It would be awful nice to come up with a consistent model for different types of testing tools. Failing that, I'd settle for a vision, or, to be honest, a "good start."
Elisabeth Hendrickson just announced the Agile Alliance Functional Testing Tools Visioning Workshop, in Portland, in October. I think all of these could fall into that vision. If I could only cover the travel expense and lost income, i'd be there. As it is, I look forward to see what falls out.
If you can make it, I'd encourage any regular Creative Chaos readers to attend. In other words, if you like this blog, I think you'd like the workshop, and, I would hope, vice versa ...
Showing posts with label Tools. Show all posts
Showing posts with label Tools. Show all posts
Friday, August 03, 2007
Saturday, July 14, 2007
Test Automation - IV
Reads of Creation Chaos has left some amazing comments on the previous post; if you haven't read them, please take a gander.
First off, I agree with Shrini that "regression" has too many definitions, and we get confused by it's use. I think that most of the time, today, when people say "regression tests", they mean what Shrini calls type II regression - "Make sure stuff that works yesterday still works."
Yet, sadly, even with automated regression tests hooked up to a CI server, you still don't get that! All you get is "The tests that passed yesterday still passed today." Or, as I prefer to say it "Automated tests give you *some confidence* that no *big* defects were injected in the code since the last build."
For the most past, I've been writing about scripted test automation. For example, If we had a GetNextDate() function, we could pick two dozen different dates and run them again and again. Of course, if something breaks on a date that is not one of those twenty-four, the automated tests won't fix it.
That's where model driven tests can help. For example, instead of twenty-four pre-recorded tests, the software could pick a random date between 100BC and 2500AD, then call GetNextDate(), then pop up Microsoft Excel and ask for the date plus one - then compare the results. This can work as long as you have something like excel to trust. (Cem Kaner calls this "High Volume Test Automation.")
Another way to do it is to have a separate programmer write his or her own GetNextDate(), then pick random numbers and compare them. A few challenges with this -
1) In this case, you're literally coding it twice. It will literally take twice as long to develop using this approach.
2) If the two developers make the same mistake (which is likely - think about leap years) the two programs will work "just fine."
3) If the requirements are vague or wrong (how often does that happen?) the software could do what the developers expect but not what the customers want.
So here's my conclusions ...
A) Automate unit tests for simple bounds and equivalence classes
B) If you produce a single output, then a simple automation regression test is possible. ("If yesterday's output the same as today’s?") This will enable refactoring and diffing the two can show new functionality.
C) Documented acceptance tests prevent the "gee, I didn't mean that, I meant this"
phenomenon and get the customers involved as part of the team. Automating those can help with communication and be a formal specification - but it might not add a lot of value in terms of finding bugs.
D) Model-Driven tests have a lot of promise for finding those quirky odd bugs, especially in the GUI. But ...
E) When it comes to the "This just doesn't look right" kind of bugs, you'll probably want exploratory testing.
But that's just me talkin'. What do you think?
First off, I agree with Shrini that "regression" has too many definitions, and we get confused by it's use. I think that most of the time, today, when people say "regression tests", they mean what Shrini calls type II regression - "Make sure stuff that works yesterday still works."
Yet, sadly, even with automated regression tests hooked up to a CI server, you still don't get that! All you get is "The tests that passed yesterday still passed today." Or, as I prefer to say it "Automated tests give you *some confidence* that no *big* defects were injected in the code since the last build."
For the most past, I've been writing about scripted test automation. For example, If we had a GetNextDate() function, we could pick two dozen different dates and run them again and again. Of course, if something breaks on a date that is not one of those twenty-four, the automated tests won't fix it.
That's where model driven tests can help. For example, instead of twenty-four pre-recorded tests, the software could pick a random date between 100BC and 2500AD, then call GetNextDate(), then pop up Microsoft Excel and ask for the date plus one - then compare the results. This can work as long as you have something like excel to trust. (Cem Kaner calls this "High Volume Test Automation.")
Another way to do it is to have a separate programmer write his or her own GetNextDate(), then pick random numbers and compare them. A few challenges with this -
1) In this case, you're literally coding it twice. It will literally take twice as long to develop using this approach.
2) If the two developers make the same mistake (which is likely - think about leap years) the two programs will work "just fine."
3) If the requirements are vague or wrong (how often does that happen?) the software could do what the developers expect but not what the customers want.
So here's my conclusions ...
A) Automate unit tests for simple bounds and equivalence classes
B) If you produce a single output, then a simple automation regression test is possible. ("If yesterday's output the same as today’s?") This will enable refactoring and diffing the two can show new functionality.
C) Documented acceptance tests prevent the "gee, I didn't mean that, I meant this"
phenomenon and get the customers involved as part of the team. Automating those can help with communication and be a formal specification - but it might not add a lot of value in terms of finding bugs.
D) Model-Driven tests have a lot of promise for finding those quirky odd bugs, especially in the GUI. But ...
E) When it comes to the "This just doesn't look right" kind of bugs, you'll probably want exploratory testing.
But that's just me talkin'. What do you think?
Thursday, July 12, 2007
Test Automation - III
Charlie Audritsh asked:
"I take you to mean what I'd refer to as a regression test. A test of mostly the old functionality that maybe did not change much.
So yeah. I have to admit there's a low likelihood of finding bugs with this. What nags at me though about this idea is that I still feel like regression tests are pretty important. We want them *not* to find bugs after all. We want to know we did not break anything inadvertently, indirectly, by what we did change."
I believe our emphasis on regression testing is mostly historical accident, and here's why:
Big Projects (and LANs) became popular about ten years before version control became popular. So, at the beginning of my career, when dinosaurs still roamed the earth, while I was working on a bug fix, another developer might be working on new functionality. If I saved my fix before she saves her changes, then she would "step" on my changes and they would be lost.
Thus the software would "regress" - fall back to an earlier state where a bug was re-injected into the code. The longer you delayed integration, the more likely that was to happen.
Today we have version control with automatic merge, automated unit tests, and Continuous Integration Servers. So this huge tar pit of regression just isn't as bad as it used to be. In fact, I distinctly remember reading a paper that showed that about five percent of the bugs introduced in the wild today are "true" regression bugs - bugs re-introduced by a mistake.
Of course, there's the second type of regression, which is to make sure that everything that worked yesterday works today. I'm all for using every tool at our disposal to ensure that, but I find that automated customer acceptance tests (FITnesse with fixtures) are very expensive in terms of set-up time, yet don't offer much value. Sure, the customer can walk by at any time and press a button and see a green light. Cool. But in terms of finding and fixing bugs?
If there is never enough time to do all the testing we would like to on projects, then I believe we are obligated to do the testing that has the most value for the effort involved.
And when it comes to bugs, I believe this is critical, investigative work done by a human. Assuming the devs have good unit tests in place, re-running tests from last week (or creating a framework so you can) probably has a lot less value than critical investigation right now, in the moment.
... but don't get me wrong. For example, for performance testing, you need to use a tool, and type some stuff in, and then evaluate the results. I'm saying that writing code on top of that, to have a single button, then evaluate the results for you and green or redbar - might not be the best use of your time.
I am very much in favor of model driven testing, but with every release, the model needs to change to test the new functionality.
"Set it and Forget it" customer acceptance testing?
Not So Much ...
"I take you to mean what I'd refer to as a regression test. A test of mostly the old functionality that maybe did not change much.
So yeah. I have to admit there's a low likelihood of finding bugs with this. What nags at me though about this idea is that I still feel like regression tests are pretty important. We want them *not* to find bugs after all. We want to know we did not break anything inadvertently, indirectly, by what we did change."
I believe our emphasis on regression testing is mostly historical accident, and here's why:
Big Projects (and LANs) became popular about ten years before version control became popular. So, at the beginning of my career, when dinosaurs still roamed the earth, while I was working on a bug fix, another developer might be working on new functionality. If I saved my fix before she saves her changes, then she would "step" on my changes and they would be lost.
Thus the software would "regress" - fall back to an earlier state where a bug was re-injected into the code. The longer you delayed integration, the more likely that was to happen.
Today we have version control with automatic merge, automated unit tests, and Continuous Integration Servers. So this huge tar pit of regression just isn't as bad as it used to be. In fact, I distinctly remember reading a paper that showed that about five percent of the bugs introduced in the wild today are "true" regression bugs - bugs re-introduced by a mistake.
Of course, there's the second type of regression, which is to make sure that everything that worked yesterday works today. I'm all for using every tool at our disposal to ensure that, but I find that automated customer acceptance tests (FITnesse with fixtures) are very expensive in terms of set-up time, yet don't offer much value. Sure, the customer can walk by at any time and press a button and see a green light. Cool. But in terms of finding and fixing bugs?
If there is never enough time to do all the testing we would like to on projects, then I believe we are obligated to do the testing that has the most value for the effort involved.
And when it comes to bugs, I believe this is critical, investigative work done by a human. Assuming the devs have good unit tests in place, re-running tests from last week (or creating a framework so you can) probably has a lot less value than critical investigation right now, in the moment.
... but don't get me wrong. For example, for performance testing, you need to use a tool, and type some stuff in, and then evaluate the results. I'm saying that writing code on top of that, to have a single button, then evaluate the results for you and green or redbar - might not be the best use of your time.
I am very much in favor of model driven testing, but with every release, the model needs to change to test the new functionality.
"Set it and Forget it" customer acceptance testing?
Not So Much ...
Tuesday, July 10, 2007
Test Automation - II
I got some great comments yesterday - Charlie and Scott made some solid points, and they are points that I will address. However, before I get there, I would like to fill in a bit more of the back story from the Software-Testing List.
Here's my reply, the next day, after a small challenge by a guy named Mike Tierney:
Mike Tierney wrote:
"I would tend to agree with everything you have written or quoted Matt. But what about automated tests that are benefits for the tester ? I am not working in a TDD or extreme programming shop ... "
Two Replies:
1) I have seen test automation work well - heck, I write perl scripts that do model driven testing on a daily basis.
When I have seen system test automation work well, the tester started out with the context of the problem and built automation on top of that. Brett Pettichord's "Homebrew Test Automation", for example, struck quite a chord(*) with me. (You'll notice, for example, that Ward Cunningham's story follows that example.)
When it comes to context-free advice for a specific tool "Just use MacOSRunner and you'll be fine", I'm much more leery.
2) I like the Michael Schwern terminology for testing - I break tests into "Those the customers care about" (FIT, fitnesse, excel spreadsheets, etc.) and "Those the developers care about."
Many shops (including mine) have very technical testers, who do system testing that is more complex than what the customers care to track. They'll care enough to understand the work, maybe, but not to define or do it. I suspect that the work Mike Tierney does, if I looked at it scrupulously, would look like "Developer Tests", even though they are behavior (black-box) oriented.
But that's just me talking.
--heusser
(*) - Must ... Avoid ... Silly ... Puns ...
Here's my reply, the next day, after a small challenge by a guy named Mike Tierney:
Mike Tierney wrote:
"I would tend to agree with everything you have written or quoted Matt. But what about automated tests that are benefits for the tester ? I am not working in a TDD or extreme programming shop ... "
Two Replies:
1) I have seen test automation work well - heck, I write perl scripts that do model driven testing on a daily basis.
When I have seen system test automation work well, the tester started out with the context of the problem and built automation on top of that. Brett Pettichord's "Homebrew Test Automation", for example, struck quite a chord(*) with me. (You'll notice, for example, that Ward Cunningham's story follows that example.)
When it comes to context-free advice for a specific tool "Just use MacOSRunner and you'll be fine", I'm much more leery.
2) I like the Michael Schwern terminology for testing - I break tests into "Those the customers care about" (FIT, fitnesse, excel spreadsheets, etc.) and "Those the developers care about."
Many shops (including mine) have very technical testers, who do system testing that is more complex than what the customers care to track. They'll care enough to understand the work, maybe, but not to define or do it. I suspect that the work Mike Tierney does, if I looked at it scrupulously, would look like "Developer Tests", even though they are behavior (black-box) oriented.
But that's just me talking.
--heusser
(*) - Must ... Avoid ... Silly ... Puns ...
Test Automation - I
(Taken From a recent post fo the software-testing email list)
It always amazes me when strong people come out and say publicly something that I have been mulling on for a few months.
James Bach's comments on sapient processes do that for me. Let me give you the back story ...
I do a lot of developer-facing test automation. That's things like asserting the return values of a F-to-C temperture conversion function.
This has a lot of value to developers. It provides a functional spec to the maintenance programmer. It allows us (I am a tester/dev) to change the internals of the code and have some confidence that we didn't break anything. It allows the developer to experience the API and change it to be easier to use.
Just about all of those are benefits to the developer - not the end customer. For the most part, they are design, maintenance, and software engineering benefits. Increasingly, I am wary of test automation that promises to "prove that the software works." I find that on every release, the things that I need check are different - they are the things that have changed. Thus, the old test suite has less and less value as we add new functionality - and providing "hooks" and automation can be very expensive.
I think that Brian Marick's Recent Comments echo this thread. Customer-Facing test automation isn't all that great at, well, finding bugs, or proving that the software works.
Next month, I'll be out in New York, trying to make this (and similar) points at the Google Test Automation Conference. Wish me luck ...
It always amazes me when strong people come out and say publicly something that I have been mulling on for a few months.
James Bach's comments on sapient processes do that for me. Let me give you the back story ...
I do a lot of developer-facing test automation. That's things like asserting the return values of a F-to-C temperture conversion function.
This has a lot of value to developers. It provides a functional spec to the maintenance programmer. It allows us (I am a tester/dev) to change the internals of the code and have some confidence that we didn't break anything. It allows the developer to experience the API and change it to be easier to use.
Just about all of those are benefits to the developer - not the end customer. For the most part, they are design, maintenance, and software engineering benefits. Increasingly, I am wary of test automation that promises to "prove that the software works." I find that on every release, the things that I need check are different - they are the things that have changed. Thus, the old test suite has less and less value as we add new functionality - and providing "hooks" and automation can be very expensive.
I think that Brian Marick's Recent Comments echo this thread. Customer-Facing test automation isn't all that great at, well, finding bugs, or proving that the software works.
Next month, I'll be out in New York, trying to make this (and similar) points at the Google Test Automation Conference. Wish me luck ...
Thursday, December 28, 2006
Interview?
Software Quality Engineering recently sent me a bunch of inteview questions for a StickToolLook e-newsletter. Writing a reply was hard, as I wanted to provide real information that was worth reading, not just pontificate or repeat the same old thing. My replies are below; I'm interested in what you think.
SQE: What are the benefits of a tool developed for an entire market? What about a tool designed to focus on a particular need?
That's a really tricky question, because within every market there is a smaller market. If we start with tools designed for everyone who ever touches software development, we can find tools inside of that. Say, tools for just testers – then just performance testers. Inside of that market are tools for web performance testers, java testers, and database testers. Inside of that we have tools for each specific flavor of database.
Overall, I like tools as niche as a I can get them, because the vendor can solve a specific problem and solve it well. The higher-level tools offer some level of integration and file sharing, and workflow that just can’t exist in a specific tool. Also, the more general a tool, the better chance there will be to find a user group, consultant, or support when things go bad.
SQE: What are the cons to using either type of tool?
Well, for example, if you use a general tool for all databases, calling your sales rep and ask about Oracle support, and he replies "yeah, there’s a driver for that." That’s bad news. At best, you’re going to muck about for a day or three before finally getting the software to mostly work. At worst, it'll go back on the shelf. General tools generally have a main market; like ordering the fish in a steakhouse, you never know quite what you’re gonna get.
I have heard of a few general tools that work in lots of environments; McCabe has amazing support and I've heard good things about Worksoft certify. You just have to be certain to check a vendors claims carefully – yourself.
Of course, if you use the specialized, Database-Specific tool and write custom code for it, then when you convert environments you might have to throw all that work away and start over.
SQE: Is it possible to develop a tool or a tool suite that can be all things to all people? Or even all things to one person?
I think it’s a matter of philosophy. UNIX, for example, is a collection of small tools that can be tied together in really interesting ways to do just about anything; I think the "Swiss Army Chainsaw" is my favorite term to describe it. Is UNIX a single tool or a collection of tools? (Grins) Who Cares?
There are certainly performance testing tools, record/playback tools, and development environments people spend all day in, and some of them have very passionate and active users. My colleague Mike Kelly is fond of the phrase "Role Players", to mean that the person likes to perform a function and do it well. If someone wants to be an excellent performance tester for websites, then sure, that person could live in one tool, align himself with that vendor, and probably make a very good living as an employee or consultant. That just isn't me, and you never know when a specialty is going to be destroyed – say, for example, when websites start to support wireless devices and the bottleneck for performance shifts.
SQE: On the other hand, can a tool be too specific?
Certainly! I don’t know anyone who uses awk or sed anymore, for example. Those were great little programming languages, but perl cam along and offered all the exact same features plus more.
Of course, if the tool is open source, it doesn’t really matter if the tool is too specific, because it's free, and you can put it away and pull it out if and when you have the need again. If it’s for-purchase, the market will sort it out – the vendor will either expand the product or fail in the marketplace. (You have to love capitalism, after all.)
SQE: Which type of tool is more prevalent in the market right now?
I think vendor-supported tools tend to go big and open source tools tend to be small. The open source tools are often developed by a specific person to solve his own problems, and growing the tool won't make him any money. So open-source tools grow through some other guy adding features that he needs. For-purchase tools are driven by companies that want to make money; they have an economic reason to try to reach as many people as possible. I hear a lot of good things about Visual Studio Team System, which is big, big, big general purpose tool. I think the market is still figuring itself out, but right now most of the new tools I see are specific tools for a small market, created by small companies who have expertise in that area – and I think that is very good thing.
SQE: Where do you see the market headed in the future?
I think software testing is an immature field, and software quality even less so. That means we need to try a bunch of different approaches, see what works, and then build up tools around that. Like I mentioned earlier, I see little companies doing just that. If they succeed, economic forces will expand those products a little bit, while hopefully the "big" products will get out of the clouds and come down to the real world a little bit more. Eventually, I predict a fight for the middle.
SQE: What are the benefits of a tool developed for an entire market? What about a tool designed to focus on a particular need?
That's a really tricky question, because within every market there is a smaller market. If we start with tools designed for everyone who ever touches software development, we can find tools inside of that. Say, tools for just testers – then just performance testers. Inside of that market are tools for web performance testers, java testers, and database testers. Inside of that we have tools for each specific flavor of database.
Overall, I like tools as niche as a I can get them, because the vendor can solve a specific problem and solve it well. The higher-level tools offer some level of integration and file sharing, and workflow that just can’t exist in a specific tool. Also, the more general a tool, the better chance there will be to find a user group, consultant, or support when things go bad.
SQE: What are the cons to using either type of tool?
Well, for example, if you use a general tool for all databases, calling your sales rep and ask about Oracle support, and he replies "yeah, there’s a driver for that." That’s bad news. At best, you’re going to muck about for a day or three before finally getting the software to mostly work. At worst, it'll go back on the shelf. General tools generally have a main market; like ordering the fish in a steakhouse, you never know quite what you’re gonna get.
I have heard of a few general tools that work in lots of environments; McCabe has amazing support and I've heard good things about Worksoft certify. You just have to be certain to check a vendors claims carefully – yourself.
Of course, if you use the specialized, Database-Specific tool and write custom code for it, then when you convert environments you might have to throw all that work away and start over.
SQE: Is it possible to develop a tool or a tool suite that can be all things to all people? Or even all things to one person?
I think it’s a matter of philosophy. UNIX, for example, is a collection of small tools that can be tied together in really interesting ways to do just about anything; I think the "Swiss Army Chainsaw" is my favorite term to describe it. Is UNIX a single tool or a collection of tools? (Grins) Who Cares?
There are certainly performance testing tools, record/playback tools, and development environments people spend all day in, and some of them have very passionate and active users. My colleague Mike Kelly is fond of the phrase "Role Players", to mean that the person likes to perform a function and do it well. If someone wants to be an excellent performance tester for websites, then sure, that person could live in one tool, align himself with that vendor, and probably make a very good living as an employee or consultant. That just isn't me, and you never know when a specialty is going to be destroyed – say, for example, when websites start to support wireless devices and the bottleneck for performance shifts.
SQE: On the other hand, can a tool be too specific?
Certainly! I don’t know anyone who uses awk or sed anymore, for example. Those were great little programming languages, but perl cam along and offered all the exact same features plus more.
Of course, if the tool is open source, it doesn’t really matter if the tool is too specific, because it's free, and you can put it away and pull it out if and when you have the need again. If it’s for-purchase, the market will sort it out – the vendor will either expand the product or fail in the marketplace. (You have to love capitalism, after all.)
SQE: Which type of tool is more prevalent in the market right now?
I think vendor-supported tools tend to go big and open source tools tend to be small. The open source tools are often developed by a specific person to solve his own problems, and growing the tool won't make him any money. So open-source tools grow through some other guy adding features that he needs. For-purchase tools are driven by companies that want to make money; they have an economic reason to try to reach as many people as possible. I hear a lot of good things about Visual Studio Team System, which is big, big, big general purpose tool. I think the market is still figuring itself out, but right now most of the new tools I see are specific tools for a small market, created by small companies who have expertise in that area – and I think that is very good thing.
SQE: Where do you see the market headed in the future?
I think software testing is an immature field, and software quality even less so. That means we need to try a bunch of different approaches, see what works, and then build up tools around that. Like I mentioned earlier, I see little companies doing just that. If they succeed, economic forces will expand those products a little bit, while hopefully the "big" products will get out of the clouds and come down to the real world a little bit more. Eventually, I predict a fight for the middle.
Subscribe to:
Posts (Atom)
