Functional Testing Tools.

This week, I've spent a fair amount of time reading about functional test tools. Both for automated testing, and for manual testing, for two different projects.

I don't know what it is about tools for testers, but they feel terribly prescriptive and constraining. We tried Seapine's TCM tool a couple of years ago. The idea was to replace our slightly flaky Excel based scripts. It was chosen because we use Seapine's defect management tool, Test Track Pro, and it made for an integrated solution.

Test Track Pro is fine. I don't think we get much out of it that couldn't be achieved with Bugzilla or a hundred other defect management tools. It's not perfect, it's not simple enough for my tastes, but it's reliable and does what it needs to do.

What happened was that I spent a couple of months writing macros to convert our Excel scripts into a form that Seapine could handle, and using their SOAP/WSDL API to write a Java utility to inject the scripts as new test cases. I also spent some time writing another utility to generate a project status summary using the fundamental metric we measure against, which is number of test steps. It's not a particularly good metric, with tens of thousands of steps, we have a good feeling for how long scripts take to execute.

We also tend to write very long scripts, some with many hundreds of steps. It's probably bad practice, but it works for us, and historically, with no TCM, it was easier to manage a few long scripts rather than many short ones. I've written some fairly extensive VBA macros to make the scripts manageable. It used to be a manual task to clear results prior to a new test run, for example, but that's now one click away. Other macros allow steps to be included or excluded from a test run quickly based on context (e.g. enable different steps for hardware keyboard and touchscreen, or disable steps for deprecated devices). These macros were written, fundamentally, to save us time. They're written by the testers for the testers.

Trying to use a COTS TCM solution after being used to this kind of freedom was a nightmare. A few of the testers never even got started. I was quite enthusiastic, but other than a trial project, to prove the feasibility, it never got adopted.

Developers don't put up with these kinds of tools, and neither should testers. I think a lot of the test tools are designed by process nutjobs. Managers writing requirements for they way they think things should work and implemented blindly by developers, but the users, the real world testers, don't get enough say. It's a shame.

On the automated test side, things look a little better. I think it's because there's more developer influence. A lot of developers will want to use the tools themselves and their ideas get incorporated. In terms of web testing, there's plenty to choose from. Selenium and Watir seem to be top. Selenium's IDE for recording scripts is nice and simple, and the ability to convert these to Java or Ruby or Perl of Python means reusable modular tests are nice and easy to write. The problem I have is that I need to test Java. Specifically in-browser applets. Selenium can apparently do that through FEST, but on reading the instructions, I really can't be arsed, I'm not convinced that I'd actually get it to work at all.

The two front runners in this contest are HP's Quick Test Pro and Froglogic's Squish. Squish would let me use Python, which is a massive win, it's Eclipse based and the object lookup appears to be astonishingly good (from the demos). Froglogic have loads of information available on their website, including the user guide, so I can really find out about how it works. HP's on the other hand seems impenetrable. Quick Test Pro really is the industry standard, it'll look great on the CV, but it'll mean a VB-like language, which is a real shame. Head says QTP. Heart says Squish.