Test automation can be a great time-saver, but it requires proper planning, implementation, and maintenance, and it should be treated the same as any other software development project. At Elastic Path, we recently started to automate the testing of the Commerce Manager client. In the past, we used automation in several areas (JUnit for unit testing, FIT for business layer testing, and Web load for load testing), but CM client testing was completely manual. Our goals for automating CM client testing were:

• Reduce time spent on tedious, repetitive testing. After each build, we perform smoke and regression testing. Some of this testing is automated. Our plan was to integrate the automated CM client testing scripts with our autodeploy build process, so that the build server would automatically copy the CM client application and the testing scripts to CM client testing server and launch the testing scripts. By automating CM client testing and integrating it with our other automated test processes, we would be able to run automated tests after work hours and check the results the next morning. This would allow our QA testers to focus on more complex, manual tests.

• Increase result accuracy. Test results can be generated automatically after the automated tests are complete, and if the test scripts were written correctly, the results will always be accurate. For CM client testing, logs and results could be automatically retrieved from the CM server and sent as part of notification email so we could immediately know how many test cases were successful and how many failed.

• Improve defect logging. The sooner a defect is found, the less expensive it is to fix. After logs are generated from the automated tests, they are automatically imported into our test case repository, Aptest. This allows QA team members to quickly create bug report based on failed test cases.

The first thing we needed to do was to select an automated test tool that met our requirements. We used the following criteria:

• Ease of use: is there a steep learning curve? How quickly can we expect to be productive?

• Integration: how well does it integrate with our tools, product, and processes (Java, ODBC, etc.)?

• Platform support: is it platform independent? If not, what specific platform does this tool support?

• Functionality: Does it support test recording and playback?

• Result reporting: What output formats does it support (XML, HTML)?

• Language: What scripting languages does it support? Do we need to learn new scripting language?

• Price: Is it within the company's budget?

• Support and training: How is the response time for support incidents? How is the documentation? Can they provide training if we need it?

We looked at many automated testing tools, including TestComplete, QFTest, and Squish for Java. We eventually selected froglogic's Squish for Java (http://www.froglogic.com/pg?id=Products&category=squish&sub=editions&subsub=java). It is especially well suited for Eclipse RCP applications, so it was a good choice for the CM client. Also, it supports commonly used scripting languages such as JavaScript and Python and we were already using JavaScript as our automation script language in other testing areas.

Next, we needed to decide which test cases to automate. For example, email notification was not a good choice, because it is difficult to send a message and check if it arrived using the same tool.

After that, we needed to prioritize test cases that we wanted to automate. We started with some easy ones, so we could gain confidence as our understanding of the tool increased. The CM client application user interface is very stable and our test cases in Aptest are in very strong form, so we were able to automate many of our test cases quickly and easily. Initially, our focus has been on automating the catalog management related areas of the CM client application. We will continue to increase our automated test coverage in the CM client over time.

CM client test automation is an important project. Initially, it required an important investment of time and resources to evaluate the tools, develop the strategy for automating our manual test cases, and finally to write the automated test scripts. This last step is still ongoing, but we have already seen a significant return on our investment in terms of better integration with existing automated tests and improved test result accuracy. Ultimately, this helps us to improve the overall quality of our product.