I'm excited to to have the honor of once again presenting at the CF.Objective() Enterprise ColdFusion Conference. This year I'll be talking about Automated System Testing for Web Applications with CFSelenium, MXUnit, and Jenkins.

I've been a Quality Assurance software developer since 2007 when I was on the ColdFusion server engineering team at Adobe. For the past couple years I've enjoyed working at FirstComp Insurance with one of the largest ColdFusion developer teams that I know of, including well known team members like Sam Farmer, Dan Vega, and Jason Delmore, as well as many others of ColdFusion's best.

Testing by Isolation
One of my goals last year was to create a test suite framework that could perform Automated System Testing of our collection of web applications that we use for our business. We run it all on ColdFusion with a truly massive code base, and we have many different web applications that drive different parts of the business, each with unique user interfaces (UI). Part of good development practices includes writing Unit Tests early in the project to test application modules (CFCs) in isolation. Unit Tests are great for catching issues early in the release cycle, but they don't test how all the parts work together across the whole application as a system.

Testing Across the Board
This is where System Testing (or UI Testing) comes in, and I'll be showing you how I built our automated UI test framework from the ground up.

Unit Testing has a kind of bottom-up approach, whereas System Testing's approach is top-down. This means testing the applications by mimicking a real user interacting with the web applications. Logging in, clicking on links, submitting forms, working with popup windows, and the like. Using tools like Selenium, the user's actions can be recorded, and using the CFSelenium plugin, tests can be recorded as CFML format appropriate for playback in the MXUnit Test Suite. System Testing tests how all the parts work together as a completely integrated application.

Static means Brittle
However, the tests recorded that way have a very static nature, recording literal paths, and literal form data, and identifying HTML elements with object locators right there in the middle of the page. If you need to test other login names, you need to copy the test and change the login name. But that creates a lot of unwanted redundancy. If a HTML Element like a form tag changes it location in the page or changes something like its ID value, then the locator used in all your tests to find that form will break. This produces brittle tests that cause much of the test suite to fail with even small changes in the UI.

By adapting your tests to fit the Page Object Model (POM) design pattern, you avoid creating brittle tests by modeling pages or page elements with CFCs. Then all your test cases refer to the CFC-based page model representing the page, not the literal page object locator. This way if your web app's UI changes, all the tests that depend on locating the changed HTML element can be quickly fixed by modifying the page object that models the element or widget that changed. POM gives your tests "robustitude" (to quote Dante Briones) and creates more flexible, maintainable code.

Parameterized Input Data
Combine POM with MXUnit's "dataprovider" feature, and you now feed in native ColdFusion data structures like a query or array to your test case methods, and your test case will iterate once for every row or record in the data structure. Need to test 10 logins? Then maybe pass in a query object with fields like username, password, fname, lname, etc to your test case. Parameterized input data greatly reduces test code duplication.

Jenkins will serve you now
MXUnit can run tests via an Ant Task and a plugin for CF Builder (or eclipse). After creating a build file for your MXUnit Test Suites, start running the test suite executions around the clock using using a Continuous Integration solution like JenkinsCI. Schedule a daily refresh job in Jenkins to grab all the latest testcases from your source control and have it restart CF and Selenium. Then schedule another Jenkins job to start shortly after that to perform the actual execution of data driven, POM-based test suites.

Other AUT Technologies
If you're a ColdFusion consultant, you can expand your reach into companies that use other web application technologies. Even if the Application Under Test (AUT) is built in HTML5, JSP, PHP, or .NET, if those companies are looking for a robust, inexpensive solution to create a dynamic test framework around their AUTs then you can use ColdFusion to develop their test suite even though their main production apps use other technologies. Given that the only client making requests to your ColdFusion server is the JenkinsCI server (and you over localhost), in concept you could use just the free ColdFusion Developer Edition for to run the test suites. If the company already has ColdFusion licenses, then that license lets you run the full CF version for free in a test environment without any additional licenses.

Thinking about attending my session this Friday at 9am at CF.Objective()? Be sure to check out the Introduction to Browser Testing seminar on Thursday afternoon by Bob Silverberg, which would make a great prerequisite. I hope you can make it!

Automated System Testing by Steven Erat at CFObjective 2012

Test Suite Framework vs Applications Under Test

Using the Page Object Model

Using External Data Providers

The JenkinsCI Dashboard