Automated System Testing for Web Apps at CF.Objective

I'm excited to to have the honor of once again presenting at the CF.Objective() Enterprise ColdFusion Conference. This year I'll be talking about Automated System Testing for Web Applications with CFSelenium, MXUnit, and Jenkins.

I've been a Quality Assurance software developer since 2007 when I was on the ColdFusion server engineering team at Adobe. For the past couple years I've enjoyed working at FirstComp Insurance with one of the largest ColdFusion developer teams that I know of, including well known team members like Sam Farmer, Dan Vega, and Jason Delmore, as well as many others of ColdFusion's best.

Testing by Isolation
One of my goals last year was to create a test suite framework that could perform Automated System Testing of our collection of web applications that we use for our business. We run it all on ColdFusion with a truly massive code base, and we have many different web applications that drive different parts of the business, each with unique user interfaces (UI). Part of good development practices includes writing Unit Tests early in the project to test application modules (CFCs) in isolation. Unit Tests are great for catching issues early in the release cycle, but they don't test how all the parts work together across the whole application as a system.

Testing Across the Board
This is where System Testing (or UI Testing) comes in, and I'll be showing you how I built our automated UI test framework from the ground up.


Could not find ColdFusion component or interface Query

My best blogging years were when I worked in ColdFusion Technical Support, from Allaire and right on thru Macromedia then Adobe. Constantly fielding customer questions provided an endless source of fodder to investigate and blog about when a solution or workaround was found. It feels a little like old times again now that my QA team is expanding and I've been helping others come up to speed with our ColdFusion driven Automated Test Suite. Although my colleagues are experienced web professionals, I'm happy there is room for mentoring in ColdFusion, and that provides me with more fodder to share here.

After helping someone install ColdFusion 9.0 and apply the 9.01 updater, they reported the updater failed to complete. We cleaned things up a bit, confirmed installers, and tried again. Success. Shortly after, we continued setting up the test suite environment they reported a very unusual error that I'd never seen before, Could not find the ColdFusion component or interface Query. With a bit of Googling, I found that there were only 2 hits, and one was in a comment on Ben Nadel's blog where he provided the winning hint. The other hit was a tweet about it when someone else encountered this issue.

Per Ben's hint, I had my colleague check the CF Admin's Custom Tag mappings, and the source of the problem was immediately evident. The core mapping for "C:ColdFusion9CustomTags" was missing. Prior to then, I thought this mapping was immutable by the end user of the CF Admin. Perhaps it was due to the initial failed 9.01 updater, I'm not really sure how that mapping got wiped out, but as soon as we restored it, everything worked.

The mapping is needed because some parts of the Core CFML language are implemented as custom tags stored in that core location. This includes the query.cfc tag, which implements the script-based version of CFQuery. Without that mapping, there will be several language areas that won't work.


Multiserver Monitor: Permission Denied and crossdomain.xml

While helping out with this issue on the Adobe Forums, I learned that the ColdFusion 9 Multiserver Monitor now requires /crossdomain.xml on target servers rather than /CFIDE/multiservermonitor-access-policy.xml. I was not aware of this change, so hopefully this post will ensure that others who administer ColdFusion will be.


Since I was actually on the ColdFusion 8.0 engineering team at Adobe and personally tested the multiservermonitor back in 2006/7, I find it very surprising to learn that /crossdomain.xml is now required in the webroot INSTEAD of /CFIDE/multiservermonitor-access-policy.xml.

I did some testing on a couple local ColdFusion 9.01 servers, and to force the requirement of the access file, I loaded the CF Admin Multiserver Monitor over localhost ( and then tried to add a different CF instance to the monitor using the other interface for the same machine As expected, I got Permission Denied. I then went to the target server that I was trying to add, and I enabled the multiservermonitor-access-policy.xml by uncommenting the appropriate line. I was really stunned to find that the target server still showed a Permission Denied status (Figure 1).


Validation Query for MySQL communications link failure

"That's probably the best answer to any forum question I've ever posted and/or seen" is what someone said in response to my reply on the Adobe ColdFusion forums.

I always enjoy helping other with ColdFusion questions whenever I have time (a 2 year old child doesn't leave much room for extra time!), but reactions like this really make it feel that the help is worth my effort. That's what being an Adobe Community Professional is all about. For what its worth, here's what I had to say about using Validation Queries to eliminate surprises when using database connection pooling in ColdFusion:

The Maintain Connections setting means that after a db connection is created for a given database, that connection will be used for the current query and then kept open in a connection pool so that they can be reused for later queries. The reason is that opening a connection is an expensive, time consuming operation, and its more efficient to only have to authenticate once. When this setting is disabled, for every page request accessing a given database, a new database connection will be created, the db authentication will occur, the query/queries on the request will happen, then the db connection will be closed.

When you maintain connections you have a pool of db connections that exist for an extended period, being frequently reused with additional requests. If the connections are idle for a period greater than the Inactive Timeout setting in the datasource definition, then those connections are closed and the pool size is reduced. Also, if a request checks out a connection from the pool, attempts to use it for a query for some request, then if that db connection produces a db error then that is another situation where the db connection will be closed and removed from the pool.

It has been known to happen that when you are pooling datasource connections like this that its possible that the TCP connection to the database has been interrupted for some reason, and when the connection is checked out for use on a page request you will get some type of "communication" error. The actual error message will vary depending on the database.

If unchecking Maintain Connections resolves this MySQL Communication Link Failure issue for you, then you are better off re-checking it AND adding a validation query. ColdFusion 8.0 introduced a field in the dsn definition for Validation Query. It works this way: When a database connection is first created AND every subsequent time that connection is checked back out from the pool, the validation query will run BEFORE any queries for the page request. If the validation query fails, your page request will never see the error because ColdFusion will throw away that db connection and get another connection from the db connection pool. It will then run the validation query for that connection too. If that one errors, ColdFusion will continue closing the bad connections and checking out other connections until there are no connections left in the connection pool. If it actually got that far (meaning every connection in the pool turned out to be bad) then ColdFusion will then create a NEW db connection and use that one, and it will run the validation query on that too. All of this happens before your request runs to guarantee that your request gets a *good* db connection from the start.

A good validation query is something is that is highly efficient so that db isn't really taxed by having to run it. For MySQL you could use: Select 1 That's it. Enter that into the validation query field for the datasource and keep Maintain Connections checked to improve efficiency with connection pooling.

ColdFusion 9.01 Server Monitoring Enhancements

I began this thought as a comment to Adobe CF QA engineer Sagar Ganatra's blog entry describing the new Server Monitor enhancements in ColdFusion 9.01 updater, however, as it grew lengthy I decided my own blog post would be a more appropriate venue.

I'll add that the main reason for why one would want to run the ColdFusion Server Monitor on its own port via the Jetty implementation is that until now requests from the Server Monitor would go through the JRPP request pool, thereby adding additional traffic to the JRun active request pool, but more importantly if the JRun active request pool was queuing then the data refreshes in the Server Monitor would also queue and the Server Monitor may appear to hang as well. By establishing a separate request pool and port for Server Monitor requests in ColdFusion 9.01, the Server Monitor will not encounter a blocking situation as it would do previously.

Any general discussion of the Server Monitor should include the caveat that the use of Profiling, Monitoring, and Memory Tracking are not intended for production use (blanket statement: see comments for more on that). Moreover, if Memory Tracking is enabled in production, perhaps to help diagnose a prod performance problem, that it will only further decrease server performance. On 3 occasions in the last year alone I've helped ColdFusion shops that shot themselves in the foot by doing this. The impact to performance was substantially worse when enabled, and having them disable it immediately alleviated most of the problems, albeit not the problem that initially prompted them to enable the tracking.

Nice enhancements to Server Monitor in the future might include:
  • Persistant Metrics: The ability to persist the Server Monitor data to a database. A restart of CF will clear data as of now. A use case would be for load testing scenarios where the a a variety of metrics need to be quantitatively analyzed, which cannot presently done easily. Ideally, you'd want to know performance metrics at different points into a load test such as during the ramp up, after X minute intervals, and during cool down.

    A second use case would be the ability to produce reports to monitor server health over time, perhaps by providing the ability to generate weekly reports of key data, possibly with green and red arrow indicator to visually identify metrics that have improved or worsened.

  • CF Request Pool Ratios: Add the ability to analyze incoming requests to determine the real time ratio of CF request types such as CF Templates, CFCs, Flash Remoting, and Web Services. When expressed as a percentage, it could used to correctly determine the values to use in the Request Tuning part of the ColdFusion Administrator. Throughout much of my experience I have found that CF shops rarely set the Request Tuning values to an appropriate range, either letting them remain unchanged at the default, or increasing them way out of range (into the hundreds even).

  • Request Tuning Calculator: Provide a real suggested starting point for all Request tuning parameters including the JRun active and queued sizes based on the number of CPUs/cores and processing speeds. Presently, even in CF 901, a server will install with a default set of values that will the same on a small box as it would on a beefy production box. To do this correctly, the total number of instances used to process production load would also have to be incorporated for proper per-instance tuning. Having a Server Monitor Request Tuning Calculator would be a big plus towards helping server admins find the right starting range for their particular hardware.

  • Out of Process Memory Tracking: Since Memory Tracking is known to consume significant resources while tracking (a Heisenberg conundrum?), perhaps Memory Tracking could be done out of process over RMI or similar, akin to the JRockit Mission Control memory analyzer (which I've used for ColdFusion, but interpreting the data is not very intuitive).

Issue with stopping ColdFusion after starting from Builder

With the release of ColdFusion Builder there is an option available that provides the ability to start and stop one or more ColdFusion servers from ColdFusion Builder. In fact, you can configure CF Builder to automatically start a CF server when Builder opens, and stop the CF server when Builder closes. The autostart/autostop is convenient for a Development box where you want to minimize resource usage on the system. You can read more about this feature here.

Server Panel in ColdFusion Builder
Server Panel in ColdFusion Builder

Server Settings Panel in ColdFusion Builder showing Auto start/stop for CF Servers
Server Settings Panel in ColdFusion Builder showing Auto start/stop for CF Server

However, if you don't enable the automatic stop/start option, if you ever start ColdFusion server from Builder then close Builder without stopping ColdFusion there, then later you will not be able to stop the ColdFusion server using the standard ColdFusion stop script. I've encountered this on Mac OS X, but since its possible to configure CF Builder to start/stop remote CF servers, it's likely that the problem might occur when using ColdFusion server on Linux or Solaris as well, even though Builder doesn't run on those platforms.

Normally, to stop / start the ColdFusion server from the command line, you would the control script located (typically) at /opt/coldfusion9/bin/coldfusion, such as with ./coldfusion stop. That control script in turn invokes /opt/coldfusion9/runtime/bin/coldfusion9. When calling stop, the control script works by first grepping for any running ColdFusion processes with fgrep, like this: ps -axc | fgrep coldfusion9. If it finds a process listing that matches for the string "coldfusion9" then it stops that process.

Here's what you might see if you try to restart ColdFusion from the command line after it was started but not stopped from Builder:

view plain print about
1$ /opt/ColdFusion9/bin/coldfusion restart
2Restarting ColdFusion 9...
3ColdFusion 9 does not seem to be currently running
4Starting ColdFusion 9...
5The ColdFusion 9 server is starting up and will be available shortly.
6There has been an error starting ColdFusion 9, please check the logs.

The problem of not being able to use that control script to stop ColdFusion server after having started it from Builder arises because of how Builder starts the CF server. Rather than invoking /opt/coldfusion9/runtime/bin/coldfusion9, Builder instead invokes /opt/coldfusion9/runtime/bin/jrun. When the control script tries to grep for the process with a "coldfusion9" string, the control script doesn't find it because Builder invoked runtime/jrun instead of runtime/coldfusion9.

Why the need for runtime/jrun AND runtime/coldfusion9? I have no idea, especially since the files are identical and not symlinked.

view plain print about
1$ pwd
3$ diff jrun coldfusion9

I logged ColdFusion server bug 82573 for this where I proposed a change to the bin/coldfusion control script. My suggested change was only shown for Mac OS X, but you can easily change it yourself for the Linux or Solaris blocks in a similar way.

If you want to use my suggested fix on your local Mac OS X dev box, then you can refer to the full example control script containing the fix here:

For brevity, I won't show the whole script in this blog entry. Instead, here's the diff between the backed up original coldfusion control script which I renamed to 'orig.coldfusion' compared to the fixed version 'coldfusion'.

view plain print about
11.    $ diff orig.coldfusion coldfusion
2    2.    13a14
3    3.    >
4    4.    34c35
5    5.    < ⬠ ⬠ ⬠ ⬠ $PSCMD | fgrep coldfusion9 ⬠> /dev/null 2>&1
6    6.    ---
7    7.    > ⬠ ⬠ ⬠ ⬠ $PSCMD | grep -i $JRUN_BIN | grep -v 'grep' > /dev/null 2>&1
8    8.    117c118,119
9    9.    < ⬠ ⬠ ⬠ ⬠ ⬠ ⬠ $PSCMD | fgrep coldfusion9 | awk '{print $1}' | xargs kill -9 > /dev/null 2>&1
10    10.    ---
11    11.    > ⬠ ⬠ ⬠ ⬠ ⬠ $PSCMD | grep -i $JRUN_BIN ⬠| grep -v 'grep' | awk '{print $1}' | xargs kill -9 > /dev/null 2>&1
12    12.    > ⬠ ⬠ ⬠ ⬠ ⬠
13    13.    130,131c132,133
14    14.    < ⬠ ⬠ ⬠ ⬠ ⬠ ⬠ ⬠ $PSCMD | fgrep coldfusion9 | awk '{print $1}'
15    15.    < ⬠ ⬠ ⬠ ⬠ fi
16    16.    ---
17    17.    >
⬠ ⬠ ⬠ ⬠ ⬠ ⬠ ⬠ ⬠ $PSCMD | grep -i $JRUN_BIN ⬠| grep -v 'grep' | awk '{print $1}'
18    18.    >
⬠ ⬠ ⬠ ⬠ ⬠ ⬠ ⬠ ⬠ ⬠ ⬠ ⬠ fi
19    19.    152c154
20    20.    < ⬠ ⬠ ⬠ ⬠ ⬠ ⬠ ⬠ PSCMD="ps -axc"
21    21.    ---
22    22.    >
⬠ ⬠ ⬠ ⬠ ⬠ ⬠ ⬠ PSCMD="ps -ef"

Adobe LiveCycle DataServices for ColdFusion at CFObjective

Allaire's CEO, David OrfaoAfter a decade of working intensely with the ColdFusion server, I'm finally getting the courage to start presenting about it on the conference circuit. As a blogger, tweeter, and contributor to mailing lists I'm very confident helping others solve ColdFusion related problems because I can do that from the quiet comfort of my own desk. However, one of my greatest fears has always been public speaking. I'm the kind of person that feels like I need to know the subject matter cold, so that I can speak from the hip without relying on looking at the slides.

Blackstone Test CDsOver the years, I had some opportunities to present to small groups, and I recall each time feeling the adrenalin surge and my heart pounding. That started with presenting ColdFusion for Unix and Linux as an internal training class at Macromedia. Later, while taking classes at the Harvard Extension School, I was honored to be asked to present to CSCI-253 Developing Web-Based Database Applications. Even more so, I presented twice there in one year. The first time on Building ColdFusion Web Applications with CFEclipse and Dreamweaver, and later on ColdFusion Server Administration

MAX in ActionI've been attending ColdFusion conferences since the days of Allaire DevCon, but had never presented at any of them including MAX. My long time friend in the local ColdFusion Community, Brian Rinaldi, continued to encourage me to present at the local Boston CFUG as a starting point, as well as the new conference that he was organizing, RIA Unleashed, held in Bentley College this past November. The members of the CFUG were kind enough to let me present a draft of a presentation that I was to later give at RIA Unleashed. My presentation topic was Adobe LiveCycle DataServices Data Management for Mere Mortals

ColdFusion 1.5 on Floppy DisksFortunately at RIA Unleashed I was among the very first sessions after the keynote, so there was no time to build up butterflies that morning. If beforehand you would have told me that among the audience front row would be Ben Nadel, Simon Free, and Ray Camden with Tom Jordahl tucked way in the back then I surely would have freaked out. But they were both kind enough to chat with me before hand and even lend some technical assistance getting setup with the A/V, so that really put me at ease. With a firm limit of 50 minutes, I pushed all the way through what should have been a 90 minute talk, all the while trying to remember to speak clearly and loudly. The talk went off pretty much without a hitch as I found myself completely focused on the technical content and not at all worrying about the large room filled with people in front of me. I was delighted at the end when Tom complemented me on talk, which to me was the ultimate satisfaction.

First Unix machine to run ColdFusionI chose LCDS for ColdFusion as a topic because while I was a QA Engineer on the ColdFusion team at Adobe, I was paired with Tom, a Computer Scientist at Adobe who architected the integration between the products. Heck, Tom architected much of ColdFusion itself, and was in fact the original engineer to have ported ColdFusion to run on Unix and Linux back in the day. Tom is a font of information, and I cut my teeth on the feature under his guidance, which was then known as Flex Data Services and later renamed under the LiveCycle brand. I spent many days last summer and fall revisiting all the LCDS documentation again to ensure the quality of my presentation and to mentally prepare me for the upcoming conference.

ColdFusion Team, BangaloreWIth my first conference under my belt, I decided to throw my hat into the ring for the ultimate ColdFusion experience, CFObjective, which is promoted as The Only Enterprise ColdFusion Conference. I'm excited to announce that I have been selected to be a speaker at the conference, which runs from April 22-24th in Minneapolis, Minnesota. The conference is divided into three tracks for technologies related to ColdFusion. I'll be speaking the last day in the Flex track, once again on the topic of LiveCycle DataServices for ColdFusion Developers. Specifically I'll be talking about the prime feature of LCDS, the Data Management capabilities. With any luck I'll be updating my presentation to consider the benefits of working with the latest versions of Adobe software. Here's the brief description and the PDF:

Discussions of Adobe's LiveCycle Data Services are often entered with the same trepidation as those of Organic Chemistry or Quantum Mechanics, but with ColdFusion, building Web applications that manage complex data sets doesn't have to be that scary. Data Management is a pillar of LCDS that offers scalable, real-time data synchronization across very large numbers of connected clients with the benefits of conflict resolution and data pagination.† Come learn how to quickly get up to speed with Data Management by letting ColdFusion do the hard work for you.

If you're seriously interested in ColdFusion, then CFObjective is the conference for you. I hope to see you there.

ColdFusion Screams

Is Your ColdFusion Support a Real Turkey?

Are you getting half baked help solving complex server problems? Is your global service provider a little off? Sure, maybe its a lot off.

Webapper wants your feedbag feedback about what's important to you. Take this survey to let us know what you really want from ColdFusion support, and the first 100 respondents win a copy of SeeFusion Enterprise, a ColdFusion bottleneck analysis tool with all the trimmings.

Webapper's consulting practice is a one-stop shop for all of your ColdFusion support needs. Our service offerings have been developed and honed over many years, and through hundreds upon hundreds of successful engagements, and they deliver a full spectrum of expertise that covers the entire "Web application stack".

Webapper's consulting services practice is a world leader in ColdFusion support expertiseour engineers are all former employees from the consulting, support and engineering teams at Allaire/Macromedia/Adobe.

Ask a Photographer: Why buy a dSLR?

I'm often asked about my opinion on camera selection, i.e. "What camera should I buy?". That's a really hard question to answer because it all depends on what you want to do with the camera. Today a more narrow question was asked that's a little easier to discuss, and that is: "What are the advantages of a dSLR over a point-and-shoot camera?". Since my reply ran on a bit, I thought it would make a good blog topic, so here you go...

The first advantage that comes to mind for reasons why to get a dSLR is that the lenses are interchangeable. Often, lenses are worth more than the camera body for good reason, but sometimes you can find good inexpensive ones.

There are several factors that make a lens good. The most interesting reason is that the well built ones have a very wide aperture, which is the F number. The lower the number, the wider the aperture, the larger the opening. This is important because wide apertures produce shallow depth of field (DOF). Shallow DOF causes the subject to be in sharp focus but other things in the foreground or in the background bcome blurry and abstract. The lower the number, the more blurry other things become. This makes for a more artistic photograph. Bright items that are blurry typically produce a distinct pattern known as bokeh, and this is considered desirable in much photography.


Recent Tweets for Fri Oct 2, 2009 Part II

Follow me on Twitter!

Tue Sep 15 9:04 PM
@iotashan @rukumar Shan meet Rupesh. Rupesh meet Shan. You guys should talk CF9 ORM. ;-) Rupesh, Shan works with me & has an ORM issue
Tue Sep 15 8:51 PM
No CF Admin DSN setting for isolation level, but u can add SET TRANSACTION ISOLATION LEVEL <level> as u're validation query as workaround
Tue Sep 15 5:36 PM
Tue Sep 15 4:13 PM
@berniedolan Yup, and I was on a downhill at 35mph, slowed to 15 then skidded to within inches as he made a blinkerless right turn


More Entries