ColdFusion Memory Tracking: Real World Performance Example

It is widely known that the built-in ColdFusion Server Monitor can in many cases cause a CF server to become entirely unresponsive if Memory Tracking is enabled. I've experienced this myself when I previously consulted with customers, and I was able to save many clients that engaged me to resolve performance problems by identifying that they had inadvertently enabled Memory Tracking in production. I've written about this before, as have others.

A Test
However, as I am currently working on a Performance Testing project for an enormously large web application, I took this opportunity to observe and measure the impact of enabling Memory Tracking on performance. I was not at all surprised by what happened (the server became entirely unresponsive in very little time), but I was pleased that I was able to document the exact impact in a more empirical manner.

Environment
This experiment took place in a staging environment with 3 machines: One to host ColdFusion, another to host IIS, and a third to host JMeter. A performance test was created in JMeter to moderately exercise the application. It was run as a stress test by applying 100 Virtual Users indefinitely with 0 think time (no delay between a HTTP Response and the next HTTP Request). This means that at any given moment JMeter is making 100 simultaneous Requests to the CF server. The CF server is a virtualized instance with a max heap size set to about 12GB, sitting on Windows 7 with 25 GB RAM and 4 x 2.4GHz processors.

Let'r Rip!
The JMeter Test Plan was started and left to run for several hours, all the while pounding ColdFusion. During this period Monitoring was enabled in Server Monitor, but neither Profiling nor Memory Tracking were enabled. The CF Server's throughput was measured in Server Montitor to be steady at about 20 Requests per Second +/- 4 requests (steady range of 16-24). Memory Used by the JVM was a steady 3.9 GB at the peak followed by troughs of about 1.2 GB, with Garbage Collection happening once a minute. This created the typical sawtooth pattern when using the -XX:UseParallelGC JVM GC option. The CPU was typically in the range of 8-12% usage (across the 4 CPUs). The total throughput and memory utilization held steady for the several hours of testing. The app was performing beautifully, with 0 errors logged.

The Death Spiral
Before terminating the JMeter Test Plan execution, I enabled Memory Tracking in the Server Monitor. The JVM memory used began to quickly rise at the rate of about +700 MB per minute. The memory used jumped from a steady 4GB up to 8GB in just 6 minutes before the Server Monitor Interface stopped updating completely. Attempts to disable Memory Tracking were futile as button clicks did not respond. I could only watch the Task Manager on the CF server to continue observing memory and CPU. During the several hours of testing, total System Memory in Task Manager showed about 7.4 GB used, but after Memory Tracking was enabled and Server Monitor became unresponsive, I observed the total System Memory to be 13.7 GB, an increase of about 6 GB. The JVM was at or very close to being at the max heap allowed of 12 GB and was not able to reclaim any memory via GC. At this point I decided to kill the process tree of jrunsvc.exe (which also killed it's child process of jrun.exe, the main server instance). I stopped JMeter and then started ColdFusion again, making sure to disable Memory Tracking before running my next performance test run.

Caveat Emptor
This was a great example to see how Memory Tracking can bring a server to its knees. Often reported and well understood anecdotally, but I thought some actual screenshots of it in action would help illustrate how dangerous this setting can be. Memory Tracking can be used effectively in development and QA for debugging or troubleshooting performance issues, but only when used under small load. I should point out that sometimes Profiling is known to similarly bring down a server, but I was not able to observe any impact of the Profiling setting on this particular application as performance seemed normal when enabled during a stress test.

Server Monitor during Test
ColdFusion Server Monitor: Memory Tracking Performance Impact

[More]

Automated System Testing for Web Apps at CF.Objective

I'm excited to to have the honor of once again presenting at the CF.Objective() Enterprise ColdFusion Conference. This year I'll be talking about Automated System Testing for Web Applications with CFSelenium, MXUnit, and Jenkins.

I've been a Quality Assurance software developer since 2007 when I was on the ColdFusion server engineering team at Adobe. For the past couple years I've enjoyed working at FirstComp Insurance with one of the largest ColdFusion developer teams that I know of, including well known team members like Sam Farmer, Dan Vega, and Jason Delmore, as well as many others of ColdFusion's best.

Testing by Isolation
One of my goals last year was to create a test suite framework that could perform Automated System Testing of our collection of web applications that we use for our business. We run it all on ColdFusion with a truly massive code base, and we have many different web applications that drive different parts of the business, each with unique user interfaces (UI). Part of good development practices includes writing Unit Tests early in the project to test application modules (CFCs) in isolation. Unit Tests are great for catching issues early in the release cycle, but they don't test how all the parts work together across the whole application as a system.

Testing Across the Board
This is where System Testing (or UI Testing) comes in, and I'll be showing you how I built our automated UI test framework from the ground up.

[More]

Issue with stopping ColdFusion after starting from Builder

With the release of ColdFusion Builder there is an option available that provides the ability to start and stop one or more ColdFusion servers from ColdFusion Builder. In fact, you can configure CF Builder to automatically start a CF server when Builder opens, and stop the CF server when Builder closes. The autostart/autostop is convenient for a Development box where you want to minimize resource usage on the system. You can read more about this feature here.

Server Panel in ColdFusion Builder
Server Panel in ColdFusion Builder

Server Settings Panel in ColdFusion Builder showing Auto start/stop for CF Servers
Server Settings Panel in ColdFusion Builder showing Auto start/stop for CF Server

However, if you don't enable the automatic stop/start option, if you ever start ColdFusion server from Builder then close Builder without stopping ColdFusion there, then later you will not be able to stop the ColdFusion server using the standard ColdFusion stop script. I've encountered this on Mac OS X, but since its possible to configure CF Builder to start/stop remote CF servers, it's likely that the problem might occur when using ColdFusion server on Linux or Solaris as well, even though Builder doesn't run on those platforms.

Normally, to stop / start the ColdFusion server from the command line, you would the control script located (typically) at /opt/coldfusion9/bin/coldfusion, such as with ./coldfusion stop. That control script in turn invokes /opt/coldfusion9/runtime/bin/coldfusion9. When calling stop, the control script works by first grepping for any running ColdFusion processes with fgrep, like this: ps -axc | fgrep coldfusion9. If it finds a process listing that matches for the string "coldfusion9" then it stops that process.

Here's what you might see if you try to restart ColdFusion from the command line after it was started but not stopped from Builder:

view plain print about
1$ /opt/ColdFusion9/bin/coldfusion restart
2Restarting ColdFusion 9...
3ColdFusion 9 does not seem to be currently running
4Starting ColdFusion 9...
5The ColdFusion 9 server is starting up and will be available shortly.
6There has been an error starting ColdFusion 9, please check the logs.


The problem of not being able to use that control script to stop ColdFusion server after having started it from Builder arises because of how Builder starts the CF server. Rather than invoking /opt/coldfusion9/runtime/bin/coldfusion9, Builder instead invokes /opt/coldfusion9/runtime/bin/jrun. When the control script tries to grep for the process with a "coldfusion9" string, the control script doesn't find it because Builder invoked runtime/jrun instead of runtime/coldfusion9.

Why the need for runtime/jrun AND runtime/coldfusion9? I have no idea, especially since the files are identical and not symlinked.

view plain print about
1$ pwd
2/opt/ColdFusion9/runtime/bin
3$ diff jrun coldfusion9
4$




I logged ColdFusion server bug 82573 for this where I proposed a change to the bin/coldfusion control script. My suggested change was only shown for Mac OS X, but you can easily change it yourself for the Linux or Solaris blocks in a similar way.

If you want to use my suggested fix on your local Mac OS X dev box, then you can refer to the full example control script containing the fix here: http://pastebin.com/Y7r6sDGu.

For brevity, I won't show the whole script in this blog entry. Instead, here's the diff between the backed up original coldfusion control script which I renamed to 'orig.coldfusion' compared to the fixed version 'coldfusion'.

view plain print about
11.    $ diff orig.coldfusion coldfusion
2    2.    13a14
3    3.    >
JRUN_BIN="${CF_DIR}/runtime/bin"
4    4.    34c35
5    5.    < ⬠ ⬠ ⬠ ⬠ $PSCMD | fgrep coldfusion9 ⬠> /dev/null 2>&1
6    6.    ---
7    7.    > ⬠ ⬠ ⬠ ⬠ $PSCMD | grep -i $JRUN_BIN | grep -v 'grep' > /dev/null 2>&1
8    8.    117c118,119
9    9.    < ⬠ ⬠ ⬠ ⬠ ⬠ ⬠ $PSCMD | fgrep coldfusion9 | awk '{print $1}' | xargs kill -9 > /dev/null 2>&1
10    10.    ---
11    11.    > ⬠ ⬠ ⬠ ⬠ ⬠ $PSCMD | grep -i $JRUN_BIN ⬠| grep -v 'grep' | awk '{print $1}' | xargs kill -9 > /dev/null 2>&1
12    12.    > ⬠ ⬠ ⬠ ⬠ ⬠
13    13.    130,131c132,133
14    14.    < ⬠ ⬠ ⬠ ⬠ ⬠ ⬠ ⬠ $PSCMD | fgrep coldfusion9 | awk '{print $1}'
15    15.    < ⬠ ⬠ ⬠ ⬠ fi
16    16.    ---
17    17.    >
⬠ ⬠ ⬠ ⬠ ⬠ ⬠ ⬠ ⬠ $PSCMD | grep -i $JRUN_BIN ⬠| grep -v 'grep' | awk '{print $1}'
18    18.    >
⬠ ⬠ ⬠ ⬠ ⬠ ⬠ ⬠ ⬠ ⬠ ⬠ ⬠ fi
19    19.    152c154
20    20.    < ⬠ ⬠ ⬠ ⬠ ⬠ ⬠ ⬠ PSCMD="ps -axc"
21    21.    ---
22    22.    >
⬠ ⬠ ⬠ ⬠ ⬠ ⬠ ⬠ PSCMD="ps -ef"

Adobe LiveCycle DataServices for ColdFusion at CFObjective

Allaire's CEO, David OrfaoAfter a decade of working intensely with the ColdFusion server, I'm finally getting the courage to start presenting about it on the conference circuit. As a blogger, tweeter, and contributor to mailing lists I'm very confident helping others solve ColdFusion related problems because I can do that from the quiet comfort of my own desk. However, one of my greatest fears has always been public speaking. I'm the kind of person that feels like I need to know the subject matter cold, so that I can speak from the hip without relying on looking at the slides.




Blackstone Test CDsOver the years, I had some opportunities to present to small groups, and I recall each time feeling the adrenalin surge and my heart pounding. That started with presenting ColdFusion for Unix and Linux as an internal training class at Macromedia. Later, while taking classes at the Harvard Extension School, I was honored to be asked to present to CSCI-253 Developing Web-Based Database Applications. Even more so, I presented twice there in one year. The first time on Building ColdFusion Web Applications with CFEclipse and Dreamweaver, and later on ColdFusion Server Administration




MAX in ActionI've been attending ColdFusion conferences since the days of Allaire DevCon, but had never presented at any of them including MAX. My long time friend in the local ColdFusion Community, Brian Rinaldi, continued to encourage me to present at the local Boston CFUG as a starting point, as well as the new conference that he was organizing, RIA Unleashed, held in Bentley College this past November. The members of the CFUG were kind enough to let me present a draft of a presentation that I was to later give at RIA Unleashed. My presentation topic was Adobe LiveCycle DataServices Data Management for Mere Mortals


ColdFusion 1.5 on Floppy DisksFortunately at RIA Unleashed I was among the very first sessions after the keynote, so there was no time to build up butterflies that morning. If beforehand you would have told me that among the audience front row would be Ben Nadel, Simon Free, and Ray Camden with Tom Jordahl tucked way in the back then I surely would have freaked out. But they were both kind enough to chat with me before hand and even lend some technical assistance getting setup with the A/V, so that really put me at ease. With a firm limit of 50 minutes, I pushed all the way through what should have been a 90 minute talk, all the while trying to remember to speak clearly and loudly. The talk went off pretty much without a hitch as I found myself completely focused on the technical content and not at all worrying about the large room filled with people in front of me. I was delighted at the end when Tom complemented me on talk, which to me was the ultimate satisfaction.




First Unix machine to run ColdFusionI chose LCDS for ColdFusion as a topic because while I was a QA Engineer on the ColdFusion team at Adobe, I was paired with Tom, a Computer Scientist at Adobe who architected the integration between the products. Heck, Tom architected much of ColdFusion itself, and was in fact the original engineer to have ported ColdFusion to run on Unix and Linux back in the day. Tom is a font of information, and I cut my teeth on the feature under his guidance, which was then known as Flex Data Services and later renamed under the LiveCycle brand. I spent many days last summer and fall revisiting all the LCDS documentation again to ensure the quality of my presentation and to mentally prepare me for the upcoming conference.




ColdFusion Team, BangaloreWIth my first conference under my belt, I decided to throw my hat into the ring for the ultimate ColdFusion experience, CFObjective, which is promoted as The Only Enterprise ColdFusion Conference. I'm excited to announce that I have been selected to be a speaker at the conference, which runs from April 22-24th in Minneapolis, Minnesota. The conference is divided into three tracks for technologies related to ColdFusion. I'll be speaking the last day in the Flex track, once again on the topic of LiveCycle DataServices for ColdFusion Developers. Specifically I'll be talking about the prime feature of LCDS, the Data Management capabilities. With any luck I'll be updating my presentation to consider the benefits of working with the latest versions of Adobe software. Here's the brief description and the PDF:




Discussions of Adobe's LiveCycle Data Services are often entered with the same trepidation as those of Organic Chemistry or Quantum Mechanics, but with ColdFusion, building Web applications that manage complex data sets doesn't have to be that scary. Data Management is a pillar of LCDS that offers scalable, real-time data synchronization across very large numbers of connected clients with the benefits of conflict resolution and data pagination.† Come learn how to quickly get up to speed with Data Management by letting ColdFusion do the hard work for you.


If you're seriously interested in ColdFusion, then CFObjective is the conference for you. I hope to see you there.






ColdFusion Screams

Recent Tweets for Fri Oct 2, 2009 Part II

Follow me on Twitter!





Tue Sep 15 9:04 PM
@iotashan @rukumar Shan meet Rupesh. Rupesh meet Shan. You guys should talk CF9 ORM. ;-) Rupesh, Shan works with me & has an ORM issue
Tue Sep 15 8:51 PM
No CF Admin DSN setting for isolation level, but u can add SET TRANSACTION ISOLATION LEVEL <level> as u're validation query as workaround
Tue Sep 15 5:36 PM
Tue Sep 15 4:13 PM
@berniedolan Yup, and I was on a downhill at 35mph, slowed to 15 then skidded to within inches as he made a blinkerless right turn




[More]

Recent Tweets for Fri Oct 2, 2009

Follow me on Twitter!



Fri Oct 02 7:26 PM
Ghosts of ColdFusion Past http://yfrog.com/3omigoj
Fri Oct 02 4:40 PM
@john_mason_ Thanks. Indeed, the server was under high load.
Fri Oct 02 4:21 PM
Ditto that! RT @awest: Working at home really blows. Not. http://bit.ly/UflXy
Fri Oct 02 3:44 PM
@charliegriefer Enjoy! Twitter is gonna have a melt down. #adobeMAX
Fri Oct 02 3:12 PM
Have you ever launched the ColdFusion Server Monitor and seen the buttons for Monitoring, Profiling, and Memory just not show up at the top?




[More]

Recent Tweets for Fri August 28, 2009

Follow me on Twitter!





Fri Aug 28 8:52 PM
Why is it that ColdFusion must be started by a privileged user on every operating system but Mac? Hmph!
Fri Aug 28 8:43 PM
@shutupdanielle Could be worse, you could be in Bolton. 2 pizza shops, that's it. Nary a chicken wing in sight (really).
Fri Aug 28 8:36 PM
@mobarger Big city turn me loose, set me free, somewhere in the middle of Montana
Fri Aug 28 8:08 PM
@styggiti Outside temp last night was 52F here, but not sure which is better lower temps or lower humidity.
Fri Aug 28 5:43 PM
ColdFusion 9 (& ColdFusion 8.01) bug for CF Multiserver's bin/cf-init.sh file. Please vote: http://bit.ly/MsgCT
Fri Aug 28 4:56 PM
Yep, verified the 3 issues with the Solaris part of the cf-init.sh boot script installer for ColdFusion 9. Oh, bugtracker.....


[More]

Recent Tweets for Fri August 14, 2009

Follow me on Twitter!





Thu Aug 13 11:30 PM
"All Washed Up", new photo in the Trash The Dress series. #photog #photoshop http://flic.kr/p/6Px1MC
Thu Aug 13 2:48 PM
RT @iotashan: queries in cfscript is as bad as doing queries in any other lang. cfquery will always be superior to Query() in ease of use
Thu Aug 13 2:47 PM
@Photocritic In digital imaging half the data is in highlights & can be recovered PDFs: http://bit.ly/1ReSZ http://bit.ly/11osFd
Thu Aug 13 12:31 PM
@stevei Great posing on Paige for the lying down, forward shot. Would have been nice to have some more crash to the waves. #TTD #Photog






[More]

Recent Tweets for Monday August 4, 2009

Follow me on Twitter!





Tue Aug 04 9:57 PM
Dream Collaboration: @wiseacre + @merkley + Sean Ellis (Cashback) + Audrey Tautou
Tue Aug 04 9:01 PM
Web form pet peeve: I enter info and tab to next field, except when next field is a State combobox then it skips to zipcode. Aarrrrgh
Tue Aug 04 4:46 PM
Can Hardware Load Balancers manage session affinity based on J2EE session ids? Wondering abt HWLB over 2 webservers each with JRun cluster
Tue Aug 04 3:32 PM
My Webapper colleague, Tyson Vanek, blogged a case study of how he improved ColdFusion performance 1000x: http://bit.ly/M2Kt1
Tue Aug 04 12:14 PM
Phenomenal portraiture: Check out Bill Simone #photog http://bit.ly/PYQRy (via @ScottKelby )


[More]

ColdFusion Request Tuning Settings in Depth

Undoubtedly, the ColdFusion Administrator settings for Request Tuning are critical to performance of Web applications running in the server. While reading the recent Adobe article on Performance Tuning for ColdFusion Applications I was surprised to find the content on this topic to be a little light. With that in mind, I set out to expand on the topic of the Request Tuning settings.

Foundations of ColdFusion Request Settings

To begin, let’s look at how the ColdFusion settings were configured in earlier versions of application server. With the release of ColdFusion MX 6.0 through versions 6.1 and 7, all editions of the ColdFusion server had one setting for Request Limits. This was referred to as the “Simultaneous Requests” setting. This single setting throttled the number of running requests to be processed concurrently. Should the running pool be fully occupied by requests that are processing but haven’t yet completed, the J2EE server underlying ColdFusion will hold requests in a queued request thread pool that are to be fed to the running request pool.

[More]

More Entries