In tests that were done a couple years ago with ColdFusion MX, it was found that there is an upper limit on the number of operating system threads that can be open by one process, not to be confused with the number of file handle descriptors or jvm threads. When that os thread limit is reached for ColdFusion jrun process, the java.lang.OutOfMemoryError: unable to create new native thread error is thrown.
The results from testing on the Linux platform with Red Hat 7.x demonstrated the error when approximately 400 OS threads were tied to one OS process, the jvm process. Current Linux versions are thought to have improved that somewhat, so the limit might be higher at this time.
Solaris also demonstrated a limit, but had a higher threshold of about 1000-2000 OS threads per process, and Windows showed a limit of about 3000. at that time.
Given that for the JVM process under ColdFusion/JRun there is at least one operating system thread open for every JVM web thread, plus other operating system threads that are not tied to a JVM web thread, it is very likely that this error will occur if the total number of web threads (queued + running) begin to approach that experimental threshold of about 400 on Linux, for example.
Slow pages that might act as application bottlenecks can cause sufficient queuing where that OS thread limit would be reached, provided the maxHandlerThreads value in jrun.xml exceeds the experimental OS thread threshold. The default for maxHandlerThreads is 1000, so on Linux a maxHandlerThreads size of 500 or greater would likely see this error if the queue size ballooned.
Similarly, setting the activeHandlerThreads size to an unusually high number might also cause the error when sufficient numbers of users are on the application even if queueing is not occuring. Recently I have seen one such server where activeHandlerThreads (Simultanous Requests) was set to 500 and the "unable to create new native thread error" was the prominent symptom.
In those hypothetical configurations, the error java.lang.OutOfMemoryError: unable to create new native thread would begin to saturate the JRun logs. To clarify the error text, the error message means that no more native, or OS, threads can be created for the JVM process, causing the JVM to throw an exception. The exception type is java.lang.outOfMemoryError, but I find that to be rather misleading since the resource limitation on the OS thread count, not memory usage.
In the end, if so many web threads are being created that you're running into this error, then you'd probably want to hunt down bottlenecks via thread dumps and logging slow pages in order to increase throughput to get a lean mean activeHandler pool and reduce queuing. But if queuing is a problem, then you're more likely to see the "unable to create new native thread" error on Linux than on Solaris, and least likely to see this on Windows, although poor throughput looks the same no matter what the platform.
Related Blogs:
|
You can have a look at the process with "pmap" to see the memory allocated by the threads. Generally speaking, if you need more than the mentioned 400 threads, try to reduce the stack size using "-Xss128k" which should give you about 4 times the threads. Most applications should work fine with this stack size. If not, you'll get a "StackOverflowError" and have to increase the stack. I found a lower limit mentioned of 96kb on Linux.
Kurt Huwig