bpurcell.org - Performance Tuning and Monitoring Coldfusion or JRun
Calendar
SunMonTueWedThuFriSat
  1 2 3 4 5 6
7 8 9 10 11 12 13
14 15 16 17 18 19 20
21 22 23 24 25 26 27
28 29 30        

Subject Archives
Amazon EC2 (15)
ColdFusionMX (155)
Corvette (3)
Flash Lite (4)
Flash Media Server (5)
Flash Player (3)
Flex (39)
General Web Dev (14)
HDTV (3)
Jboss (1)
Jquery (2)
JRun (59)
Max 2003 (3)
Other (33)
PC Hardware (16)
Software (17)
SpatialKey (7)
Wireless (8)
Working Out (1)

RSS Feed
Feed Listing

Site Contents
Home
My Blog
ColdFusion
JRun
Flex
Wireless & Networking
Hardware & Gadgets
HDTV
Software Picks
Pictures
Contact Me
Search


My Articles & Papers
Flex Performance
Server/Client
mm.com Under the Hood
Multiple Instances of CFMX
Multiple Instance Config
NLB with ColdFusion
Clustering CFMX for J2EE
Multi-Tier Hardware LB w CFMX
Cisco CSS & Coldfusion MX
JRun 4 Jini based Clustering
WiFi Growth

2ID Tacweb

Other Hobbys
Body-For-Life Challenge

Personal Projects
Family Pool
Deck Gate

username:
password:
 

 
Viewing Individual Entry / Main
October 31, 2003

Many times I visit customers for performance tuning engagements and I have a process that I go before I start load testing. This process allows me to really see what is happening inside the server. After running a few tests and monitoring this data I can make adjustments to JRun, CFMX, and the JVM. Unless you know what is going on you are really guessing with performance turning. These steps should get you started in the right direction. I plan on working on several other articles geared around load testing, performance tuning, and JVM optimization. Feel free to leave a comment if you have any suggestions for things you would like to learn more about.

You should also review the following presentation.

Use the following steps to configure CFMX 6.1 with the JRun install. It is a bit different with the standalone install.

  1. Enable JRun Metrics in /jrun4/servers/{servername}/SERVER-INF/jrun.xml
    Uncomment the entry
    <service class="jrunx.metrics.MetricsService" name="MetricsService">
    <attribute name="bindToJNDI">true</attribute>
    </service>


    Inside of <service class="jrunx.logger.LoggerService" name="LoggerService"> modify
    <attribute name="metricsEnabled">true</attribute>
    <attribute name="metricsLogFrequency">5</attribute>


    By setting the metrics Log Frequentcy to 5 the metrics data will be logged every 5 second with the following format
    10/24 12:34:51 metrics Web threads (busy/total) 0/1 Sessions: 0 Total Memory=12032 Free=5880
  2. Configure Logging to log to separate files by modifying the entry
    <attribute name="filename">{jrun.rootdir}/logs/{jrun.server.name}-event.log</attribute>
    in jrun.xml. Change it to the following
    <attribute name="filename">{jrun.rootdir}/logs/{jrun.server.name}- {log.level}. log</attribute>
  3. On Unix create a custom start script so JVM out stream gets logged to a file
    Script name: start-cfusion
    /web/jrun4/bin/jrun -nohup -start cfusion 1>/web/jrun4/logs/cfusion-out.log 2>/web/jrun4/logs/cfusion-err.log

    This script will log all console output to logs and allow you to capture thread dumps and other data from the logging stream.

    On Windows you need you can follow the following technote
  4. Enable GC details (only available in JVM 1.4.x/CFMX 6.1 ships with 1.4.2 and this works) 
    -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC

    For more details see Tuning Garbage Collection

Areas to monitor under load

  1. JRun Metrics - /jrun4/logs/servername-metrics.log
    Monitor the total and in-use threads, also used for monitoring free memory in the JVM
  2. GC Logs - /jrun4/logs/servername-out.log

    After enabling the verbose garbage collection options you can monitor GC times and heap memory allocation. This is the kind of data you will use to tune the JVM arguments.

Example Verbose Output: See Tuning Garbage Collection for details

Heap
PSYoungGen total 3584K, used 3583K [0x027f0000, 0x02bf0000, 0x060d0000)
eden space 3072K, 99% used [0x27f0000,0x2aeffe8,0x2af0000)
from space 512K, 100% used [0x2b70000,0x2bf0000,0x2bf0000)
to space 512K, 0% used [0x2af0000,0x2af0000,0x2b70000)
PSOldGen total 1408K, used 1287K [0x060d0000, 0x06230000, 0x227f0000)
object space 1408K, 91% used [0x60d0000,0x6211d10,0x6230000)
PSPermGen total 6528K, used 6489K [0x227f0000, 0x22e50000, 0x2a7f0000)
object space 6528K, 99% used [0x227f0000,0x22e46470,0x22e50000)
6.47463: [GC 4871K->2199K(5376K), 0.0118130 secs]
Heap after GC invocations=7:

  1. Server CPU
    Top or grstat on Unix
    Task Manager on Windows
  2. ColdFusion Logs (Application, Exception) located in /jrun4/servers/{servername}/{cfusionhome}/WEB-INF/cfusion/logs

    Look for Application errors or any out of the ordinary messages in the logs

Troubleshooting:

Under load the application server can appear to slow down and even in some cases hang. Using the data above and JVM Thread Dumps we can quickly diagnose and identify the problems. If a server stops responding to requests you will need to take a thread dump. You should also take one in a case where the thread count in metrics does not return to 0 after a load test ends.

The following document can assist in steps for obtaining the thread dump and diagnosing the results

http://www.macromedia.com/support/jrun/ts/documents/tn18362.htm

Steps for diagnosing a hung JRun Server:
http://www.macromedia.com/support/jrun/ts/documents/4.0_unresponsive.htm

Comments

This information is invaluable and I sincerely thank you for posting it. If you don't mind, I'm going to send this out to cf-talk, as there are posts almost daily looking for this sort of information.

I look forward to the other articles you refer to that are in-progress.

Thanks again, Dave.


Dave feel free to post it wherever you want


hey brandon,

one article I would be very interested in reading is the effects of the JVM config under multiple instances in a cluster. We have found the right maxPermSize/maxHeapSize settings for a particular instance....then we went to a cluster. The question becomes - how is memory utilized (if at all) differently wihtin a cluster?

I know I could just load test to see where the breaking point is, but you have all this good data in your head.

Thx...


I was reading the article that you linked to in your post regarding tuning garbage collection. While I understand how I can apply some of the suggestions based on my application code, is there a document anywhere that describes the types of objects the JRun/CFMX engines themselves create regarding young vs. permanent generation objects? It would seem that most objects created by the underlying JRun/CFMX engines would be permanent generation objects, but I don't really know for sure. I know that my application code is mostly short-lived objects, but I don't want to tune my JVM for that only to find out that JRun/CFMX needs more room in the permanent generation because of long-lived objects it uses. How would I go about figuring this out? What JVM arguments/garbage collector do you use?


Primarily CF uses short term objects and they are GC'd in the young generation. These objects are created on a request and dereferenced at the end of the response. The core classes of CF are stored in the permanent generation at startup and the cached template files are stored there as well. If you are caching other data in the shared scopes the would usually be placed in the Old Generation when they are promoted from the young generation. So, really GC settings are application specific with you CF app. You can usually get away with the default settings and just bump up the Permsize to 128 or 192. The best way though is to load test with the verbose options on and watch the details of the JVM. Run the test for 8 hours or more and checkout memory utilization. You can then bump up the young, old or perm to numbers that suit your app. I will try to write a posting including more details on this.


Quick update on this blog entry I had an error in the verbose GC settings: -XX:+PrintGCTimeStamps -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC

should have been: -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC

I corrected this in the entry. It now prints the timestamps for time it takes to do GC.


We've been using Perfmon to monitor CF in stand alone server mode.

Since switching to a dual-instance J2EE install on JRun 4, we've lost the ability to use Perfmon to get stats on CF.

Is there anyway to reenable this? Google and Macromedia searches are not turning up too much.


Tom, I need the same thing. If you get any information regarding using perfmon in the J2EE config, please post back hear. And I'll do the same.


This is a very important blog post for the CFMX development community. Thanks Bruce.


On my default instalation of 6.1 the logging section of JRUN.XML had this line in it:

<attribute name="metricsFormat">Web threads (busy/total): {jrpp.busyTh}/{jrpp.jrtotalTh} Sessions: {sessions} Total Memory={totalMemory} Free={freeMemory}</attribute>

I had to remove the "jrpp." to get it to work. That reference is apparently assumed. I'm not sure if it's like that on every default 6.1 installation, but this format works.

<attribute name="metricsFormat">Web threads (busy/total): {busyTh}/{totalTh} Sessions: {sessions} Total Memory={totalMemory} Free={freeMemory}</attribute>


Thanks for the info. Another useful article http://slidesix.com/view/10steps-to-a-stable-and-scalable-ColdFusion-application


This is a really great site. I like to tell that this is really a great experience to b here. I like to share my creation here. You can find out it at http://www.researchonindia.com/industry/agriculture-145/agrochemicals-and-fertilisers/fertilizer.html


 
Page Render Time:218