bpurcell.org - Using JRun Metrics Instead of CFSTAT on ColdFusion MX
Calendar
SunMonTueWedThuFriSat
  1 2 3 4 5 6
7 8 9 10 11 12 13
14 15 16 17 18 19 20
21 22 23 24 25 26 27
28 29 30 31      

Subject Archives
Amazon EC2 (15)
ColdFusionMX (155)
Corvette (3)
Flash Lite (4)
Flash Media Server (5)
Flash Player (3)
Flex (39)
General Web Dev (14)
HDTV (3)
Jboss (1)
Jquery (2)
JRun (59)
Max 2003 (3)
Other (33)
PC Hardware (16)
Software (17)
SpatialKey (7)
Wireless (8)
Working Out (1)

RSS Feed
Feed Listing

Site Contents
Home
My Blog
ColdFusion
JRun
Flex
Wireless & Networking
Hardware & Gadgets
HDTV
Software Picks
Pictures
Contact Me
Search


My Articles & Papers
Flex Performance
Server/Client
mm.com Under the Hood
Multiple Instances of CFMX
Multiple Instance Config
NLB with ColdFusion
Clustering CFMX for J2EE
Multi-Tier Hardware LB w CFMX
Cisco CSS & Coldfusion MX
JRun 4 Jini based Clustering
WiFi Growth

2ID Tacweb

Other Hobbys
Body-For-Life Challenge

Personal Projects
Family Pool
Deck Gate

username:
password:
 

 
Viewing Individual Entry / Main
January 12, 2004

If you install ColdFusion MX 6.1 in the Standalone mode and you want to get realtime statistics on how the server is operating then you can use JRun metrics.  There are several advantages that the metrics offers over cfstat. The one disadvantage is that it does not provide exactly the same data.

Advantages of JRun Metrics over CFSTAT

  1. Logs directly to the JRun logs with timestamps
  2. Format is customizable
  3. Provides JVM memory information

It is pretty simple to enable JRun metrics Open the file /CFusionMX/runtime/servers/default/SERVER-INF/jrun.xml

Uncomment the entry
<service class="jrunx.metrics.MetricsService" name="MetricsService">
<attribute name="bindToJNDI">true</attribute>
</service>

Inside of <service class="jrunx.logger.LoggerService" name="LoggerService"> modify
<attribute name="metricsEnabled">true</attribute>
<attribute name="metricsLogFrequency">10</attribute>

Modify the attributes (remove the jrpp. prefix)
<attribute name="metricsFormat">
  Web threads (busy/total): {jrpp.busyTh}/{jrpp.totalTh} Sessions: {sessions} Total Memory={totalMemory} Free={freeMemory}
</attribute>
change to
<attribute name="metricsFormat">
  Web threads (busy/total): {busyTh}/{totalTh} Sessions: {sessions} Total Memory={totalMemory} Free={freeMemory}
</attribute>

By setting the metrics Log Frequentcy to 10 the metrics data will be logged every 10 second with the following format
10/24 12:34:51 metrics Web threads (busy/total) 0/1 Sessions: 0 Total Memory=12032 Free=5880

One note, if you do not have J2EE sessions enabled in ColdFusion Admin then you will not see the sessions in the metrics

If you want to customize the data that metrics provides then you can use the following attributes

AvgQueueTime, AvgReqTime, busyTh, bytesIn, bytesOut, delayMs, delayRq, delayTh, droppedRq, freeMemory, handledMs, handledRq, idleTh, listenTh, sessions, sessionsInMem, totalMemory, totalTh

In the example below I will log not only busy and total threads I will also log; listening, idle, delayed and total handled.

This format works for ColdFusion MX Standalone:
<attribute name="metricsFormat">
  Web threads (busy/total/listen/idle/delay/handled): {busyTh}/{totalTh}/{listenTh}/{idleTh}/{delayRq}/{handledRq} Sessions: {sessions} Total Memory={totalMemory} Free={freeMemory}
</attribute>

This format works for ColdFusion MX J2EE version on JRun:
<attribute name="metricsFormat">
  Web threads (busy/total/listen/idle/delay/handled): {jrpp.busyTh}/{jrpp.totalTh}/{jrpp.listenTh}/{jrpp.idleTh}/{jrpp.delayRq}/{jrpp.handledRq} Sessions: {sessions} Total Memory={totalMemory} Free={freeMemory}
</attribute>

For information about these attributes see the JRun documentation. If you are on CFMX standalone then ignore the jrpp. and web. prefixes. If you are running JRun with CFMX J2EE on top then you will need the jrpp. and .web prefixes. 

Comments

Are there significant performance implications of running this on a production site? I know the initial reaction is to test internally, but this would help pinpoint an issue we seem to be only having in production. But I don't want to jeopardize our site's uptime due to crashes from enabling this.


You can also use tail to show the log file as it is written.

If you are on windows, you can get tail from here: http://www.weihenstephan.de/~syring/win32/UnxUtils.html


The metrics are built into a core part of JRun so they are very efficient. With load testing I have found no differences in performance with metrics on or off. When we went live with macromedia.com we left them on for several weeks to monitor the site live.


I've put in a log format using the example format (busy/total/listen/idle/delay/handled), but I'm not sure I understand how to determine exactly what's going on. What's the difference between the different thread types? Are there certain threshold numbers that I should look out for? LiveDocs is no help.


Dave, I will do my best to explain the threads below. It is hard to tell exactly what is happening unless you are under load

busy - Threads that are running (executing code) this number will not exceed jrpp active threads for JRun or simultaneous request set in the cfadmin

total - Total number or jrpp and web threads that are currently available.

listen - number of threads that are listening and waiting for a connection from the webserver connector.

idle - threads that have been created, may have been used but sitting idle in the pool

delay - number of threads that were delayed during the metrics snapshot time, should always be 0 unless the server is very heavily loaded

handled - number of requests by CF/JRun handled during the snapshot time of metrics.

Pay attention to busy most of all, let me know if you have any other questions.


If you are using the J2EE with JRun 4 the line is:

<attribute name="metricsFormat">Web threads (busy/total/listen/idle/delay/handled): {jrpp.busyTh}/{jrpp.totalTh}/{jrpp.listenTh} /{jrpp.idleTh}/{jrpp.delayRq}/{jrpp.handledRq} Sessions: {sessions} Total Memory={totalMemory} Free={freeMemory}</attribute>


Is there a way to get the metics through a web page? Perhaps a JSP or Servlet?


What is the correct syntax to get AvgQueueTime & AvgReqTime to display on CFMX J2EE on JRun? Are these 2 metrics even possible on CFMX J2EE?


You would do something like the following: <attribute name="metricsFormat"> Web threads (busy/total/listen/idle/delay/handled): {jrpp.busyTh}/{jrpp.totalTh}/{jrpp.listenTh}/{jrpp.idleTh}/{jrpp.delayRq}/{jrpp.handledRq} Sessions: {sessions} Total Memory={totalMemory} Free={freeMemory} AvgQue={jrpp.AvgQueueTime} AvgReq={jrpp.AvgReqTime} </attribute>


I was incorrect when I stated that AvgReqTime and AvgQueTime were available. I pulled them from CFStat on accident. The only available options for metrics are documented here

http://livedocs.macromedia.com/jrun/4/JRun_Administrators_Guide/netmon2.htm

Instead you could use jrpp.handledMS and jrpp.delayMs


Has MM resolved product issue 53191, where the metrics logs negative values, i.e sessions: -21?


Matt, This issue has not been resolved, the product team has been unable to reproduce the problem and I have only seen it happen in rare cases.


I've got the negative session problem too.

06/24 21:25:01 metrics Web threads (busy/total): 0/7 Sessions: 0 Total Memory=1028224 Free=852391 06/24 21:25:39 metrics Web threads (busy/total): 0/7 Sessions: -2 Total Memory=1029120 Free=853802

The server was started on 06/23 15:33:31


And also, the Free Memory value is incorrect. Metrics are reporting

06/25 05:45:54 metrics Web threads (busy/total): 0/4 Sessions: -12 Total Memory=1048448 Free=907466

but the garbage collector says

Heap PSYoungGen total 88576K, used 922K [0x44900000, 0x4bac0000, 0x4bac0000) eden space 60672K, 0% used [0x44900000,0x44900000,0x48440000) from space 27904K, 3% used [0x49f80000,0x4a0668c8,0x4bac0000) to space 27904K, 0% used [0x48440000,0x48440000,0x49f80000) PSOldGen total 932096K, used 173003K [0x4bac0000, 0x84900000, 0x84900000) object space 932096K, 18% used [0x4bac0000,0x563b2c48,0x84900000) PSPermGen total 29184K, used 29147K [0x84900000, 0x86580000, 0x94900000) object space 29184K, 99% used [0x84900000,0x86576f98,0x86580000)


My bad. That was just the end of the last log file, which I wasn't tailing any more because it got rolled over. It seems that the JRun metrics have actually recovered by themselves:

06/25 09:37:33 metrics Web threads (busy/total): 1/13 Sessions: 3 Total Memory=1019328 Free=786915


Hello, there's a couple of things I don't understand. If anyone could explain I'd be very grateful. Why are the memory values nowhere near my Maximum JVM Heap Size which is set at 512MB, or my maxPermSize set at 128MB? Also, am I right in thinking that "busy" is limited by "activeHandlerThreads" and "Total" by "maxHandlerThreads"? If so, do they correspond to the SchedulerService's children or the ProxyService's service's children, or should they be exactly the same anyway?

Donal

04/06 15:34:40 metrics Web threads (busy/total/listen/idle/delay/handled): 1/5/3/0/0/91 Sessions: 0 Total Memory=14976 Free=7044 04/06 15:34:50 metrics Web threads (busy/total/listen/idle/delay/handled): 1/11/4/6/0/102 Sessions: 0 Total Memory=15552 Free=7153 04/06 15:35:00 metrics Web threads (busy/total/listen/idle/delay/handled): 1/11/3/7/0/102 Sessions: 0 Total Memory=15552 Free=7094 04/06 15:35:10 metrics Web threads (busy/total/listen/idle/delay/handled): 1/11/4/6/0/102 Sessions: 0 Total Memory=15552 Free=7035 04/06 15:35:20 metrics Web threads (busy/total/listen/idle/delay/handled): 2/11/3/6/0/102 Sessions: 0 Total Memory=15552 Free=6976 04/06 15:35:30 metrics Web threads (busy/total/listen/idle/delay/handled): 1/11/4/6/0/102 Sessions: 0 Total Memory=15552 Free=6917 04/06 15:35:40 metrics Web threads (busy/total/listen/idle/delay/handled): 2/15/3/10/0/104 Sessions: 0 Total Memory=16576 Free=4411 04/06 15:35:51 metrics Web threads (busy/total/listen/idle/delay/handled): 1/15/4/10/0/105 Sessions: 0 Total Memory=16384 Free=7597


I'm trying to turn on Metrics following your example here, Brandon, but I'm getting the following error in the CFMX 7 server log:

The metrics service is disabled for the J2EE edition

Um, is this saying it's just disabled by default (and I've overlooked something in my attempt to turn it on), or it is saying that I *can't* turn it on? The version I'm playing with here at work is the Developer edition. Thanks for any info.


Is it possible to poll JRUN to get the current log format?

Since the log format is so configurable, I'd like to be able to determine which column/element in the .log file is what without having to hardcode it in my reader. So that, if one of my servers outputs to the log as {scheduler.listenTh};{scheduler.delayTh};{scheduler.idleTh};{scheduler.busyTh};{scheduler.totalTh}; and another of my servers outputs to the log as {scheduler.idleTh};{scheduler.totalTh}; I can use the same code to read both logs and determine what values correlate to what attributes.

I thought about reading the JRUN.XML... but in many cases, the actual elements are not specified in the metricsFormat attribute.


I also notice the error in the server log

The metrics service is disabled for the J2EE edition

How does one actually monitor the performance after setitng up jrun metrics. I guess it is not through perfmon.


> How does one actually monitor the performance after setitng up jrun metrics.

I put together a pretty simple app to monitor JRUN metrics. I posted it to the Macromedia ColdFusion Exchange under ColdFusion Exchange -> Applications -> Business Systems. Called "J2EE/JRUN Metrics Log Viewer"

http://www.macromedia.com/cfusion/exchange/index.cfm?view=sn131&extid=1029407


 
Page Render Time:203