WebSphere Issues


Analyze javacore files – hung thread detection in WebSphere Application Server

Filed under: Java, WebSphere Application Server — Tags: , — Ishtiaque @ 12:02 pm


A javacore is a small diagnostic text file that is produced by the JVM. It contains a lot of vital information about the running JVM process. It provides a snapshot of all the running threads, their stack traces and the monitors (locks) held by the threads. It can be a key in detecting hang or deadlock conditions.

You can manually trigger javacore file either by OS (UNIX/Linux)) command kill -3 <JVM PID> or by using wsadmin commands that’s documented here https://websphereissues.wordpress.com/2012/07/16/generating-a-jvm-thread-dump-javacore-using-wsadmin-commands/

  • Take a few (at least three) snapshots of the JVM (about 2–3 minutes apart).
  • Analyze the javacore to see what different threads are doing in each snapshot.
  • Example: A series of snapshots where container threads are in the same method or waiting on same monitor or resource would be an indication of a bottleneck, hang, or deadlock.

Hung thread detection:

Hung threads can be hard to diagnose. They are often not noticed until many threads are hung resulting in a performance problem. Application threads can hang for a number of reasons, including infinite loops, deadlocks or inaccessible resources.
There is a component known as the ThreadMonitor that monitors the Web Container, ORB, and Async Bean thread pools for hung threads.

When a thread is suspected to be hung, then notifications are sent in three ways:

  • JMX notification for JMX listeners
  • PMI Thread Pool data is updated for tools like the Tivoli Performance Viewer
  • Message is written to the SystemOut log.

The thread monitor does not try to deal with the hung threads, it just issues notifications, so that the administrator or developer can deal with the issues.

  • When the thread pool gives work to a thread, it notifies the thread monitor.
  • Thread monitor notes thread ID and timestamp.
  • Thread monitor compares active threads to timestamps.
  • Threads active longer than the time limit are marked “potentially hung”.
  • Performance impact is minimal (< 1%)

How the thread monitor works:

  • The thread monitor architecture monitors Web container, ORB, and asynchronous bean thread pools. It’s enabled by default.
  • No action taken to kill the thread; only a notification mechanism
  • When a thread is suspected to be hung, a message is written to the server’sSystemOut.log
  • Example: thread monitor message

[7/15/08 15:03:11:502 EDT] 3c3b4e37 ThreadMonitor W WSVR0605W: Thread “Servlet.Engine.Transports : 0” (37c18e37) has been active for 68,839 milliseconds and may be hung. There are 1 threads in total in the server that may be hung.



Leave a Comment »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at WordPress.com.

%d bloggers like this: