Analyze javacore files – hung thread detection in WebSphere Application Server

Javacore:

A javacore is a small diagnostic text file that is produced by the JVM. It contains a lot of vital information about the running JVM process. It provides a snapshot of all the running threads, their stack traces and the monitors (locks) held by the threads. It can be a key in detecting hang or deadlock conditions.

You can manually trigger javacore file either by OS (UNIX/Linux)) command kill -3 <JVM PID> or by using wsadmin commands that’s documented here https://websphereissues.wordpress.com/2012/07/16/generating-a-jvm-thread-dump-javacore-using-wsadmin-commands/

  • Take a few (at least three) snapshots of the JVM (about 2–3 minutes apart).
  • Analyze the javacore to see what different threads are doing in each snapshot.
  • Example: A series of snapshots where container threads are in the same method or waiting on same monitor or resource would be an indication of a bottleneck, hang, or deadlock.

Hung thread detection:

Hung threads can be hard to diagnose. They are often not noticed until many threads are hung resulting in a performance problem. Application threads can hang for a number of reasons, including infinite loops, deadlocks or inaccessible resources.
There is a component known as the ThreadMonitor that monitors the Web Container, ORB, and Async Bean thread pools for hung threads.

When a thread is suspected to be hung, then notifications are sent in three ways:

  • JMX notification for JMX listeners
  • PMI Thread Pool data is updated for tools like the Tivoli Performance Viewer
  • Message is written to the SystemOut log.

The thread monitor does not try to deal with the hung threads, it just issues notifications, so that the administrator or developer can deal with the issues.

  • When the thread pool gives work to a thread, it notifies the thread monitor.
  • Thread monitor notes thread ID and timestamp.
  • Thread monitor compares active threads to timestamps.
  • Threads active longer than the time limit are marked “potentially hung”.
  • Performance impact is minimal (< 1%)

How the thread monitor works:

  • The thread monitor architecture monitors Web container, ORB, and asynchronous bean thread pools. It’s enabled by default.
  • No action taken to kill the thread; only a notification mechanism
  • When a thread is suspected to be hung, a message is written to the server’sSystemOut.log
  • Example: thread monitor message

[7/15/08 15:03:11:502 EDT] 3c3b4e37 ThreadMonitor W WSVR0605W: Thread “Servlet.Engine.Transports : 0” (37c18e37) has been active for 68,839 milliseconds and may be hung. There are 1 threads in total in the server that may be hung.

Source:

Advertisements

About Ishtiaque

I am IBM Certified Infrastructure Systems Architect, TOGAF 9 Certified, ITIL V3 Foundation certified and Oracle Certified Export in Java Web Component Developer with over 10 years of support and development experience in IBM middleware software and Java. Additionally, have a sound grip in databases, Linux and OpenStack administration. I hold the following certifications: IBM Certified Infrastructure Systems Architect TOGAF 9 Certified Oracle Certified Expert, Java EE6 Web Component Developer Oracle Certified Professional – Java 6 Programmer ITIL v3 Foundation Certified IBM Certified Solution Architect – Cloud Computing Infrastructure V1 IBM Certified System Administrator – WebSphere Portal V8, V7, V6.1, V6 IBM Certified System Administrator – WebSphere Application Server V7, V6.1 IBM Certified System Administrator – AIX V7 IBM Certified System Administrator – WebSphere MQ V7 IBM Certified Deployment Professional – Business Process Manager Advanced V7.5 IBM Certified Solution Advisor – Cloud Computing Architecture V3 IBM Certified Solution Developer – WebSphere Portal V5.1
This entry was posted in Java, WebSphere Application Server and tagged , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s