How to analyze the thread stack

2) Generation principle of thread stack and related tools.

3) The format differences of different JVM thread stacks (Sun HotSpot, IBM JRE, Oracal JRockit).

4) Introduction and parsing method of thread stack log.

5) Analysis of thread stack and related technologies.

6) common problem templates (thread failure, deadlock, IO call deadlock, garbage collection /OutOfMemoryError problem, infinite loop, etc.). )

7) Example analysis of thread stack problem

I hope this series of training can bring you real help, so please pay attention to the weekly article update at any time.

But what should I do if I have questions or can't understand the content in the article during my study?

Don't worry, just consider me your mentor. You can ask me any questions about the thread stack (provided the questions are not too low-level). Please feel free to contact me in the following ways:

1) comment directly at the bottom of this article (you can be anonymous if you are embarrassed)

2) Submit your thread stack data to the root cause analysis forum.

Send me an email to @phcharbonneau@hotmail.com.

Can you help me analyze the problems encountered by our products?

Of course, if you like, you can send me your stack site data by email or forum root cause analysis forum. Dealing with practical problems is the king of learning and improving skills.

I sincerely hope you enjoy this training. So I will try my best to provide you with high-quality materials and answer your questions.

Before introducing thread stack analysis technology and problem pattern, let me tell you the basic content. So in this post, I will cover the most basic content first, so that everyone can better understand the interaction between JVM, middleware and Java EE container.

Overview of Java virtual machine

Java virtual machine is the foundation of Jave EE platform. It is the place where middleware and applications are deployed and run.

The JVM provides the following contents for middleware software and your Java/Java EE program:

-Java/Java EE program running environment (binary form)

–Some program features and tools (IO infrastructure, data structure, thread management, security, monitoring, etc. )

–Dynamic memory allocation and management with garbage collection

Your JVM can reside on many operating systems (Solaris, AIX, Windows, etc.). ), and according to your physical server configuration, you can install 1 to multiple JVM processes on each physical/virtual server.

Interaction between JVM and middleware

The following figure shows the high-level interaction model among JVM, middleware and application.

The diagram shows some simple and typical interactions among JVM, middleware and application components. As you can see, the thread allocation of standard Java EE applications is actually done between the middleware kernel and the JVM. (Of course, there are exceptions. Applications can directly call the API to create threads, which is not common and should be used with special care. )

At the same time, please note that some threads are managed by JVM, and a typical example is garbage collection thread, which JVM uses for parallel garbage collection.

Because most thread allocation is done by Java EE container, it is very important to understand and identify thread stack trace and identify it from thread stack data. This will enable you to quickly know what kind of request the Java EE container is executing.

By analyzing the thread dump stack, you will be able to understand the differences between thread pools found in JVM and identify the types of requests.

The last section will give you an overview of what is the JVM thread stack of HotSop VM and the different threads you will encounter. Details of the IBM VM thread stack form will be provided to you in the fourth section.

Note that you can get the thread stack example of this article from the root cause analysis forum.

JVM thread stack-what is it?

The JVM thread stack is a snapshot of a given time, which can provide you with a complete list of all created Java threads.

Every discovered Java thread will give you the following information:

–The name of the thread; Middleware vendors often use it to identify threads, and it usually carries the name and status (running, blocking, etc.) of the allocated thread pool. )

–Thread type &; For example, priority: daemon priority = 3 * * Middleware programs generally create their own threads in the form of background guards, which means that these threads run in the background; They will provide services to their users, for example, to your Java EE application.

–Java thread id, for example: tid = 0x00000000 1 kloc-0/E52A800 * * This is the Java thread ID obtained through java.lang.Thread.getId (), which is usually realized by the self-growing long integer1... n * *.

–Native thread ID, for example: nid=0x25 1c**, the key reason is that Native Thread ID can let you get relevant information, such as which thread in your JVM occupies most of the CPU time from the perspective of the operating system. * *

–-Java thread status and details, such as: waiting for the monitor entry [0xffffffe A5AFB000] java.lang.thread.state: blocked (on the object monitor).

* * You can quickly know the thread status and possible reasons for the current blocking * *

–-Java thread stack trace; This is the most important data you can find from the thread stack so far, and it is also the place where you spend the most time analyzing, because Java stack tracing provides you with 90% information you need to understand the root causes of various types of problems in the later practice.

–-Java heap memory decomposition; Starting from the version of HotSpot VM 1.6, we can see the memory usage of HotSpot at the end of the thread stack, such as Java's YoungGen (OldGen) & PermGen space, which is very useful in analyzing the problems caused by frequent GC. You can use known thread data or patterns to locate quickly.

The heap PSYoungGen is 466944K in total, with178734k [0xffffff45c00000, 0xffffff70800000, 0xffffff70800000] Eden space of 233472K used and 76% space of 20 used.

Thread stack information disassembly

In order to make you better understand, I will provide you with the following figure, in which the thread stack information and thread pool on HotSpot VM are disassembled in detail, as shown in the following figure:

As can be seen from the above figure, the thread stack consists of many different parts. This information is very important for problem analysis, but different parts will be used in the analysis of different problem patterns (problem patterns will be simulated and demonstrated in later articles). )

Now, through this analysis example, I will explain in detail the various components in the thread stack information on HoteSpot:

# Full thread dump identifier

"Full thread dump" is a globally unique keyword, which can be found in the output log of thread stack information of middleware and independent Java (for example, used under UNIX: kill-3)

Threads dump Java HotSpot(TM) 64-bit server virtual machine (20.0-b 1 1 mixed mode);

# Java EE middleware, third-party threads and customized applications

This part is the core part of the whole thread stack, and it is usually the part with the most analysis time. The number of threads in the stack depends on the middleware you use, third-party libraries (there may be independent threads) and your application (this is usually not a good practice if you create custom threads).

In our sample thread stack, WebLogic is the middleware we use. Starting from Weblogic 9.2, a self-managed thread pool uniquely identified by "weblogic.kernel.default (self-tuning)" will be used.

"[standby] execute thread:' 414' for queue:' weblogic.kernel.default'" daemon prio = 3tid = 0x00000001091 6a800nid = 0x2613 inobject. wait () [0xfffff9edf000] java.lang.thread.state: waiting (on the object monitor) waits on java.lang.Object.wait (native method) 0xffffffff27d44de0 >: (a weblogic.work.execute thread) at java.lang.object.wait (object.java: 485) at weblogic.work.execute thread.wait for request (execute thread.java:160)-locked <; 0x ffffffff 27d 44 de 0 & gt； (a weblogic.work.ExecuteThread) is located in weblogic.work.executeThread.run (executeThread.java:181).

# Hotspot virtual machine thread

This is an internal thread managed by Hotspot VM to perform internal native operations. Generally speaking, unless you find that the CPU usage is high (through the related thread stack and prstat or native thread Id), you don't have to worry too much about it.

"VM periodic task thread" prio = 3tid = 0x00000001kloc-0/238800nid = 0x19 waiting condition.

# HotSpot garbage collection thread

When HotSpot is used for parallel GC (which is very common in the environment where multiple physical cores are used now), when the default HotSpot VM or each JVM manages a GC thread with a specific identifier, these GC threads can make the VM perform its regular GC cleaning in parallel, which will lead to an overall reduction in GC time. At the same time, the cost is that the usage time of CPU will increase.

"GC task thread # 0 (parallel GC)" prio = 3tid = 0x00000000100120000nid = 0x3runnable "GC task thread # 1(parallel GC). " prio = 3 tid = 0x 00000000 100 13 1000 NID = 0x 4 runnable…………………………………………………………………………

This is very critical data, because when you encounter GC-related problems, such as excessive GC and memory leaks, you will be able to use the operating system or Java threads associated with the native Id values of these threads, and then find any high usage of CPI time. In future articles, you will learn how to identify and diagnose such problems.

# JNI Global Reference Count

The global reference of JNI (Java local interface) is the basic object reference from local code to Java objects managed by Java garbage collector. Its function is to prevent garbage collection of objects that are still used by local code, but are not technically referenced by "activities" in Java code.

At the same time, in order to detect leakage related to JNI, it is also important to pay attention to JNI reference. If your program directly uses JNI or third-party tools such as listeners, it is easy to cause local memory leakage.

JNI Global Reference: 1925

# Java stack uses views

These data have been added back to JDK 1 .6, providing you with a short and quick view of the hotspot stack. I find it very useful when I deal with the problem of high CPU usage related to GC. You can see the thread stack and Java heap information in a single snapshot, so that you can analyze (or exclude) any key points in the memory space of a specific Java heap at that time. If you are in our sample thread stack,

The heap PSYoungGen is 466944K in total, with178734k [0xffffff45c00000, 0xffffff70800000, 0xffffff70800000] Eden space of 233472K used and 76% space of 20 used.