Misc

Tuesday, May 05, 2009

Javadoc tool with @exclude tag

The standard javadoc utility shipped with JDK doesn't provide an option to exclude some public methods from public API documentation. Currently we can filter only at file level (either all methods from all classes in the file or none).
Javadoc specification proposes a new tag called '@exclude' for this purpose - any method which @exclude tag shall be excluded from javadoc documentation. However this has not been implemented yet.
Here is a modified javadoc utility which can implements @exclude tag - any method with @exclude tag will be excluded from javadoc.
Usage:
1) download the javadoc.jar
2) create a file with list of source files which contain the methods to be documented
eg: inputs.txt
---------------------------------------------
c:\src\com\abc\spi\Abc.java
c:\src\com\abc\util\Xyz.java
----------------------------------------------
3) invoke javadoc as follows
javadoc -docletpath -d -doclet com.sun.tools.doclets.standard.Standard1 @
Please note that name of the file has been renamed to Standard1.

Sunday, April 19, 2009

Method Tracing

Assume you have a complex application probably developed by somebody else. Sometime you have an application which runs on top of another platform like application server or portal. Sometime your application may be using some other third party component.
You may want to know trace the execution of specific requests or transactions in the system probably for debugging. This is also helpful to understand the architecture/design of the system.
You can do that using this tool. This tool traces the progress (execution) of each thread.

Steps:
0) Download and copy the following 3 jars to a convenient location. (all jars has to be in the same directory). (Click on the link to download files).
MethodTracer.jar
asm-3.1.jar
asm-commons-3.1.jar
1) Add the following JVM options during startup.
-javaagent:[path_to_MethodTracer.jar]
If you are using some startup script you may need to add it there. If you are using some exe to invoke java then there may be some file where you will be able to add this.

2) Extract trace.properties from the jar and edit it to change the configurations. Put it back. You can use the following commands for extracting the property file and put it back into jar.
jar -xvf MethodTracer.jar tracer.properties
jar -uvf MethodTracer.jar tracer.properties
Following are the options available.
ExcludedPackages – classes from this coma separated list of packages shall be excluded from instrumentation and hence execution of such methods shall not be traced.
IncludedPackages – If specified only classes from this packages shall be traced. You can use includePackages and excludePackages together to get fine control
allThreadsLogged – if true, all requests by threads shall be traced. You can trace the requests by specific threads by turn this flag false and use threadList
threadList – if allthreadsLogged is false, then only the threads specified in this list shall be traced.
This is useful in the case of systems like application server where there shall be lot of threads running always. Different kinds of requests are typically processed by separate threads.
loggingOnAtStartup – if true, tracing shall start from application startup. It may produce lot of verbose output. If you want to trace only certain specific transactions . You can turn it false and start/stop tracing from jconsole.
File/console – turn true/false appropriately to send the trace output to file (method_trace.log in the current directory of the application) or console window.

3) start the application. Name of all instrumented classes are printed out to the console. If classes from any of the packages of not interest are instrumented, you can modify the tracer.properties file (includePackages/excludePackages). (requires re-start of the application).

4) start jconsole and connect to this application. Jconsole is tool comes along with JDK. You can start jconsole by executing the jconsole.exe in the JDK/bin directory. While start up it will display the list of all available JVMs in the machine and ask to choose the one to connect. Choose the JVM of your application. Goto the ‘MBeans’ tab. Expand tracer >> custom >> configuration >> operations. Use enableLogging method to start/stop logging (as Boolean as parameter). Use addThread and removeThread to trace one (or list of) thread (name of the thread as parameter. You can see the list of all threads and it’s name from the ‘Threads’ tab. Copy the name from that tab and paste it here.

5) go to the jconsole > MBean tab, go to tracer >> custom >> configuration >> operations
a) select appropriate threads for tracing.
b) start logging (enableLogging method)

6) invoke the operation/transaction of your application, to be traced.

7) stop the logging from jconsole (enableLogging method in jconsole)

8) Open the method_trace.log file in the current directory.

How Method Tracing works?
It uses dynamic byte code instrumentation. It uses java.lang.instrument.* classes for bytecode instrumentation. It uses ASM Framework for modifying byte code. It modifies the byte code while loading the classes into JVM by Class Loader. The class files in the jar is not being modified.
It inserts tracing code at the beginning and end of each method.

Is it modifies your application binary?
No. It modifies the byte code while loading the classes into JVM by Class Loader.

Is it recommended to use in production?
No. This is only beta quality and there shall be considerable overhead because of method tracing.

Saturday, April 04, 2009

Analyzing Memory Leak in Java Applications using VisualVM

Introduction
Memory Leak is a decease and OutOfMemoryError (OOM) is the symptom for that. However all OOM doesn’t necessarily implies Memory Leak. OOM can happen due to the generation of large number of local variables – particularly with large number of concurrent requests (in the case of server applications). On the other hand all memory leaks not necessarily manifest into OOM – especially in the case of desktop applications or client applications (those are not run for long without restart).
Memory leaks also can be in heap, perm space, or in native memory.
Following section talks only about memory leak in heap. How to verify that memory leak is the reason for OOM in your case is also out of scope (probably will discuss some other day). How to attack memory leaks in perm space is also out of scope (again another day’s work). I have absolutely no clue about debugging memory leak in native memory (Google search with error messages).

Memory leak in heap

1) Run the application with latest available JDK (JDK 1.6 as of now). New versions of java contain lot of improvements in debugging tools. Run the application with different operations repeatedly to identify the operation that causes memory leak. We don’t need to wait until OOM to identify memory leak. Monitoring the heap using any tool can identify memory leak. Run the application until it reaches a steady state. Monitor the heap size over a period of time. If the ‘heap size after full GC’ (full GC remember) increases each time, that implies a memory leak. We can use visualVM also for this purpose.

Now we have identified the operation that causes memory leak – means if we invoke this operation repeatedly we will get OOM. (What if we could not identify the operation that causes memory leak?). Now onwards we assume that this operation runs continuously.

2) Install VisualVM (https://visualvm.dev.java.net/download.html). You need to install VisualVM separately – unlike many other java debugging tools this one doesn’t come along with JDK [ Thanks to Jiri Sedlacek's comment! VisualVM is part of JDK from jdk1.6 update 7 onwards ]. Just unzip the file to any convenient location.

3) Run visualVM - you can run visualvm by executing ‘visualvm.exe in the bin directory. (You should have jdk 1.6 in path?)

4) Connect to the application – all the java processes running in the machine are listed in the left side pane (name of the main class shall be listed). You can double click the one you are interested. Right side there are four tabs – monitor and profiler are the ones we are interested in.

5) Go to the profiler tab. Check the settings checkbox it will display different options. Go to the memory settings tab. Check the ‘record allocations stack traces’. Uncheck the ‘settings’ checkbox; that will close the settings options display (all changes are automatically saved).

6) Now click on the ‘memory’ button – this will start memory profiling. Wait for few seconds – it will take a while. A tabular display of different classes, its instance count, total byte size etc are displayed.
7) Wait for a while to allow the application to be in stead state.
8) Take a snapshot of the objects by clicking the button on top of the tabular view (see below screen shot). A snapshot is generated and labeled in the left side pane as well as opened in the right side pane.

9) Wait for some more time to allow some memory leak. Take one more snapshot as explained in the previous step. Now we have two snapshots displayed at the left side pane (also opened in the right side pane).
10) Select both of these snapshots (in the left side pane, by using ‘ctrl’ button we can select multiple items), right click and select ‘compare’. A comparison tab shall be opened in the right side pane. That tab will display items that have been increased during the interval of first and second snapshot. The top item is the suspect of memory leak.

11) Go to the profiler tab again. Select the identified item in the previous step. Right click and select ‘Take snapshot and show allocation stack traces’. One more snapshot is generated. This time in the right side pane, an allocation stack traces sub tab also visible. It lists different places where this particular item is instantiated and it’s percentage of total count.

12) Wait for some more time to have more memory leak. Take one more snapshot with allocation stack traces.
13) Take a heap dump of the application. We can take heap dump by right clicking the application in the left side pane and select ‘heap dump’. Heap dump is also displayed in the left side pane as well as opened in the right side pane.
13) Now come back to the tow snapshots with stack traces. Compare these two snapshots with allocation stack traces. Identify the methods where there is large difference in contribution to total count. These are the places where leaked objects are instantiated.
14) Go to the heap dump tab in the right side pane. Go to the Classes view. Double click on the item identified in step 10. Instances of that class are displayed. Choose one instance which you feel could one of the leaked objects (just intuition). Look at the right bottom pane. Expand the view. You will find the objects that is holding referenced to the leaked objects. This is the reason for memory leak.

15) From the items identified in step 10, and clue we got from step 13 and 14 we could resolve the memory leak issues. We get the object which leaked, the place of instantiation, and the objects that hold reference to it.

Wednesday, April 01, 2009

Thread wise CPU monitoring for Java process

We come across many occations where java application is hung or slow with high CPU usage. We want to analyze which thread is causing this high CPU usage.
Java 1.5 provides a API "ThreadMXBean.getThreadCPUTime()"
We can use the following code in anywhere in the JVM to get the thread wise CPU usage of that JVM.
ThreadMXBean threads = ManagementFactory.getThreadMXBean();
ThreadInfo[] threadInfos = threads.getThreadInfo( threads.getAllThreadIds() );
for( int i=0; i < threadInfos.length; i++ )
{
long cpuTimeCumulative = threads.getThreadCpuTime( threadInfos[ i ].getThreadId() ); // in nano seconds
}
There is a utility called JTop available with JDK 1.6 (JDK_insall_dir/demo/management/JTop). This can be invoked as a JConsole plugin as well.
You can invoke the JConsole with plugin as follows:
/bin/jConsole.exe -pluginpath
eg: c:/jdk160/bin/jConsole.exe -pluginpath c:/jdk160/demo/management/JTop/JTop.jar
The default JTop shows only the cumulative CPU time (not the current CPU usage as if in Task manager).
I have modified the JTop little bit to include the current CPU usage as well.
You can download the modified JTop.jar from the following path.
http://rejeev.googlepages.com/JTop.jar
This will help us to identify the threads causing high CPU. Typically this will help us to resolve infinite loop issues.

Sunday, February 08, 2009

Emails with inline Images

If you want to send a Email with an image inline, you can do that from Outlook or many other mail clients supports that. However if you send the mail programatically how do you do that?
1) Attach the image with the mail using MIME content type "multipart/related".
2) Set "Content-Id" header (eg Content-Id="image1")
3) In the HTML body of the email, include the image by

Reference: http://java.sun.com/developer/EJTechTips/2004/tt0625.html#1
http://www.ietf.org/rfc/rfc2387.txt

Tuesday, November 18, 2008

Heard of Tell-a-Friend from SocialTwist?

I am posting this using a new service called Tell-a-Friend from SocialTwist. This service lets me announce about a site or a blog to my friends using Email, IM, Blogs and Social Networks. It can be used to spread the word about search engines results or interesting product offers, blog posts, etc.

This is so different from other such services because: