Martins Irbe

how to deal with a leaky system?

Almost every software engineer has encountered a memory leak at least once in their system. A memory leak occurs during the memory allocation management stage, when memory that isn’t needed anymore isn’t being released, therefore after some time when there is no more available memory, the out of memory (OOM) error occurs. This has happened with some of the applications that I have been working on and I would like to share my experiences so far about how I have dealt with patching up memory leaks in systems. I have encountered two different types of memory leaks.

visualvm

VisualVM is a visualisation tool used for profiling purposes. This is the tool that I’m using to determine any memory leaks in the application. Tom some while ago wrote about how to profile a JVM in EC2 with VisualVM, which provides useful steps of how to connect VisualVM to a remote application. Once VisualVM has been connected to the remote application, we can see beautiful graphs displaying the current CPU usage and heap statistics of currently used and available heap size, see image 1.

cpu_heap_graphs

image 1 – CPU usage and heap statistics graphs.

new threads

This type of memory leak happens when new threads are created each time a piece of code is being executed. This particular memory leak isn’t hard to identify. With the help of VisualVM, it’s possible to connect remotely to the affected application and observe the current state of all threads that are being used by the application.

visualvm_threads_view

image 2 – CPU usage and heap statistics graphs.

Image 2 shows an example view of the threads window. This particular view is useful when there is a memory leak that creates a new thread when a specific part of the code is executed. It would be easily identifiable as the number of threads would keep growing and the state of the thread wouldn’t change.

growing thread

This type of memory leak is very hard to fix as it isn’t easy to determine which part of the code is responsible for the memory leak. Therefore, to determine where the memory leak is, you would need to go carefully over your code to identify places where, for example, a new object would be created each time code is executed. In that case the profiler can be useful as it allows the user to see the heap histogram that provides valuable information about which classes are being used the most.

thread dump

To dig deeper, it’s possible to use thread dumps to see the stack trace of each thread, this is useful as it might help to identify where the memory leak is happening. I would suggest downloading and installing a VisualVM plugin called Threads Inspector. This little plugin is very helpful as instead of making huge thread dumps, you have to select a thread of your interest in the threads window to see the stack trace for the particular thread.

It’s important to use profiling software to identify any possible memory leaks in your system before it goes to production. Furthermore, it’s important to patch the memory leak to make the system more reliable by making sure that you won’t get the OOM errors in your system. However it can be a tricky task, especially when dealing with slowly growing threads that burst after a couple of weeks.

If you enjoyed the read, drop us a comment below or share the article, follow us on Twitter or subscribe to our #MetaBeers newsletter. Before you go, grab a PDF of the article, and let us know if it’s time we worked together.

blog comments powered by Disqus