Usually, the culprits are either some action listeners that are left hanging, or some references in some long forgotten collections (Singletons really are evil sometimes!).

In our case, it was neither. Here are the symptoms: the application does a whole lot of calculation and uses some tree-like data structures to do the math. We feed it with a lot of data and the memory footprint at one moment in time is ~ 700megs (we’ve got ~ 1G of heap). Then we’re done. And it’s not yet freed. But that’s OK, since GC tends to be lazy. The problem occurs when you want to do the same operation twice - the second time time, JVM runs out of memory.

So, after a couple of days of tracing the leak (my first suspicion was that we had some sort of a leak in our tree-like structure), it can all boil down to this small code snippet:

public class MemoryLeakTest {
    public static void main(String[] args) {
        for (int i = 0; i < 1000; i++) {
            JDialog dialog = new JDialog() {
                int[] take4Megs = new int[1000000];
            };
           dialog.setDefaultCloseOperation(JDialog.DISPOSE_ON_CLOSE);
           dialog.setVisible(true);
           dialog.setVisible(false);
        }
    }
}

Here, we create a JDialog instance that has one neat attachment to it - a 4meg worth of empty integer. Set up -Xmx64m for this test case and only after a few runs, you’ll run out of memory. How is it possible that I run out of memory here? There are no collections, all variables are locally scoped - GC should be picking JDialog up after each loop iteration (or at least after a few). However, it seems that this does not occur. The only thing that seems to fix the issue is calling dispose manually like this:

public class MemoryLeakTest {
    public static void main(String[] args) {
        for (int i = 0; i < 1000; i++) {
            JDialog dialog = new JDialog() {
                int[] take4Megs = new int[1000000];
            };
           dialog.setDefaultCloseOperation(JDialog.DISPOSE_ON_CLOSE);
           dialog.setVisible(true);
           dialog.setVisible(false);
           dialog.dispose();
        }
    }
}

Please note that the tested platform is JVM 1.6.0_19, Windows7 Ultimate (it also happens on XP). Here is what the Java documentation says about dispose:

“Releases all of the native screen resources used by this Window, its subcomponents, and all of its owned children. That is, the resources for these Components will be destroyed, any memory they consume will be returned to the OS, and they will be marked as undisplayable.”

Inspecting heap dumps of both cases revealed that if we don’t explicitly call dispose(), that JDialog (actually, MemoryLeakTest$1 instances) instances remain in memory (I guess Swing has some references to them - we obviously don’t have them) and are not garbage collected. However, doing dispose() seems to have cleaned up the system resources and removed those references in Swing because we can’t see a single dialog instance on the heap anymore. The worst thing here is that it seems that this behaviour is not documented in the documentation (dispose() docs don’t say anything about this - see http://java.sun.com/javase/6/docs/api/java/awt/Window.html#dispose() ).

The shortcut does a simple thing - it reopens the last closed tab. If you press it more than once it will reopen the tab you closed before the last one etc. Since I’m quite a tab juggler while browsing the web (meaning I usually open 7 tabs at once, reorder them, then read them and close them) I often close a tab prematurely. Also, sometimes I realize there’s something I wanted to check on a tab that I just closed.

Before I used to go through the recently closed tabs menu option, or even the history one if I couldn’t find it in the recently closed tabs (and it could happen). Now all I need to do is click 3 buttons and the magic happens! Things like these really make me happy :)

While the previously mentioned shortcut might not be revolutionary and not be considered “important” for one’s daily work, there’s another example - the Eclipse IDE. I’ve been using Eclipse on a daily basis for almost 3 years now. And only a few months ago I became aware of the beautiful shortcuts ctrl+shift+R and ctrl+O. For those of you who aren’t aware of them - ctrl+shif+R opens a wildcard-search window for finding any resource (for example a class file) in your workspace. Similarly ctrl+O opens a wildcard search for the members/fields of the currently open class.

Finding a class in a project was a frustrating task before - I’d wildly scroll through the project tree trying to remember which package the class belongs to. Finding a method inside a class was usually easier, but frustrating nonetheless. Now I feel happy each time I use those shortcuts (which is probably a few hundred times a day). Happy and stupid because I didn’t use them before. I wonder how much time I’ve wasted on searching for classes manually - time which could have been spent in a more productive way. Nowadays I spend a few seconds on what took me a minute or more before - the most basic task during software development - finding the code you’re working at.

The unawareness of the capabilities of the tools we use daily is truly an amazing thing. I recall I tried to find such a functionality in Eclipse, but couldn’t find it in the menus so I assumed it wasn’t there. I even considered writing my own Eclipse plugin to help me find classes. I failed to realize, at the time, that I’m not the only one with that problem and that someone else probably already solved it - what’s more, the solution was already there.

The bottom line would be, I guess, that learning about software development doesn’t start with learning the programming language we use. It starts with learning about the tools we use for coding it - something we can forget and lose valuable time. For starters - try pressing ctrl+shift+L in Eclipse. I’m sure you’ll find something you didn’t know ;)

After the process is done, your boss wants some kind of reporting - how many entries were processed, how many were processed sucessful, how many errors were there and which errors they were. Not much of a problem when working on a UNIX machine:

A:    cat out.txt | grep 'COMMAND SUCCESS' | wc -l
B:    cat out.txt | grep 'COMMAND FAILED' | wc -l

… and just to make sure, lets check if A+B = C:

C:    cat in.txt | wc -l

Now, lets report on errors:

cat out.txt | grep 'ERROR_CODE:' | sort | uniq

returns a list of errors:

ERROR_CODE: 10065
ERROR_CODE: 11245
ERROR_CODE: 19543


and now just lets find out how many of each we got:

cat out.txt | grep 'ERROR_CODE: 10065' | wc -l
cat out.txt | grep 'ERROR_CODE: 11245' | wc -l
cat out.txt | grep 'ERROR_CODE: 19543' | wc -l


Email to the boss, and we’re done. All good? Great. But, another email comes in: “Could you please send me a list of all the IDs that caused an error #11245″. Sure, no problem:

cat out.txt | grep -B 7 'ERROR_CODE: 11245' | grep 'REQUEST_ID' | awk '{ print $2; }' | sed 's/REQUEST_ID:\([0-9*]\)/\1/g’ > ids.txt

Lets explain this one a bit:

• the initial request that was sent to the system was logged 7 lines before the ERROR_CODE (therefore the “-B 7″)
• the line with the request had the following format:
  START REQUEST_ID:XXXXX SOME_OTHER_STUFF
  (therefore the awk part)
• with sed we just extracted the number from the request_id column

Can it get any more powerful than this?

Of course, I like to at least try to code stuff up, instead of just writing to Santa. However, I usually go and try to generalize a bit. Let’s say that I want to monitor a few files or outputs of various commands at the same time. Moreover, let’s say I want to add these things as I go. Enter
remotespy.

Remotespy is a script that parses a list of commands to run and compiles their output into a HTML webpage. The page is then saved to a local folder, e.g. public_html. Additionally, it can be scp-ed to a destination of your choice. This is useful if your “workhorse” server doesn’t have a http server setup, but you do have some other place to post your output to.

The syntax is simple:
remotespy COMMANDFILE HTMLFILE"
The spirit of the command file is similar to the one of crontab scripts: every line represents a command that is run. Syntax of every
line is:
TITLE ## COMMAND ## COMMENT

A sample script looks like this:
Hostname ## hostname
OS ## uname -orpi ## OS version
Disk ## df -h ## Free disk space
Top ## top -b -n 1 -i ## Process state
Vmstat ## vmstat ## Memory state

Every command is run as-is and its output captured and formatted into HTML. The HTML code is hardcoded in the script, but, script being a script, it can be easily edited. To make remotespy into a tool satisfying the initial desire of remote monitoring, it can be paired with cron and ssh to periodically run commands from the script and send the files to a (remote) public_html folder which is available to the internets. For the SCP upload to be useful, you should set up a ssh key pair between the source server (one running remotespy) and destination http server. These two articles can help if you don’t know how to do it yet:

• http://www.linuxjournal.com/article/8600
• http://mindspill.net/computing/linux-notes/ssh-or-scp-without-password.html

The refresh meta tag is also set for the page so you don’t have to refresh the page manually to check if cron ran the command.

Remotespy was written in Python and tested on versions 2.4 - 2.6. I don’t know how far back the compatibility actually extends. I like to use it as it allows me to edit the command script without having to reinsert anything into cron. The HTML output is simple enough to be quickly read from mobile devices so I can check on my simulations state whenever I feel like it. You can find it in my Mercurial repository among other scripts I’ve written:
http://bitbucket.org/mbudisic/cl-goodies/src/tip/remotespy
(The repo name was taken from el-goodies collection for Emacs).

Let me know if you have suggestions for improvements. Future versions should probably migrate from “commands” to “subprocess” Python module. Also, I might make remotespy into a daemon to make it cron-independent, but that’s a low-priority for me. This does the job well. If you would like to extend remotespy, please let me know, I’d love to hear from you. If once wasn’t enough, you can read this post at http://mbudisic.wordpress.com/2009/04/03/monitoring-from-afar/ too.

BTW, did you know that the next two suggestions for the company name were InCumulo and FooBar? Whoever can guess the explanation for those two names has a beer ;-)

I could browse the map, draw circles of various sizes and associate tasks to them.

Later, as I enter the designated areas (which is easily determined via GPS or cell tower info), I receive an SMS with the reminder.

Bonus: conditional reminders –> “if(time >= 6pm)” or “if(date >= 16.3.2009. and mySpeed < 10 km/h)”

Recently, we installed the F@H app on our 4-core Ubuntu server. In fact, we installed it 4 times, since it seems that’s the only way (we could find) to use all the CPU cores on 32-bit systems. Despite the fact that 16 “FahCore_nn.exe” processes are always running, 12 of them sitting idly, the thing is working. All the cores are used, working units are crunched, we feel good for contributing every time we run the top or ps command :)

And then we started wearing t-shirts in winter.

All the CPUs are working at 100%, ventilators are working on full speed, the office is 10 degrees warmer. We tried to limit the worker processes to some CPU percentage only to find that limits.conf does not support it. Thankfully, Google pointed us to the cpulimit utility.

It worked perfectly. At least until the day after, when some of the folding worker processes were again consuming 100% of CPU. So, on day 2, we learned that there is the “main” F@H process which fires “worker” F@H processes for each working unit. Once the job is done, the process is killed leaving a zombie cpulimit process that’s trying to limit a non-existing process.

Day 3: Perl :)

The only solution we could think of, for limiting dynamically generated processes to a certain amount of CPU percentage, was a periCRONically executed Perl script that kills hanging cpulimit processes and fires new ones. The input parameters are a string for matching the process name and a number indicating the CPU percentage. Works fine, but a hackish taste is left in my mouth.

Any ideas for something smoother? I cannot accept that Perl has to be used for every real-life problem :D

Btw, the perl script can be found here.

On monday, you could already feel the eagerness of people (well, not really people, developers) coming to the office. But as of today, the eagerness went through the roof - we bought a projector! For code reviews.

The following video is the projector’s version of a developer’s introduction, the beloved “Hello World!”:

Since our dissatisfaction with the e-banking web site we all use (described in a previous post) grows by the day, we decided to tackle the problem ourselves :)

To cut a long story short, our beloved e-banking web app lets you see your transactions for a certain period - and that’s about it. No export to CSV/XLS possibility, no graphic representations.

That’s why we decided to build some kind of add-on which will help you visualize your data. We chose to implement it as a bookmarklet, using the JavaScript plotting library flot. Here’s the result:

[caption id="attachment_136" align="aligncenter" width="480" caption="ZaBaPlot"][/caption]

The functionality is rather limited - you can track your account balance, individual transactions and some kind of moving average. Zooming in/out is also possible. + the killer feature: tooltips with transaction details :)

The source code is yours to do whatever you want.

Installation and usage guide. (WARNING: Croatian language)

Luckily, everything turned out fine in the end. The office is upgraded, no reboot needed. Pablo is now satisfied.

[caption id="attachment_124" align="aligncenter" width="372" caption="Approval, we haz it!"][/caption]

p.s. A quite fascinating thing just happened. Venus aligned with the moon to form something looking extremely similar to the Turkish flag inverted horizontally.

This must be an omen, the One has come! :)