Calyx blog

I’m amazed at how little we know about the tools we use for our everyday work/leisure. For the past 3/4 days I’ve been amazed with the discovery of the ctrl+shift+T shortcut in Firefox. I’m not aware of when they added it to Firefox, but now I use it all the time!

The shortcut does a simple thing - it reopens the last closed tab. If you press it more than once it will reopen the tab you closed before the last one etc. Since I’m quite a tab juggler while browsing the web (meaning I usually open 7 tabs at once, reorder them, then read them and close them) I often close a tab prematurely. Also, sometimes I realize there’s something I wanted to check on a tab that I just closed.

Before I used to go through the recently closed tabs menu option, or even the history one if I couldn’t find it in the recently closed tabs (and it could happen). Now all I need to do is click 3 buttons and the magic happens! Things like these really make me happy :)

While the previously mentioned shortcut might not be revolutionary and not be considered “important” for one’s daily work, there’s another example - the Eclipse IDE. I’ve been using Eclipse on a daily basis for almost 3 years now. And only a few months ago I became aware of the beautiful shortcuts ctrl+shift+R and ctrl+O. For those of you who aren’t aware of them - ctrl+shif+R opens a wildcard-search window for finding any resource (for example a class file) in your workspace. Similarly ctrl+O opens a wildcard search for the members/fields of the currently open class.

Finding a class in a project was a frustrating task before - I’d wildly scroll through the project tree trying to remember which package the class belongs to. Finding a method inside a class was usually easier, but frustrating nonetheless. Now I feel happy each time I use those shortcuts (which is probably a few hundred times a day). Happy and stupid because I didn’t use them before. I wonder how much time I’ve wasted on searching for classes manually - time which could have been spent in a more productive way. Nowadays I spend a few seconds on what took me a minute or more before - the most basic task during software development - finding the code you’re working at.

The unawareness of the capabilities of the tools we use daily is truly an amazing thing. I recall I tried to find such a functionality in Eclipse, but couldn’t find it in the menus so I assumed it wasn’t there. I even considered writing my own Eclipse plugin to help me find classes. I failed to realize, at the time, that I’m not the only one with that problem and that someone else probably already solved it - what’s more, the solution was already there.

The bottom line would be, I guess, that learning about software development doesn’t start with learning the programming language we use. It starts with learning about the tools we use for coding it - something we can forget and lose valuable time. For starters - try pressing ctrl+shift+L in Eclipse. I’m sure you’ll find something you didn’t know ;)

Let’s just assume that you have some important stuff being done on your server. A computation running, like I do, something getting compiled for a long time, or whatever else you might need. You don’t want to be sitting all the time, sshed in and running tail -f dump.log So you think to yourself: “If only I could have this file posted to a website so I could just check it here and there from my cellphone”. That’s your wish (believe me — I can read minds). Well, I had the same wish a while back and this little script came out of it.

Of course, I like to at least try to code stuff up, instead of just writing to Santa. However, I usually go and try to generalize a bit. Let’s say that I want to monitor a few files or outputs of various commands at the same time. Moreover, let’s say I want to add these things as I go. Enter
remotespy.

Remotespy is a script that parses a list of commands to run and compiles their output into a HTML webpage. The page is then saved to a local folder, e.g. public_html. Additionally, it can be scp-ed to a destination of your choice. This is useful if your “workhorse” server doesn’t have a http server setup, but you do have some other place to post your output to.

The syntax is simple:
remotespy COMMANDFILE HTMLFILE"
The spirit of the command file is similar to the one of crontab scripts: every line represents a command that is run. Syntax of every
line is:
TITLE ## COMMAND ## COMMENT

A sample script looks like this:
Hostname ## hostname
OS ## uname -orpi ## OS version
Disk ## df -h ## Free disk space
Top ## top -b -n 1 -i ## Process state
Vmstat ## vmstat ## Memory state

Every command is run as-is and its output captured and formatted into HTML. The HTML code is hardcoded in the script, but, script being a script, it can be easily edited. To make remotespy into a tool satisfying the initial desire of remote monitoring, it can be paired with cron and ssh to periodically run commands from the script and send the files to a (remote) public_html folder which is available to the internets. For the SCP upload to be useful, you should set up a ssh key pair between the source server (one running remotespy) and destination http server. These two articles can help if you don’t know how to do it yet:

• http://www.linuxjournal.com/article/8600
• http://mindspill.net/computing/linux-notes/ssh-or-scp-without-password.html

The refresh meta tag is also set for the page so you don’t have to refresh the page manually to check if cron ran the command.

Remotespy was written in Python and tested on versions 2.4 - 2.6. I don’t know how far back the compatibility actually extends. I like to use it as it allows me to edit the command script without having to reinsert anything into cron. The HTML output is simple enough to be quickly read from mobile devices so I can check on my simulations state whenever I feel like it. You can find it in my Mercurial repository among other scripts I’ve written:
http://bitbucket.org/mbudisic/cl-goodies/src/tip/remotespy
(The repo name was taken from el-goodies collection for Emacs).

Let me know if you have suggestions for improvements. Future versions should probably migrate from “commands” to “subprocess” Python module. Also, I might make remotespy into a daemon to make it cron-independent, but that’s a low-priority for me. This does the job well. If you would like to extend remotespy, please let me know, I’d love to hear from you. If once wasn’t enough, you can read this post at http://mbudisic.wordpress.com/2009/04/03/monitoring-from-afar/ too.

[caption id="attachment_111" align="aligncenter" width="500" caption="Paul Newline"][/caption]

After inserting the contents of a multiline .NET TextBox into the postgreSQL database, I went to check the results.

Three operating systems, three different representations. Nota bene, we’re talking about newline characters here, not really something exotic.

I know there’s a series of unfortunate historic events that lead to this situation - but you just have to love it :)

p.s. notice the letter “n” from “Wien” missing in the pgadmins :)

Saturday afternoon. Perfect timing for an exhaustive debug session.

So here we are, debugging a relatively straight-forward algorithm that handles objects based on their Date field. Supposedly, a piece of cake.

A few hours later (and 3 mins prior to solving the issue), we notice that a collection of Date objects sometimes contains objects of type TimeStamp.

A quick glimpse on the TimeStamp class javadoc reveals the following:

TimeStamp is a composite of a java.util.Date and a separate nanoseconds value

and

The Timestamp.equals(Object) method never returns true when passed a value of type java.util.Date because the nanos component of a date is unknown.

To put the same statements into something more comprehensible:

Date d = new Date();
Date ts = new Timestamp(d.getTime());
System.out.println(ts.equals(d));	 //returns false
System.out.println(d.equals(ts));	//returns true
System.out.println(ts.compareTo(d)); //return 0
System.out.println(d.compareTo(ts)); //returns 1

Now, it’s not very polite to have two references of type Date, with ‘o1.equals(o2) = true’ and ‘o2.equals(o1) = false’ :) If you have an application that fetches data from the underlying database via ORM (in this scenario we use Hibernate) and from the GUI, chances are you will have this situation.

But the biggest problem has yet to come!

By now, we all learned that TimeStamp extends Date and adds the nanosecond ’support’. OK, let’s accept that. So when implementing the compareTo method, they obviously took care to check if it the argument they’re comparing to was a TimeStamp or a Date. If it was a Date, they created a temporary TimeStamp object from its milliseconds-from-1-1-1970 value (getTime() method), and the output was correct (in the example above, it returns 0). But what about the last example? ‘d.compareTo(ts) = 1′ ?! How can that be? If the Date object has no perception of nanos and the TimeStamp object is created from its getTime() method, how can the Date be *after* the TimeStamp?

By bad design decisions, i dare say :)

In the last paragraph, when i told you about the TimeStamp’s compareTo(Date d) method, I left something out intentionally. Now i can drive the point home :) Here’s the original method implementation (taken from the java.sql.TimeStamp source code):

public int compareTo(java.util.Date o) {
if(o instanceof Timestamp) {
// When Timestamp instance compare it with a Timestamp
// Hence it is basically calling this.compareTo((Timestamp))o);
// Note typecasting is safe because o is instance of Timestamp
return compareTo((Timestamp)o);
} else {
// When Date doing a o.compareTo(this)
// will give wrong results.
Timestamp ts = new Timestamp(o.getTime());
return this.compareTo(ts);
}
}

Notice the comments in the ‘else’ block? Nice!

After some more digging through the TimeStamp and Date implementation, we found the epicenter of this stupidity :) Take a look at the TimeStamp constructor:

public Timestamp(long time) {
super((time/1000)*1000);
nanos = (int)((time%1000) * 1000000);
if (nanos < 0) {
nanos = 1000000000 + nanos;
super.setTime(((time/1000)-1)*1000);
}
}

See the first line? It removes the millisecond part and calls the Date constructor. Everything smaller than a second is kept in the ‘nanos’ field. Very clever. Although it’s not a photon, TimeStamp is of dual nature :)

When calling the compareTo method on a Date object it compares the ‘fastTime’ fields. Unfortunately, TimeStamp changed the original ‘fastTime’ field by subtracting everything that’s smaller than a second and putting that data in the separate field called ‘nanos’. Meanwhile, calling the equals() method will compare the same object by calling their getTime() method which will produce the correct result since the TimeStamp implementation takes care of adding back the ‘nanos’ part to the ‘fastTime’ field.

Sun Microsystems, you’re doing it wrong.

Conclusion: when using TimeStamp and Date objects interchangeably, the preferred way to go is force casting to one of them…

p.s. credits for this post go to !alk and fressner, we all invested a few hours in this post :)

[Update 16/10] Seems we were the only ones not using the Joda Time library