Calyx blog

I’m amazed at how little we know about the tools we use for our everyday work/leisure. For the past 3/4 days I’ve been amazed with the discovery of the ctrl+shift+T shortcut in Firefox. I’m not aware of when they added it to Firefox, but now I use it all the time!

The shortcut does a simple thing - it reopens the last closed tab. If you press it more than once it will reopen the tab you closed before the last one etc. Since I’m quite a tab juggler while browsing the web (meaning I usually open 7 tabs at once, reorder them, then read them and close them) I often close a tab prematurely. Also, sometimes I realize there’s something I wanted to check on a tab that I just closed.

Before I used to go through the recently closed tabs menu option, or even the history one if I couldn’t find it in the recently closed tabs (and it could happen). Now all I need to do is click 3 buttons and the magic happens! Things like these really make me happy :)

While the previously mentioned shortcut might not be revolutionary and not be considered “important” for one’s daily work, there’s another example - the Eclipse IDE. I’ve been using Eclipse on a daily basis for almost 3 years now. And only a few months ago I became aware of the beautiful shortcuts ctrl+shift+R and ctrl+O. For those of you who aren’t aware of them - ctrl+shif+R opens a wildcard-search window for finding any resource (for example a class file) in your workspace. Similarly ctrl+O opens a wildcard search for the members/fields of the currently open class.

Finding a class in a project was a frustrating task before - I’d wildly scroll through the project tree trying to remember which package the class belongs to. Finding a method inside a class was usually easier, but frustrating nonetheless. Now I feel happy each time I use those shortcuts (which is probably a few hundred times a day). Happy and stupid because I didn’t use them before. I wonder how much time I’ve wasted on searching for classes manually - time which could have been spent in a more productive way. Nowadays I spend a few seconds on what took me a minute or more before - the most basic task during software development - finding the code you’re working at.

The unawareness of the capabilities of the tools we use daily is truly an amazing thing. I recall I tried to find such a functionality in Eclipse, but couldn’t find it in the menus so I assumed it wasn’t there. I even considered writing my own Eclipse plugin to help me find classes. I failed to realize, at the time, that I’m not the only one with that problem and that someone else probably already solved it - what’s more, the solution was already there.

The bottom line would be, I guess, that learning about software development doesn’t start with learning the programming language we use. It starts with learning about the tools we use for coding it - something we can forget and lose valuable time. For starters - try pressing ctrl+shift+L in Eclipse. I’m sure you’ll find something you didn’t know ;)

Let’s just assume that you have some important stuff being done on your server. A computation running, like I do, something getting compiled for a long time, or whatever else you might need. You don’t want to be sitting all the time, sshed in and running tail -f dump.log So you think to yourself: “If only I could have this file posted to a website so I could just check it here and there from my cellphone”. That’s your wish (believe me — I can read minds). Well, I had the same wish a while back and this little script came out of it.

Of course, I like to at least try to code stuff up, instead of just writing to Santa. However, I usually go and try to generalize a bit. Let’s say that I want to monitor a few files or outputs of various commands at the same time. Moreover, let’s say I want to add these things as I go. Enter
remotespy.

Remotespy is a script that parses a list of commands to run and compiles their output into a HTML webpage. The page is then saved to a local folder, e.g. public_html. Additionally, it can be scp-ed to a destination of your choice. This is useful if your “workhorse” server doesn’t have a http server setup, but you do have some other place to post your output to.

The syntax is simple:
remotespy COMMANDFILE HTMLFILE"
The spirit of the command file is similar to the one of crontab scripts: every line represents a command that is run. Syntax of every
line is:
TITLE ## COMMAND ## COMMENT

A sample script looks like this:
Hostname ## hostname
OS ## uname -orpi ## OS version
Disk ## df -h ## Free disk space
Top ## top -b -n 1 -i ## Process state
Vmstat ## vmstat ## Memory state

Every command is run as-is and its output captured and formatted into HTML. The HTML code is hardcoded in the script, but, script being a script, it can be easily edited. To make remotespy into a tool satisfying the initial desire of remote monitoring, it can be paired with cron and ssh to periodically run commands from the script and send the files to a (remote) public_html folder which is available to the internets. For the SCP upload to be useful, you should set up a ssh key pair between the source server (one running remotespy) and destination http server. These two articles can help if you don’t know how to do it yet:

• http://www.linuxjournal.com/article/8600
• http://mindspill.net/computing/linux-notes/ssh-or-scp-without-password.html

The refresh meta tag is also set for the page so you don’t have to refresh the page manually to check if cron ran the command.

Remotespy was written in Python and tested on versions 2.4 - 2.6. I don’t know how far back the compatibility actually extends. I like to use it as it allows me to edit the command script without having to reinsert anything into cron. The HTML output is simple enough to be quickly read from mobile devices so I can check on my simulations state whenever I feel like it. You can find it in my Mercurial repository among other scripts I’ve written:
http://bitbucket.org/mbudisic/cl-goodies/src/tip/remotespy
(The repo name was taken from el-goodies collection for Emacs).

Let me know if you have suggestions for improvements. Future versions should probably migrate from “commands” to “subprocess” Python module. Also, I might make remotespy into a daemon to make it cron-independent, but that’s a low-priority for me. This does the job well. If you would like to extend remotespy, please let me know, I’d love to hear from you. If once wasn’t enough, you can read this post at http://mbudisic.wordpress.com/2009/04/03/monitoring-from-afar/ too.

I’m fairly sure that readers of this blog are acquainted with the folding@home project.

Recently, we installed the F@H app on our 4-core Ubuntu server. In fact, we installed it 4 times, since it seems that’s the only way (we could find) to use all the CPU cores on 32-bit systems. Despite the fact that 16 “FahCore_nn.exe” processes are always running, 12 of them sitting idly, the thing is working. All the cores are used, working units are crunched, we feel good for contributing every time we run the top or ps command :)

And then we started wearing t-shirts in winter.

All the CPUs are working at 100%, ventilators are working on full speed, the office is 10 degrees warmer. We tried to limit the worker processes to some CPU percentage only to find that limits.conf does not support it. Thankfully, Google pointed us to the cpulimit utility.

It worked perfectly. At least until the day after, when some of the folding worker processes were again consuming 100% of CPU. So, on day 2, we learned that there is the “main” F@H process which fires “worker” F@H processes for each working unit. Once the job is done, the process is killed leaving a zombie cpulimit process that’s trying to limit a non-existing process.

Day 3: Perl :)

The only solution we could think of, for limiting dynamically generated processes to a certain amount of CPU percentage, was a periCRONically executed Perl script that kills hanging cpulimit processes and fires new ones. The input parameters are a string for matching the process name and a number indicating the CPU percentage. Works fine, but a hackish taste is left in my mouth.

Any ideas for something smoother? I cannot accept that Perl has to be used for every real-life problem :D

Btw, the perl script can be found here.