One particularly annoying bug for French users was caused by a wrong string in the translation, which caused some watches to be “stuck” and not show notification balloons. See the release notes for more details on what has changed in this minor release.

A tarball is available, and I will try getting someone to build Ubuntu packages (if anyone wants to volunteer to maintain it, you’re welcome to do so), and, if it’s not too late, get it into Ubuntu 9.10.

As you may have noticed, I have made some subtle changes to the actual contents of the notifications. The notification summary is now a proper summary with the watch name instead of “Specto”, and this means we put less “redundant stuff” in the actual “long description”.

Overall, this whole “conform to notify-osd” issue had a positive side-effect: it kicked my ass into actually doing the content cleanup I had on the back of my mind for quite some time.

When the notifications are not attached to the tray icon anymore, messages like this one become glaringly wrong: their uselessness becomes quite apparent. I therefore filed bug 221.

All these changes are currently available for testing in my personal Specto branch. Translators will need to update some strings (mainly just remove obsolete stuff), so please send your updated translations for them to be included in Specto 0.3 final.

Specto is a free and open-source desktop application that will watch configurable events (such as website updates, emails, file and folder changes, system processes, bazaar branches, etc.) and then trigger notifications.

This is a “release candidate” for the 0.3 release. It is considered stable and feature-complete and is intended for last minute testing and to allow you to send us your translations. If you are proefficient in your native language, please help by translating Specto into your language. It is fun, easy, and you get your name in the translator credits. Instructions are available on this page.

Upgrading to this release is strongly recommended, as it solves a huge amount of issues reported in the 0.2.x series, is more stable, implements new features, is more secure and easier to maintain.

This release contains lots of architectural changes. The watch system is now a plugin system, the watch list has changed, GNOME keyring is now used for encrypting passwords by default (though you can disable it), and the configuration/cache directories are now based on the XDG user directory specification. As such, you will need to import your watch list if upgrading from Specto 0.2.x.

Updated packages for Ubuntu are already available on our download page, and might also appear in Ubuntu’s universe in time for 9.04. If you want to provide packages for your favorite distribution, feel free to contact us.

What is new in this release: see the release notes

We are a volunteer project, free “as in beer” and free “as in GNU GPL v2”, and contributions are very welcome. If you like this software and want to fix annoyances that itch you, we value all the help we can get. You don’t need to be a hacking guru! [see contributing page].

Le COMiC et GULUM prévoient faire un installfest Linux le 17 janvier à l’école Polytechnique (Pavillon Lassonde, local M-2101). Vous pouvez aller vous inscrire dans la liste des participants ou installateurs.

Si le public le veut, je donnerai deux présentations au cours de l’événement, traitant de:

• GNOME Power Manager: pourquoi c’est bien pour toi
• Specto: tour d’horizon, historique et découvertes en matière de gestion d’un projet de logiciel libre

Si jamais y’a d’autres suggestions…

What happened there? Well, for making sure that we are using the same “internal” watch list as the actual watches in ~/.config/specto/watches.list (if I’m not mistaken), Specto loaded the file each time and parsed it. The problem is that this is useful (not even 100% sure about it) only when you manually ask Specto to refresh, not during startup, where the watch list is obviously up to date. What Wout did to fix the problem is create a special parameter that makes those method avoid opening and parsing the watch list all the time on startup.

The question remains, however, if the startup time can be further optimized (with the profiling technique in my previous post), or if 2 seconds really is the bottomline; my understanding of profiling outputs is not deep enough to answer that, so any input would be welcome.

Being able to figure out why Specto takes 21 Mib of RAM would also be a nice thing. If it’s Python + GTK’s fault, well, “sho ga nai, desu yo~”. If we can optimize something in our existing code, however, that would be a problem that we should try to address.

I talked about this problem with Wout when I first found out about it, and looked at various profiling tools (sysprof, oprofile, valgrind…), but could not find anything that was supposed to work with Python directly, let alone anything a mortal could use. Thus, we shrugged, and the problem sat there without further investigation, lacking the tools (and time) for that.

Tonight, I wanted to be the next Federico Mena Quintero, and so I tried it again. My first hypothesis was that Specto surely must be doing way too much work, or doing inefficient I/O in some way. I consider it unacceptable that a simple application like Specto does not launch instantly, instead taking as much time as a warm-start OpenOffice.

If you don’t know about software performance profiling, there is some truly awesome “hecho in Federico” material that you must see first. It is easy to understand and funny as hell:

• http://ftp.belnet.be/mirrors/FOSDEM/2007/FOSDEM2007-ProfilingDesktopApplication.ogg
• http://www.gnome.org/~federico/docs/2005-GNOME-Summit/html/index.html
• http://www.gnome.org/~federico/docs/2006-GUADEC/how-much-faster/index.html

So, tonight, I tried Googling for once, and immediately discovered that Python 2.5 already has three official profiling modules! Aided by the quickstart guide, I found to my great delight that I could simply run cprofile over the entire application (since I’m only interested in the startup anyway), instead of butchering the code and wasting tons of time on getting it working.

The normal command that launches Specto if you use “launch.sh” is like this:

PYTHONPATH=$PYTHONPATH:src ./specto $@

The command I used to do the profiling is this, instead:

PYTHONPATH=$PYTHONPATH:src python -m cProfile -s 'cumulative' specto $@ >profile-cumulative.txt

This launches Specto, and when it finally appears and starts refreshing its watches, I immediately hit the “Stop” button and close Specto with a keyboard shortcut, to profile just the bare minimum I need.

I repeated this command with “cumulative”, “time” and “calls” to change the sorting order of the results (other sorting orders are possible).

The results confirmed (it seems) my hunch: iniparser (the module that reads/writes the watch list) is likely the bottleneck: out of the 10 seconds of run time (running the profiler slows things down a bit), about 7 seconds seems to be eaten by iniparser (cumulative time). Now, I’m no profiling guru, so bear with my sloppy analysis, I may be completely wrong here, but I will highlight the parts that seem to me like they are problematic:

Look at all that crap! It must be doing some serious sh** to be able to run over 213 thousand function calls and hog the CPU (a Pentium 4, 2.4 GHz) for over 7 seconds…

Remember Federico’s law? Trim the fat in this order (after having asked the profiler):

1. Excess work
2. Excess I/O
3. Bad algorithms
4. Bad memory access patterns
5. Micro-optimizations (function call overhead, mis-predicted branches, CPU-level foo)

… and it seems like Specto’s iniparser module is doing a lot of “excess work”. At the same time, why is “time.sleep” eating 1.6 seconds of our startup time? If I recall correctly, we don’t actually use a sleeping timer during startup before the watches start refreshing. This could be another path of investigation.

Now what? Throw iniparser away? I guess not, not until someone sits down and actually investigates (better than what I did here):

1. What to blame: our code, or iniparser? We should look in Specto’s code, to see if there is some dumb/flawed logic, like “open and read the file each time for every watch”.
2. If iniparser is at fault, can it be optimized, or must it be replaced by some other data storage method?

So, this concludes my initial blog post on profiling Specto. I have here identified some potential performance bottlenecks, and I would appreciate any help or feedback on analyzing this issue more accurately. If someone wants to actually tackle the problem, you’re more than welcome. I personally don’t have time, at least not until Christmas holidays, and even then, I can’t promise anything.

The other command I use quite often is “meld .” to get a graphical diff of my current directory, or “bzr gdiff” to get a more textual view.

What is interesting in the screenshot above is that you can see the result of the bazaar bug tracker integration. In this case, it allows doing stuff like:

bzr commit --fixes specto:155 --fixes specto:156

Which then tells bazaar to indicate that these two bugs were fixed by that revision, and link to them. Before you do that, you need to set your bug tracker in ~/.bazaar/bazaar.conf. In my case, it is Google Code Hosting, so I had to enter it as a custom bug tracker, with a line like this:

bugtracker_specto_url = http://code.google.com/p/specto/issues/detail?id={id}

Wout Clymans has kept wearing his Hero Hat over the past year, developing his own branch of Specto and enduring the pain I must have been (as a Quality Assurance freak and project manager).

Well, I can say for certain that his branch is pretty much rock-solid now, as I have been using it exclusively for the past few months without problems, and that it will become Specto 0.3.

Bazaar Woes

Therefore, after months (no kidding) of trying to stitch the mess that we had made with our version control systems (cvs, to svn, to bazaar, without keeping the history between each! for some reason Wout branched specto-woutc off from nothing initially, and it has gone by unnoticed for months, then it was too late), it has finally come to this:

“History be damned, let’s make a bzr branch out of specto-svn and merge everything Wout did over the entire year as one single, gigantic commit.”

Not elegant by any stretch of the imagination. Here’s a hint to bazaar developers: for crap’s sake, if your bazaar rebase/replay plugins can’t handle situations like this because they only care about file IDs and don’t give a rat ass about filenames, you have epic failed. It completely defeats the purpose of the darn thing.

So here’s today’s lesson, kids: make sure to preserve history, because Bazaar’s plugins can’t do anything to pull you out of that crap unless you’re a kernel developer guru it seems.

The near future

Specto 0.3 is way overdue (in my mental agenda). I am somewhat ashamed to say that I haven’t touched a single line of code in the past year, all the great work was done by Wout. Actually I did write exactly one line of code, it’s a patch sitting in my Frontbringer stuff to fix issue #1. I will apply it to specto-main soonish, hopefully.

Specto-woutc has been mostly ready for months, but the clusterfuck (as LinuxHater likes to say) that was our bazaar branches kept me unmotivated to do anything until it was solved. So that was one reason why I was not involved much except as a management and support role (bug triaging, QA testing on Wout’s branch, etc), the other two reasons being that Specto has now reached its initial role (it fulfills my daily needs) and that university keeps me way too busy.

Enough blah blah, what now? Well, since this bazaar mess is cleaned, I will feel a bit more confident in doing commits or branching. So what I plan to do is:

1. check for last minute regressions in the user interface experience compared to the 0.2.x series
2. search for any strings in the code that could be translated but are not yet marked as translatable
3. update the translation template
4. translate into French (as a benchmark)
5. email all previous translators requesting help and testing
6. maybe publish a 0.3-beta or release candidate? Anyone knows if some folks would be interested in testing it thoroughly?

Anyone is welcome to give feedback on the new specto-main branch.

Damn. that means I should stop being lazy/being a good student, and get a 0.3 out the door soon (by poking at Wout’s awesome code).

Or at least squash the silly watch list corruption bug that has been plagueing me for years and don’t know how to fix!

The trick is just to add a web watch and point it to that document’s RSS feed. The drawback to this approach, however, is that it requires you to publish that document. I made a short screencast (demonstration video) of that technical prowess ;) if you have other tricks and cool stuff that you can do with Specto (or google docs + Specto), let me know.