Grumpy Old Programmer

It’s a long time since I last wrote about Lotus Notes and the unlimited joy that is its, er, idiosyncratic interface, not least since IBM’s decision¹ to host the client in Eclipse.

Too long, really – it’s such a rich source of oddness. For example, today I recevied an invitation to join some colleagues (located in Frankfurt) in a video conference. The heading in the message informed me that:

Which seemed a little odd, since a quick phone call earlier had seen some time on Wednesday morning identified as the preferred time. No matter, I opened the message to accept the invitation (it’s not clear why I couldn’t do that from the inbox/preview, but I can’t). Clicking “Accept” put the meeting into my calendar:

Whoops! Well, it was more in keeping with what I expected. To settle myself, I went for a coffee (you can tell Christmas is coming, btw: Starbucks are using the Red Cups). When I got back, Notes had been busy – the Inbox message now had this:

OK, it’s now accurate, but I’m not sure how I feel about a message being modified in any way after I’ve opened and read it.

Of course, we’re 0.5 of a release behind the current version, so maybe stuff like this has been fixed by now.

The BBC reports that, in an Australian Science magazine article, an Australian psychology expert “who has been studying emotions has found being grumpy makes us think more clearly”.

To which I can only say “hmph”.

(stackoverflow rep: 7284, Project Euler 83/252 complete)

My main work PC was upgraded to IE7 yesterday. That’s one less IE6-infected machine to worry about. Unrelated to that (I suppose) is that the Aventail VPN product that I have to use each day decided it wanted to upgrade. I’m still trying to figure out how to make that work on IE7 but fortunately I also have an older machine that seems to have been immune to the upgrade, so I switched to that.

After some back-and-forth, I saw the happy news that this was happening:

All going according to plan?

While this was cogitating, a message popped up, partially obscured by the progress dialog. So I moved it. The dialog, that is, not the message. And I saw this:

Dude, where's my progress bar?

How confused must the developer of this part of the installer have been to have built the progress bar as an entirely separate window? And how much more difficult must it have been to do it that way? I amused myself dragging the main dialog all over my desktop while the progress bar stayed resolutely where it was until the install completed.

Sometimes I write legacy code. There, I’ve said it. The secret’s out, the dirty laundry’s aired and the cat’s out of Pandora’s box.

I don’t think it makes me a bad person. I understand the value of tests, especially in the highly incremental development world I currently inhabit, and I strive to use tests to drive my code. Sometimes it doesn’t happen, for many reasons, none of which I’m proud of, such as my Excel test framework being a little clumsy, there not being anyone around me to nag, the “quick” change that turns into just a little more uncovered code than I expected (but it works and the user needs it Right Now).

I understand that all that code sitting in the wild in an uncovered state is Legacy and represents an accumulation of Technical Debt that will have to be repaid. By me.

It’s not that code without tests is necessarily bad. I mean, heck, even Kent Beck sometimes flies without a safety net. If it’s write-once-and-throw-away code, then there’s an argument for just getting it done. But honestly, how often does that code get resurrected months, maybe years later? It’s when code needs to be changed, often requiring some refactoring in the process, that the absence of that warm, covered-by-tests feeling starts to be felt.

Still there, still doing a job, but just try changing it...

The trouble with beginning to repay technical debt is that interest tends to accrue, compounded, at some arbitrary (but almost always positive) rate. So no matter how trivial the original untest-covered change seems to have been, the longer you leave it, the more unpleasantness tends to have accreted around it by the time you come back to it. I suppose there’s always the possibility that you’ll come back to discover a glistening pearl, but in my programming life I’ve never returned to anything other than a thick coating of rust.

Worse than that, getting the encrusted nodule of code under test usually turns out to be painful: it’s seldom structured as it would have been had tests been used in the first place, so unwanted and hard-to-separate interdependencies are rife and the whole thing becomes, well, a bit tricky.

I’d been meaning to buy Michael Feathers‘ “Working Effectively with Legacy Code” for years and recently got around to buying it. Scott Hanselman’s interview last week with Mr Feathers was a serendipitous bonus. The book, even for non-Java or C++ programmers, is excellent, full of practical advice on how to break things up in a way that can let you get tests around the locus of change. In fact, when you see how just plain nasty working with C++ code can be, working with code in any other language starts to look like a breeze.

I wish I’d read the book earlier.

(stackoverflow rep: 3856, Project Euler 63/235 complete)

Good grief. I wrote the first draft of this about a month ago, planning on completing and posting it when the code was done. I expected that to take a few more days. A little more work required on the estimating front, then.

I’m starting to go off Oracle.

Let me put that into context a little. I first encountered Oracle some time in 1998, when version 5 was all the rage. I’d actually taught data analysis, third and fifth normal form, stuff like that for a few years previously but actual hands-on table creation had to wait. Strange but true. Anyway, over the next two or three years, some of which I spent as “Technical Architect” for the investment bank where I worked, I got to be something of a whiz with both version 5 and the swanky new version 6. Heck, I know the query optimiser’s rules off by heart. I’m not just blowing my own trumpet, mind: when I was untimately (fortuitous typo retained) laid off, I was offered a job by Oracle, which I rejected because I didn’t want to take a pay cut.

I spent five more years in Oracle-land with another bank before drifting into the realms of Sybase in its early MS SQL Server guise, and then Sybase itself across three jobs and four years (it seems like longer). Now, fourteen years after we parted company, Oracle and I are back together.

But we’ve both changed. I no longer code in COBOL and have acquired a pathological dislike of business logic in the database. Oracle has a cost-based optimiser, loves to grab all your business rules (more processors = more revenue) and has become a fat bloated porcine creation. Even the free “personal” 10g Express Edition for Windows is a 165MB download.  (OK, SQL Server Express 2008 is even larger, I checked). When running, the thing takes out a 642MB virtual machine. OK, it’s almost entirely swapped out, but still.

How we did parallel processing in the old days

But Oracle is still a helluva fast platform. Unoptimised I was seeing about 8K inserts a minute on my development PC, three times that on a real server. Unfortunately our db server currently lives abroad for tax reasons (or something) and the network latency is fierce. About 900 inserts a minute fierce. So I needed to batch up my inserts or enter the living hell that is SQL Loader.

In order to get multiple insert processes working within my Ruby On Rails-based code, I split each file into several fragments, then run a separate process on each fragment. This takes a bit of doing, generating lots of CMD files that run ruby scripts with “START [/WAIT] CMD /C whatever_I_want_goes_here“.

My file-splitting code, I thought, was rather spiffy – it needs to put the headings from the original to each fragment (because they’re used to figure out what’s in the file) then it starts dealing out the records:

def create_fragment_files(paths)
  File.open(file_path, 'r') do |fin|
    hdgs = fin.readline.chomp
      files = paths.map { |path| File.open(path, 'w+') }
      files.each { |fout| fout.puts hdgs }
      fin.each_line do |line|
        files.first.puts line
        files.push files.shift # the first shall be last...
      end
    files.each { |fout| fout.close }
  end
end

There are faster ways, I’m sure – I could calculate the “ideal” file size and dump records into a file until it’s reached, but this is fast enough (well under a minute for an 85MB file) and it pleases me.

There’s a handy little library, ar-extensions, that makes batching of inserts possible within ActiveRecord (which is the default data mapping library within Rails). It works nicely with MySQL, but turned out to have the Oracle code stubbed and invalid. It only took me a day or two to find a solution to that problem, although I still haven’t figured out how to push an update through a proxy server to github. Finally a chance to do something open sourceful, and I’m thwarted at every turn.

So all in all, it’s taken a month. OK, a month in which a lot of other stuff got done, but still.On the plus side, I just fired it up and I’m watching about 36,000 inserts a minute go through. It’ll be faster when the lookup tables are fully populated. (Another day on, and I’m looking at it: 46,000 – and I still have a few tricks up my sleeve)

While the nearly-two years’ of data is backfilling I now get to rewrite the front end.

And the point of this post? In no small part, to remind me of what I actually spent the lion’s share of the last month doing. Also, to record my first-ever open-source contribution, even if I still haven’t worked out how to get my source out into the open.

If you have been, thanks for your forebearance.

(stackoverflow rep: 2906, Project Euler 59/230 complete)

Below (lightly edited) is a recent answer to a question on StackOverflow. The question is pretty much a waste of time, but what an answer!

Laziness is indeed the first of the Three Programming Virtues, but it is misunderstood. Programming Perl defines it well:

* LAZINESS: The quality that makes you go to great effort to reduce overall energy expenditure.

Good programming calls for laziness, but laziness requires hard work. Good programmers must constantly think of and implement new ways to be lazy. The first compiler had to be written in assembly, and the first assembler had to be written in machine language. Wonderfully lazy, but hard. You don’t get to call it a day after an hour just because what used to take a day takes an hour.

That’s about as close to an encapsulation of my personal programming philosophy as I’ve ever seen. The virtue of Laziness is closely related, if not identical to the DRY (Don’t Repeat Yourself) Principle as espoused by the Pragmatic Programmers in The Pragmatic Programmer.

(I just found myself wondering if there’s a way to refactor that last sentence to remove some of the duplication)

Laziness - all I have to do is this...

The idea behind xlUnit (which has been languishing somewhat since being codeplexed in, good grief, September) was to minimise the amount of manual repetition involved in testing Excel/VBA code as it is developed. As a useful side-effect, it also made TDD (Test-Driven Design, or Development, depending on your choice of definition) possible. A fair amount of the code complexity of the framework is involved in eliminating repetitive manual tasks such as test class creation.

There are only three significant UI-level entry points in xlUnit as it stands: application creation (used rarely, only about once per app) class creation, used rather more often, and test execution, which is used all the time. Oh, and there’s a little “Options” dialog, but it only has one setting and that’s not really very interesting so we’ll gloss over that.

Since the first two things are used relatively infrequently, I tucked them away on a menu, whereas the test execution, which is an all-the-time thing, gets a toolbar button. I could have been fancier with that, but I never got past the simple text “Run Tests”, which had the extra benefit of giving a larger target area for the mouse.

...and I get this for free.

There is no interface within the VBE, which is most definitely a shortcoming. To be honest, I only even tried briefly (failing, obviously) to get it to work: with a dual screen setup, which my working environment has had for the best part of a decade, both workbook and VBA are usually visible, so I can get to the “Run” button easily enough. For a similar reason, there is no built-in keyboard shortcut at present. Of course, in Excel 2007, it’s all a bit of a mess. I have the (massive) RibbonX book, by the way, I just need the intestinal fortitude to sit down and read it.

The “Create Testable Application” and “Create Testable Class” routines both live in the add-in itself and are located within UserAccessibleEntryPoints.bas. I’ll come back to “Run Tests” another time – not only was it a bit tricky but I have some new ideas that would make it even trickier.

Choosing to create an application causes a throwaway instance of the xlUnitCodeBuilder class to create a workbook and its tester and create references from the tester to both the application (so it can call classes there) and the framework (so it can use the framework). The same class contains the code to create new classes. I think I put the functions together because they’re about building code but I don’t think I’ve achieved very good separation of concerns here: there’s workbook creation and test class creation and they would probably be better off being separated. Since there’s work to do with Excel 2007 anyway, I think I’ll refactor that bit next time it’s open.

I mentioned “throwaway instance” above without explanation: it’s a way I describe this VB pattern:

    With New UsefulClassThatDoesntReallyNeedToBeDimmed
        .DoSomethingProfound
        .DoSomethingSimilarlyDeep "AndMeaningful"
    End With

Is there a more “standard” way to describe this? Something using “anonymous” perhaps? Anyway, I use it a lot (and I’m now fervently hoping that no-one points out some awful risk that I’m running as a consequence) as it’s a way to get part of the economy of class (“static” methods in Java or C#, where the word means something rather different than in VBA) method.

Here’s another piece of idiocy from Lotus Notes. I should say that I found this in version 6.5 and in the version that has now been inflicted on me, 8.0.1, it has been fixed. But I won’t let that stop me, it’s still a marvellous example of an entire development team failing to make that one last simple connection:

Could you just for once actually do it, could you?

Have you spotted the idiocy? Notes has recognised that I have the post (sort of) open in the Preview Pane (much as in Outlook) and it doesn’t feel that it can delete it, perhaps because it might leave a decision about what to do with the empty space. So it closes the preview and invites us to try the delete again. Duh.

Just in case anyone’s struggling, I’ll spell it out: the program tells us it will do something we may not have known it could do (automatically close the Preview Pane) and then tells us to manually repeat an action that we know it definitely can do: delete a post. It says “document”, which might offer a clue, but I’m a user here – these are just emails as far as I’m concerned.

In the time it took to code the “problem” identification and the explanatory dialog, the developer could have just deleted the damn post.

Software like this does wonders for my self-esteem, I tell you.

Browsers++, eh? Google have launched their browser, in beta form at least. Of course, “beta” for Google doesn’t always mean what it means for others – is gmail still in beta, by the way?

Anyway, ever ready to while away half an hour of work time looking at something new, off I went to the download page. A smallish (475KB) bootstrapper pulled down the actual installer, managing to find the necessary information about our somewhat complex Monte-Carlo proxy-server load-balancing  script without grief (presumably by digging into the IE or Firefox connection settings) and ran. Pretty pain-free, apart from this:

Getting warm, getting warmer, oops!

Ah well, it is a beta, after all. And it appears that the crash may have occurred at the run-after-install bit, since by the time it happened I had a desktop icon that seems – touch wood – to work.

A little detail that I really appreciated was that the install option page included a setting to make Chrome my default browser but it was unchecked by default. Nice one.

And then it just mostly worked. Some minor issues with font sizes, which seemed to randomly apply changes across tabs when I zoomed in or out using Control +?- or Control-mousewheel, but otherwise my regular stuff all seemed to render pretty well, internal or external.

It appears that the Chrome rendering engine shares the same standards book as Firefox’s – both render our IE-specific corporate intranet home page with the same set of “errors”. I tried looking to see what in the CSS was causing the problem but my limited skills weren’t up to the task. But while I was searching the source, which on a right-click/”View Source” request opens in a new browser window, which is nice, I discovered something nice. Nothing earth-shattering, but nice. I hit Control-F, which did what I expected, typed a few characters and the page was scrolled to the first found instance. As expected. Then I noticed something.

Score points for attention to detail

See what they did? No? Look at the vertical scroll bar. That’s a really nice touch. I like the way the “what to find” box organically grows from the surround too, and the animation is smooth, too. I suppose they could have made it slightly bouncy, in the way that Flash apps seem to like to work these days, although that can make one a little nauseous when over-done.

Oh, and another little plus on the view-source-in-the-browser thing is that links to, for example, stylesheets, are navigable. That removes a tiny piece of Firefox excise that I didn’t previously even know existed.

I’m sure there are all sorts of other little things. The address-bar within each tab may prove to be a boon, and the process-per-tab thing could be useful, although I can’t say crashing ranks very highly on my list of browser annoyances. We’ll have to keep sucking it to see.

I won’t be deleting Chrome. Neither will it be elevated to the status of default browser in the short term – I’m far too fond of my little set of FF add-ins. When Chrome has features that give me the capability provided by, at least, AdBlock, Firebug and Greasemonkey then we may be in business. But I think it’s going to be something of an uphill struggle until something really compelling and unique is offered. The thing is, Windows users who cared have already switched from IE to (mostly) Firefox and I don’t see a reason, other than possibly the bleeding-else coolness, to change again.

At least, I don’t see the compelling reason to switch yet.

A recent post by Mike Gunderloy on Web Worker Daily had me thinking about the keyboard vs mouse debate that simmers gently in the background most of the time, punctuated by bubbles of gas rising to the top, exploding moistly into splatters of mild controversy.

Look: a keyboard shortcut!

Let’s face it, who hasn’t had to rein themselves in when they see a colleague (or worse, a loved one) taking their hands from the keyboard to mouse to the “File” menu and search for “Save”, when a simple “Control-S” would have got the job done? You may safely put me on the guilty side of that line.

Software, where possible, should provide support for both forms of user interaction. It’s usually the keyboard snobs who get upset, seldom the mousers, which may be indicative of some fundamental personality trait…

Now I get as annoyed as the next keyboard snob when someone takes their hands from the keys, ignoring a perfectly good keystroke combo, to reach for the mouse to get to a menu item. I think that at least sometimes I am right – the person should know the efficient way to get the result because detailed knowledge of the package is something that is expected of them in the performance of their job. In those situations I let them know. Sometimes even diplomatically.

As often though, the individual is an occasional or inexperienced user who doesn’t need to know the ways of the expert and frankly doesn’t care – the program is a means to an end and the second or two of excise is insignificant.

Microsoft will probably move this in the minute or two after I post, but right now at the very moment I’m typing this, there’s a whole bunch of shortcuts for Office if you go here.

As may well become apparent over time, I’m not too enthusiastic about Lotus Notes. It’s fair to say that, had I remembered that my current employer is a corporate Notes user, I would have tried to negotiate a higher salary by way of compensation.

Be that as it may, for a developer Notes is a glorious nugget-laden river of blogging opportunities. The client software on my work PC was recently upgraded to version 8.0.1, which seems pretty current. I think, therefore, that it’s fair game.

Let’s look at one aspect that I noticed and delve a bit. In a mail folder, there are columns, as you would expect, some of which use small graphics to provide information. It’s a common UI metaphor. These are the column headings I see:

Lotus Notes message list headers

Nothing surprising there, I think we’d all agree. Look at the paperclip toward the right side. What would you expect it to denote? If you muttered “attachment” then award yourself a small non-monetary prize.

A slightly harder question, now: what would you expect to see when a message has an attachment? A paperclip? Another prize. Moving swiftly on, and for the hat-trick, what do you expect to see if you hover your mouse pointer over such a paperclip icon? Something like “1 attachment(s)” or “114K attachment enclosed” or some such? Whoops, hit the bar.

What you get is this:

WTF?

…which, I trust we can all agree, is exactly what it is. Perfectly accurate and utterly useless.

What can we glean, in a sort of software-archaeological sense, from this? I’m thinking we can make the following predictions, in vaguely ascending order of likelihood:

1. Within the Notes team at IBM there’s a standard that says non-textual elements should have tool-tips (or whatever the Eclipse/Java name for such things is).
2. If there was a formal specification for this element of the program, it was incomplete. Or worse, the spec’s author thought “Paperclip Icon” was appropriate and nobody thought to question it.
3. QA within Notes is non-existent or pitifully understaffed.
4. The Notes development team is woefully inexperienced and/or has very little interest in delivering a quality product, choosing instead to ship a (bad) copy of Outlook (which has its own flaws, let’s not deny it) to institutions who have yet to acquire the corporate gumption to obliterate it.
5. The Notes team don’t eat their own dogfood, for email at least. Let’s face it, who would?

Which is quite a lot for one funny little tooltip.