The Lair

Freakishly warm weather yesterday, so I went for a walk around town. This was also an exploratory journey because my local supermarket is closing from the start of next month - and I needed to check out the closest places for fresh veggies and fruit. And other stuff.

So I was walking along - dressed in my overkill-for-20c-autumn-temperatures jacket when I was accosted by a formally dressed guy. Polite, wanted to speak to me about something. Now, ignoring the guy or grinning and saying I’m busy and walking away would have been my normal practice. There are people who seem to be prone to encounters with zealous missionaries - some more so than others - and I happen to be targeted quite frequently whenever I venture out in public. Maybe I have this I-am-a-lost-soul please-save-me look on my face, I dunno. Previous experience with people wandering streets with a book in their hand and wanting to speak to me generally means that what they really want is to sell me something. Even if it doesn’t cost me a penny in cash.

I was, as usual, thinking about something else. I assumed this guy wanted directions to somewhere. In a corner of my brain where all the grey cells had perished, maybe I thought he was a fresher stumbling around lost trying to find the university. Sadly, I didn’t notice a plastic nametag attached to his lapel identifying him as an Elder in a umm.. for want of a better word, sect ?

Read the rest of this entry »

Got back on Tuesday night from a whirlwind trip to ye olde capital city - London. Other than a very interesting dinner at a Turkish restaurant where I gorged myself on baklava and the intriguingly named kayisi tatlisi (sundried apricots filled with cream and stuffed with pistachio nuts), there wasn’t really much to note. Oh darn, almost forgot. I tried Turkish coffee - which is much like ordinary coffee except that filters and strainers are considered redundant according to Turkish custom. Drink too deeply of the coffee and be prepared to cough up coffee grounds for the next couple of days.

My check for open access wifi continued on this trip. I don’t actively try cracking anyone’s encryption, that being both illegal and rather silly - my proof of concept is to hop on an open AP and surf to Google or Yahoo or some other random page once, then leave. Armed with the directory and a few tools of the trade, I tried checking for access points from the 6th floor hotel room. Active APs found: 40+. Unencrypted : just about 10. Most of them belonged to the hotel - and that throws every request into a login page. Wow. No open APs near Bloomsbury. As a colleague noted, it must have been a poor area of town.

Undeterred, I tried again when we moved to the conference venue on Savoy Place. Same result. About 10 APs found, most encrypted. The only open APs belonged to the conference venue, so using those would be uh - cheating. And even those weren’t open in the sense of being freely accessible, but required a username and password for access.

So I’m sitting in one of the conference halls with a colleague, both of us have our notebooks propped on our knees when this guy walks in holding a notebook computer of his own. He sees the two of us working - yes, we were actually working ! - walks over and asks “Oh I say, you wouldn’t happen to have found an access point for wireless, would you ?”. The conference was mostly inhabited by senior management types from various companies. We were the supposed academics in the bunch. When senior management types know about wardriving and hopping onto open APs, well.. that means it’s pretty much mainstream, huh ? Say bye-bye to free wireless internet, folks.

And I finished book 11 of the Wheel of Time series last night. Things happen in this book and several major plot threads are resolved. Most tellingly though, the one before last paragraph of the book has the same phrase that set the theme for Book 6. I suspect a red herring in the use of that phrase, but who knows ? Maybe RJ has abandoned all attempts at subtlety. As with the celebrated Harry Potter series, it’s now possible to see the light, however fleeting, at the end of the tunnel. Considering that the first RJ Wheel of Time book was released in 1990, I’d say it’s about time too. But overall, this book is lots better than book 10 and probably better than book 8 or 9, purely because more story lines are resolved. Click to read some spoilers… If you’re reading this off a feed, uh .. sorry.

Read the rest of this entry »

Why is it that some reviewers (anonymous ones, the cowards!) spend 95% of their allotted review space on hurling potshots at your paper and then terminate with one, measly, stinking sentence at the end ?

Good paper.

Dude (adjust gender as appropriate), don’t jerk me around like that! With each progressive sentence, my litany of insults at this anonymous coward grew ever more audible. I went from muttering “oh, what does he know, that twat” under my breath to a full on roar. “Why that stupid eejit. That was the fricking point, you moron. And you found that point like Columbus fricking found India. Ha!”. I had this guy made. He didn’t get it and he never will. I’m not going to bother wasting my time on his vile comments.

And then he turns my opinion on its head and offers praise. Aargh. Consistency, Senor Reviewer, consistency. If you’re going to diss my (hopefully novel, but probably mediocre) work, do it throughout your review so I can get drunk and call you all sorts of nasty names. Don’t sweeten it with praise! That means I’ll have to listen to you. Aargh. And actually make the fixes that you suggest. Double Aargh.

Yes, so that was mostly written tongue in cheek. The reviewer chap’s comments were insightful, if not always constructive - and he did toss me a huge meaty bone at the end of it, so I can’t complain much, can I ?

Unfortunately, that’s about the only thing that’s gone right today, so it’s been a day of general disaster and higgledy piggledy (I love that phrase!) packing. Not to mention running around imitating a chicken handler who’s discovered one of his flock has the avian flu. Yep, “running around like a headless chicken” is so 90s. It’s all about the avian flu now, peeps.

Ah well. They do say that if you swallow a live toad in the morning, nothing worse is going to happen to you for the rest of the day. I’ve swallowed no toads, yet, but if anything else goes wrong, I’ll consider it - just to make sure nothing else that happens will top it.

Well, I’ve almost finished the first round of tweaking for the application alluded to earlier. It’s my first lifesize encounter with asynchronous Javascript calls to the server (yes, I’ll call it Ajax if you insist) and my impression thus far has been remarkably positive. The Greasemonkey experience taught me lots about DOM manipulation which I’ve put to good use here and astonishingly, the app functions pretty much as I envisioned. And that was actually without sacrifices to the dark altar of browser detection. Ye gods!

Somewhat weirdly though, the otherwise marvellous Opera has some deficiencies with image opacity support - but that’s a minor blip as far as the overall app is concerned. At one point, I feared that Opera would start declining in quality and features compared to other browsers once it became free - but at least as far as the latest snapshot is concerned, this doesn’t appear to be the case. The list of changes include a Firefox/Mozilla style about:config interface, some revamped keyboard shortcuts (I won’t need to redefine them any more, thank heavens) and brilliantly, support for XPath 1.0

Staying with the browser discussion, I’ve also noticed that the Firefox nightly builds have changed the version string from 1.4 to 1.5 in the last couple of days… breaking extension compatibility yet again *sigh*. Well, on the plus side, this indicates that a Firefox 1.5 final release is drawing ever closer. Sadly, there are still several bugs that I can see (the addressbar bug is particularly annoying) - so there could be a release candidate or two after the announced RC1. Some of the more irritating UI bugs don’t even have an assignee yet.

Lots of words have been written about Flock recently - ranging from the gushing “Oh my, this is so cool and the theme is so beautiful” to skepticism about the need for another browser, albeit a specialized Firefox variant. Funnily enough, lots of the truly innovative parts of this new social browser (that term makes the browser sound very outgoing, doesn’t it ? Not like your stay-at-home fuddy duddy Firefox) like, for example, the Shelf is called Places in Mozilla-speak - and the interesting idea of integrating bookmarks with a social bookmarking service was discussed here - both features planned for Firefox 2. Perhaps Flock might never ever be as widely adopted as Firefox is at present - and I’d still think that’s ok. All I really hope that Flock accomplishes is to spur on the mainline Firefox. We don’t have a testbed for experiments with radically different user interfaces and toolsets for browsers - and Flock could provide this - even if they’re discarded or never make it into the mainstream. Remember Firefox was spun off from the main Mozilla and eventually became bigger than its parent ? Who’s to say the same won’t happen with Flock some day ?

Having said that, I do far more things with Firefox than merely bookmark stuff and write blog posts (the length of this post notwithstanding) - so I’m holding onto my bleeding edge Deer Park nightly for the moment.

The axiom that computing seems to live by, at least in this neck of the woods, is that hardware is fast, software is slow. Not an unreasonable axiom, when it comes to that. Hardware is extremely fast at pretty much everything, but it’s also expensive to produce. Doing something cheaply in hardware requires economies of scale, mass production till it hits a price point comparable with software.

Confused yet ? As we know, there are known knowns. There are things we know we know. We also know there are known unknowns. That is to say things we know we don’t know. But there are also unknown unknowns, the ones we don’t know we don’t know. Yes, that’s a famous Donald Rumsfeld quote and I trust it made things much clearer.

One of the biggest surprise discoveries that I made this week is that doing something in hardware doesn’t necessarily imply speed. Like the merits of RedBull, there is more imagery in the advertising than might be found in real life. For the purposes of some research (tangentially relevant to what I’m doing, but I’m always up for some dabbling in a new area), we performed a straight shootout between Lucene and a custom indexer written in VHDL. We knew (or thought) that even with the constraints of VHDL, a simulator environment and a drastically slow clock speed, the hardware version would slaughter the software indexer. Oh, and this is Java, so cue the mockery. (It might be worth noting that this particular piece of software outperforms some widely available C implementations)

As my telegraphed buildup would have doubtless tipped off the more astute among you, the hardware version lost. Not by much, but it still lost. Now, in the world where I come from, calling this a loss for hardware would be harsh. The hardware version was running on a 66Mhz processor. Yes, that’s just two digits. The software was running on a 1.4Ghz processor. All other things being equal (and they were, more or less), it would seem that the hardware was labouring under a massive clock cycle disadvantage. And to have come within 30-40 milliseconds of matching the software ? Phenomenal. But by the arcane rules of the hardware expert, it’s a loss. Back to the drawing board, said he. Off he went, muttering mock curses at smart professional software developers and their silly optimized code.

Actually, I was going to give the ending of this story too… but again, I think most of you can already guess. Clearly, writing a mystery novel or a whodunit isn’t a good career option for me. It was all to do with the width of the average word size. Halve that, and the hardware version has far fewer lookups - and thus performs about twice as fast.

What was my takehome message from all this ? How about “It’s not the size of the chip that matters, it’s the width of the bus“. Yes, that sounds about right. I do innuendo, badly.

Recovering from an interesting (that’s a mild understatement for hellish) week. Had several deadlines that got postponed, then reinstated and above all, I was fighting the wordprocessor more than writing the things I was supposed. Despite a self-proclaimed competence with Word, I was left all at sea when it came to the weird formatting requirements of the document. A word (heh) of advice: don’t try formatting a document into two column format in a hurry. It ain’t gonna work. Actually, it will work, unless there are awkward diagrams which break things up. In which case, prepare to reflow the document from start to finish - and pray. Prayer is essential. I’ve read an assertion about the Word document format being one huge memory dump - and that would explain why some things just plain refuse to work till Word is restarted. More interesting Word war stories here. The bottom line: I should have spent the time making a LaTex class, but I didn’t. That was a bad mistake.

Since then, I’ve been working on a full-on Ajaxification of an existing application that I find quite useful. It’s been educational, to say the least… mostly because I had no idea cross browser DOM support was so far advanced. Consider, I did stuff with Javascript that I only knew for sure worked on Firefox. And it actually performed the same task in IE. And Opera. Opera support was expected but IE ? I nearly fainted with shock. This application is not for public consumption though, so Opera and Firefox are all the browsers I need. Safari support matters to me, not at all.

And almost unnoticed by yours truly, Robert Jordan has released Book 11 of the Wheel of Time series. I honestly didn’t notice, the hype passed me by. Well, now that it’s been released, I’ve started reading the prologue and I’m vaguely cheered. Of course, it could be some vile marketing tactic, but the first 60 odd pages of the prologue have had several Hmmmm moments already so I’ve holding out hope. It’s funny, when I read the utter disappointment that was Book 10 - I blogged about how sickeningly anti-climatic it all was - and got some indignant comments about characters being maneuvered across a continent and how the next book would be good and so on. Despite my deep seated (and justified!) cynicism then, I’d have to say that the comments may (maybe!) have been right.

It’s a brilliantly sunny October day and I was almost tempted to duck into the pub for lunch. And this is still Monday. Thoughts of productivity lost, looming deadlines and other mundane work related activity denied the Charles - even though work is proceeding at a reasonable pace. I usually have miniature crises of this nature because things are being left too close to the deadline for comfort. Unstructured work like writing is harder for me than it really should be … I spend too much time tweaking tiny little experiments and not enough time writing about it - consequently I’m always short of time near a deadline. And this October/November qualifies as one large, 5 week long deadline in itself.

Lots of lost freshers asking for directions to various points. Well, three, which is unusual considering I usually wander around with a distant stare; not exactly the best advertisement for a situationally aware person capable of giving lucid directions. The university does seem more crowded the first few weeks of term - people milling around, wandering hopelessly lost. Usually takes a while before the freshers settle down to a seemingly endless round of pub visits and parties - oh, with a few lectures inbetween.

And I whiled away the best part of last night reading Thud - hence the title of the post. Although there is no plot discussion, I do give away some of the punch lines and references - so you’ll need to click to read the rest.

Read the rest of this entry »

I upgraded from the currently stable Hoary Hedgehog Ubuntu release to the almost ready Breezy Badger release a couple of days ago. Just felt too sickish to write about it till now.

All in all, the upgrade was seamless, with very little fiddling and poking of config files required. One regret I have is my (unfounded) belief that package maintainers know more about my system than I do. Each time I was asked if my configuration file should be overwritten (application_menu, fontconfig, gdm config) I had a quick look at the diff and said yes. Ultimately, this meant I lost all my fancy font tweaks for the laptop LCD screen and gdm threw a warning when I tried logging in. Easily fixable in both instances. I just needed to turn on subpixel rendering again - and gdm threw an error because the new gdm gets annoyed if a custom gdm theme is installed - the factory gdm.conf needs to be used and the theme reset. Not a major inconvenience.

Other than the new and shiny versions of the software that I use, Breezy Badger doesn’t seem to offer me a whole lot that is significantly different on the exterior. Perhaps this is because I adopt a slightly unconventional approach and mix in conventional Debian software, backports and standard Ubuntu style software in one gigantic mishmash. Somewhat more distressingly, nothing packaged by default in Ubuntu can read or understand the battery status on my laptop yet (a huge drawback) and encrypted wireless is still a no-go. Somewhat ironically, this means that when I’m travelling - if I need wireless on Linux, my only option would be to wardrive someone’s open connection; seeing that my legal-to-use WEP encrypted connection isn’t accessible.

And mad props to the Beeb for The Space Race. The final part screened a couple of days ago and I’m waiting for the DVD release. All about America vs the USSR on their race to outer space and the moon, mostly focused on the development of rocket technology and the political infighting. Historical video footage meshed with actors playing the major players on both sides. It was easily the best thing I’ve seen on TV in ages.

And I just couldn’t resist poking in a quiz at the last moment - which discworld character are you ? . (I’m guessing it will be one of Greebo, Vetinari or Cohen the Barbarian. Possibly a smattering of Rincewind thrown in for good measure. And hey, I like bananas, so maybe the Librarian ?)
Update: Ye gods and little fishes. I am Granny Weatherwax, Death and Vetinari combined. All of them equally probable. Sweet!

So they’ve resurrected the lethal 1918 Spanish flu. And it’s now in a laboratory in Atlanta - and the gene sequence is also available online. Normally, I’d be indifferent about this - having more things to worry about than day to day biological breakthroughs and potential Michael Crichton movie scripts. But you see, I think I’m inflicted with the flu right now. And that leaves me less than sympathetic to people trying to foist yet more variants of viral disease on us. If Murphy has anything to say about it though, we’ll all be watching for people to steal this deadly strain of influenza - and then some enterprising terrorist will actually sneak through anthrax or bird flu or something similarly vicious. Or something more mundane, maybe ?.

In other news, the weather forecast is going to sound a lot chirpier and more cheerful. A simplification of weather forecasts is what it’s called. Now, as El Reg already says, the weather forecasts have a tiny credibility problem. When the hawt weather chick blithely announces that “most of the country will be bright and sunny” and the evidence of your own eyes gives the lie to that particular tale - well, that’s when I’ll start mocking the forecasts. Er. Wait. I mock the weather forecasts already. Never mind then. At any rate, the first few forecasts which were supposedly bright and cheerful sounded pretty samey for me. Fog and bad conditions on Yorkshire roads, drive with care, etc. If they were really following orders, they’d have just said “a tiny part of the massive Yorkshire province is bathed in sunshine and this makes for perfect visibility. It’s as warm as the Costa del Sol too. The rest - you don’t want to know about.“

I was all set to do a number crunch to end all number crunches. Fortunately, I didn’t need to expend lots of CPU cycles because a shortcut occured to me. Say no to global warming via wasted CPU cycles!

The problem: I have a set of words, which can be mapped as points on a long linear scale. I have no idea about the absolute position of any single word (hereafter referred to as a point) nor where each point resides on that scale. The only calculation I can perform is the computation of distance between any two given points. My problem is to find the largest possible distance between any two points. As a bonus, I also need to know which two points are the furthest apart.

I’m simplifying the calculations to assume everything is an Euclidean distance apart on the same plane although the reality is probably closer to a 3D arrangement resembling this.

I started working my way down the first word (or point), calculating the distances between every single word in my vector space and this first word. This yields (138354 - 1) values. The problem all along was that computing all the distances for a single word wouldn’t yield the largest value for the entire set; hence my decision to try and brute force a solution.

The answer is to brute force all the distances for a second word. This yields (138354 - 2) values. Since we already have distance(first_word, second_word) it is now possible to calculate where the first word resides in relation to the second word. With the application of some geometry (I know what to do, I just don’t know what it’s called, dammit), we can now discover the maximum distance between any two points in our set of words. The answer can be checked by brute forcing all the distances for a third word (138354 -3 calculations) and applying the same principles to either the (first_word, third_word) or (second_word, third_word) pairs.

I’ve got about 4 blocks of words left to compute. Three machines have been running the “blocks” of words and distance computations non-stop for about a day or so now. Just as well I don’t have to brute the entire 19 billion odd, because the distance calculations are taking far longer than I originally envisaged.

It appears that the answer to life, the universe and everything is not 42, but something closer to 22.06006. Heh.

And this is the first time I’ve ever run any machine’s CPU utilization upto 100%, kept it there for nearly 72 hours (just one machine, the others have been running for less than 24 hours) and not had any problems. I did need to nice the process a bit so I could use the machine for other things though. This is less impressive than it sounds at first glance because the distance calculation is probably more IO bound than CPU bound. Fetching the words inbetween for the distance calculation accounts for more time than actually performing the calculation.

I’m doing something really stupid now. This shouldn’t completely surprise the regular readers of the blog, but this time, I feel I may well have outdone myself.

I am attempting to run a (non-trivial) pairwise computation on 138354 words. Essentially, each word will have a possible (138354 - 1) calculations … That’s, if my math mojo works, factorial 138354 calculations. The only online calculator I could find, Javascript based, was here. That gives up on the number and says “infinity”. Not altogether unsurprising; but then, the Javascript calculator also gives up on the measly number 2000 and says “infinity”. Hmph.

I have three machines (well, 4 - but I want my laptop to be free) to devote to this task. I’ve partitioned the search space out so that I can farm it out in blocks. Right now, I’ve been running the experiment for about half an hour and I’ve processed 400+ pairs. On a single machine, mind you. But at this pace, the heat death of the universe may happen sooner than my calculation - clearly not a good thing if I hope to graduate in this century.

Well, brute force calculations are always the fallback. I’ve now bought myself at least a week where I can dodge requests to write errant pieces of Powerpoint and reports. (“ooh, I’d love to, but my machines are busy, you see” in a tone of pseudo-sincere regret should do nicely, I think). I’m actually trying to calculate the maximum distance between nodes in a large (really large) tree. Now to actually do something (hopefully) clever like a tree walk. Onwards and upwards.

Update (3 hours later): Well, 92k lines processed (from 138k). This is just the first pass. Since this calculation I need to perform is quite dependent on external factors, the number of valid calculations is also very small: currently only 155 valid results from 92k. Incidentally, Google functions as a calculator. Although I haven’t figured out if it does factorial (I suspect not), an approximation of the number of calculations to be performed is the square of 138354. Which is only about 19 billion or so. Hey, that doesn’t sound like a lot, does it ?