Technology

Truly Smart People Realize There's No Silver Bullet

There’s no silver bullet when it comes to software, hardware, operating systems or complete systems. Fanatics, especially Mac fanatics, will tell you that Mac is that silver bullet, but truly smart people know better. Take this brief blog entry about installing Leopard. That doesn’t sound like a great user experience; not at all. That doesn’t sound like something my Mum would be able to get through without help.

This isn’t a dig on Mac or Apple. I agree that more often than not, things work when dealing with a Mac. But the same is true of Linux. And Windows for that matter. The point is that you can’t judge an entire technology because of one person’s experience. More so when that person is using a laptop with recently released hardware on a beta version of an operating system. If you were so quick to judge, then you certainly wouldn’t ever install Leopard given the number of people who have encountered issues when upgrading.

But I won’t stoop to such hasty judgment because I truly believe in using the right technology for the right job. And to figure out what the right technology is, you need to be unbiased and approach everything with a clarity not found within fanatics.

Noticeable Spam Increase

Several technology news sources have noted in the past month that the amount of spam being sent out has increased dramatically. On my mail server, I didn’t notice any changes until last week when I saw a huge jump in the number of spam emails – from 4 or 5 emails getting through my filter to an average of 20 per day. The emails are actually all the same content and with only slight variations in the subject line and different sender name/email. So I’ve once again dropped my filter score in the hopes of catching more of this crap. It’s a bit of a pain as dropping the score in simply a change to a configuration file but I need to scan my email inbox to ensure that dropping the score won’t trap any emails I actually want to receive.

Also, I’ve now set up my server to automatically remove old emails caught by the spam filter. I’ve set virus emails to be removed after 7 days and other emails to be removed after 14 days. That way any emails which are caught unnecessarily can be recovered. I set up a daily cron job which runs the following script:

#!/bin/sh

cd /var/amavis/quarantine/
find -atime +14 | grep "badh-" | xargs rm
find -atime +14 | grep "spam-" | xargs rm
find -atime +7 | grep "virus-" | xargs rm

Now I just need to find a web-based application that will allow me to preview the emails caught by the spam filter and allow them through if necessary. The Astaro firewall product has a similar feature and it was invaluable.

Sometimes Things "Just Don't Work"

It’s often said that the great thing about Windows-driven machines is that everything just works. And then Mac fanatics take that a step further and note that their machines not only work but also are stable and, in fact, work that much better. And often this is a response to the growing Linux user base. But with all things, things often don’t work not matter what you are using. My wife’s Windows-based laptop 50% of the time is unable to find the wireless network in our house, even if we put the laptop right beside the router.

I mention this because in catching up on some RSS feeds, I read Kris’ entries on his issues with his MacBook: battery problem, then battery problem take #2 and finally, a problem with wireless connections. I mention this not as part of an argument to switch to Linux, but to note that when it comes to computers there really isn’t a silver bullet. But the nice thing with Linux is that if need be I can dive in and muck around with the config files directly or the code if need be.

But my point is that even with non-Linux systems there are headaches, frustrations and millions of users who throw their hands up in the air and just accept the way something is functioning.

Fanaticism As A Lifestyle

When I started using Linux back in the mid-1990s, Macintosh computers were crap. The machines were expensive, the OS was unstable and there weren’t much in the way of applications outside of some stuff for publishers and artists. Windows wasn’t much better but at least the hardware was cheaper and people were building applications like they were going out of style.

I chose Linux because for me, it was the best tool for the job at the time. And as a programmer it was exciting to be a part of this growing, viral movement called open source. And it was even more exciting to help out, whether it was actual coding or tracking down bugs or in the case of the more complex stuff, simply reporting bugs and feeding other developers more information. Sure, at times it was a struggle but the world needs people willing to dig in an help out. The problem with today’s world, no matter what the realm? Everyone wants to take the easy route.

Linux is the near-perfect operating system for me. Everything I need works. Yes, there have been hiccups, issues, and various other struggles. And if I hadn’t been using this particular laptop, Linux would be THE perfect operating system. But I like the portability of a laptop and it’s nice to be able to work on various projects no matter where you are, rather than working on business stuff on a business computer and personal stuff on a home computer. But since I was using a laptop for it’s portability not for it’s suspend/resume, the fact that ACPI-driven sleep modes didn’t work really didn’t matter; it was simply a ’nice to have’. So did I mind? No. Was I able to build a 10 employee business on top of Linux and then sell it? Yes. Was I able to build a handful of personal and side projects on top of Linux? Yes. Do I regret my choice of Linux? Not for a second.

And why would I switch to OSX now? Because some stuff works? I’m sure some of it does, but not everything, as I wrote last year. Every product, whether it’s a car, an aeroplane, a MP3 player or an operating system has it’s good side and it’s bad side. The grass is always greener somewhere else and in the case of Macintosh, that grass looks like every neighbourhood dog has done it’s business on it. No that’s okay, I’ll stick with what’s best for me, not what some Jehovah’s Witness like evangelist tells me I should switch to.

Now, ten years later after starting out with Linux, what’s changed? Well, Macintosh machines are still more expensive. The operating system is more stable, though that’s only because Apple built their OSX ontop of BSD. And although some people are quick to critisize open source software, Linux and BSD are basically fancy kernels with open source software built around them. In fact, Safari is built using KDE’s KHTML code. And the biggest change of all is that under Apple’s executives “great” leadership, Apple has seen it’s Macintosh market share eroded down to a handful of people who buy machines because they’re pretty looking and because Apple makes them. Instead of buying something that suits their needs, they buy something because the product name starts with a styled, italic letter ’i’.

But I won’t tell you to switch to a different OS. What you decide to use should be a personal choice and/or an informed decision based on various factors. And how presumptuous of me to know what you, the reader, might want? That said however, personally, I can’t wait to see all those suckers lay down their money for iCrap.

Three Year Struggle Over

Since owning this Dell Latitude C840, I’ve been in a constant struggle with various aspects of it’s use. I’ve had nvidia driver issues, wireless issues, modem issues, fan issues, docking station issues and external monitor issues. But no issue has driven me crazy more than the fact that I couldn’t suspend the laptop. At one point when I first got the laptop and was running Red Hat Linux I think it would suspend using APM but that’s so long ago I no longer remember if that is truth or fantasy.

But recently this year I started mucking around with suspend again. It was the first time I’d really tried since I last wrote about my troubles. The problem with trouble shooting suspend problems is that it’s very disruptive to doing actual work; if things go bad (which they inevitably did) you end up rebooting and trying again. I spent a few hours trying in early July while on holiday at North Buck Lake without success and given the age of the laptop decided that it was simply time to give up. The machine would suspend and resume just fine now, but on resume the LCD backlight would fail to come back on. And without that backlight, reading the LCD is next to impossible.

But I absolutely hate unsolved problems; it’s a curse for me. If something doesn’t work, I think about it until it gets solved. So this morning, I tried again. I did a bunch of research with Google and discovered that various people had been successful, using xset or vbetool or combination thereof. So I wrote a shell script to run on suspend, which was a mish-mash of various things people had tried, but again no backlight.

That is until, I tried using those tools to first turn off the backlight (via dpms) before the suspend. And, that did the trick. As such, my three year struggle with suspend/resume on my Dell Latitude C840 is over.

Here’s the script I use to suspend the laptop:

#!/bin/sh

#------------------------------------------------------------
# ACPI - suspend script
#  - enhanced by Douglas Robertson
#
# To Do:
#  - track what networks were brought down and bring back up
#  - track what modules were loaded and reload on resume
#  - ???
#------------------------------------------------------------

if which fgconsole > /dev/null 2>&1 ; then
    XHACKS_ORIGINAL_VT=`fgconsole`
else
    XHACKS_ORIGINAL_VT=1
fi

chvt 1
VBEMODE=`vbetool vbemode get`
vbetool dpms suspend


# down with the network!
ifconfig eth0 down
ifconfig eth1 down
ifconfig eth2 down
ifconfig wlan0 down

# remove usb modules
rmmod uhci_hcd
rmmod ehci_hcd

# remove any other modules that are known to wreak havoc
rmmod tg3
rmmod ide-cd
# ???

sync

echo mem > /sys/power/state

sleep 1

modprobe uhci_hcd
modprobe ehci_hcd
modprobe tg3
modprobe ide-cd
modprobe ndiswrapper

chvt $XHACKS_ORIGINAL_VT

vbetool dpms on
vbetool vbemode set $VBEMODE
xset dpms force on

# up with the network!
ifconfig wlan0 up
#ifconfig eth2 up
ifconfig eth1 up
ifconfig eth0 up

It works for me; hopefully it helps those who have also struggled in the quest to suspend/resume with a working backlight. If you have issues you can try contacting me, but I’ll warn you that I’m no expert. Not at all. Though after thre years you think I would be.

Speeding Up Gallery 2

I’ve written a couple posts in the past raving about Gallery 2, an open source, web based photo album organizer. An I’ve imported most of my digital photos and media, bringing the total ‘items’ up to over 16,000. But I have been finding Gallery 2 to be incredibly slow. And I don’t remember if it was always like this or if it’s a result of me adding more and more photos. But this afternoon I decided to get to the root of the problem(s).

I did some quick searching around on Google and discovered some other users have found the app slow, but no solid answers as to why. I didn’t bother posting to the forums myself because I run Gallery on top of PostgreSQL and far too often I’ve been given the flame answer of ‘Oh, well it runs fine for me on MySQL so that’s likely your problem’. And perhaps, in this case, it is partially to do with PostgreSQL. I knew it wasn’t an issue with the server because the load is low and there was no real hit on apache until just before the page rendered. Unfortunately, that left the database. And truth be told, this might also be an issue under MySQL in some cases, but since I use PGSQL, I can only talk about it.

If you’re finding Gallery 2 slow, start by disabling the ‘Image Block’ module. It has some SQL in it that can be the problem. It’s not that the SQL is bad, it’s just that the Image Block module runs a couple of queries that if not using an index will take far longer than necessary. That’s both a PostgreSQL and a Gallery problem. If that works, and you can live without the nice random image you get with the Image Block, then you’re done. If not, then you need to have access to your database so you can do some tweaking.

There are two slow queries, both which should be using an index. Load up PostgreSQL’s psql program, or some other DB admin tool that lets you run queries, and try this:

EXPLAIN SELECT COUNT(*) FROM g2_ImageBlockCacheMap WHERE g2_ImageBlockCacheMap.g_userId = 10;

You should see:

QUERY PLAN
———————————————————————–
Aggregate (cost=11042.91..11042.91 rows=1 width=0)
-> Index Scan using g2_imageblockcachemap_1627 on g2_imageblockcachemap (cost=0.00..11007.43 rows=7095 width=0)
Index Cond: (g_userid = 10)
(3 rows)

The important part is that you see Index Scan and NOT Seq Scan. If you see Seq Scan, then the database is going through the entire table. Twice, in fact, since the second query I mentioned is likely going to give you a similar result. At this point you have two options, and you may want to consider doing both.

First, you can tweak the PostgreSQL configuration options that deal with query plans. I suggest reading up on the options first and doing trials on a non-production database. Especially since a PG restart is required for any changes to take effect. Every database config will be different based on processor power and memory and such, so I leave it up to you to figure out how best to change things for your setup. The key thing is though, that you want to tell PostgreSQL’s planner that whenever possible it should use indexes. And sequential scans should be a last resort, except for small tables.

That done, it still may not be enough. And it wasn’t enough to work for me. I’m not entirely sure why the index on the g2_ImageBlockCacheMap table still wasn’t being used but I suspect that it might have to do with the fact that it’s a tuple. Regardless, all that was required was to add an index and vacuum the DB, as such:

CREATE INDEX g2_imageblockcachemap_idx_extra ON g2_imageblockcachemap(g_userid);
VACUUM ANALYZE;

Then re-run that EXPLAIN to see if the index is used instead. It worked for me. And once again I can use the Image Block module.

30,000 Junk Emails

Today is a big day. In late December 2002, I started collecting junk emails. I’m not quite sure of the actual date nor am I quite sure why I started keeping these junk emails, but for whatever reason I did. And I only kept junk emails that were sent to me, via any of my email accounts (I have several addresses). I never asked for anyone else’s junk nor did I go out of my way to get junk mail. I simply collected the stuff that was sent. And now, three and a half years later, my junk mailbox is full of 30,000 emails which were a waste of my bandwidth and time. Not to mention service providers around the world.

Luckily, however, I’ve been using Mozilla Thunderbird since its inception. And one of Thunderbird’s best features is it’s adaptive junk mail detection. Thunderbird uses Bayesian filtering, which was made popular in part thanks to Paul Graham’s ”A Plan For Spam”. Unlike traditional junk mail filters at the time, which were mostly based on the sender’s email or perhaps on specific words, Bayesian was more mathematical in it’s filtering approach and with proper training could reduce false positives to a miniscule amount. If you’re interested in more information, you should read Paul’s essay but in short, filtering based on the senders email address doesn’t work because the spammers simply make up random names and email addresses. And judging spam based on single words like viagra, for example, doesn’t work because while an advertisement for purchasing the drug is spam, a joke email from a friend about taking viagra is not. Read the original essay though as it’s quite enlightening. And there’s a follow up article also.

Since training Thunderbird as to what I consider junk email, I have yet to have any false positives. Though, to be perfectly honest, I’m so convinced of Thunderbird’s adaptive filter that I no longer check for false positives. So the chance exists that there may be a couple. But I doubt it. I have seen a slight increase in junk emails that don’t get caught recently but I don’t mind because marking them as junk is a single click task. And just like life for me, where my learning never stops, Thunderbird continues to learn too. Even after 30,000 junk emails.

Gallery 2.1

I updated to Gallery 2.1 on one of my Gentoo servers last week. And once again the process was amazingly slick. The Gallery developers have gone out of their way to make the process as seemless as possible. It’s not at the stage where my Mum could install it but given that this is a server-based application, it’s about as close as you can get.

I took the lazy route this time and didn’t bother cleaning up the old files and instead simply copied the new set of files overtop of the old ones. I hate doing this because if there’s any kind of problem with permissions, some new files may not get installed. But not to worry with Gallery 2 because the app not only checks the validity of the files when upgrading, it also checks for old files. At that stage of the install, I was presented with a long list of old files and groaned at the thought of having to delete them all. At least until I read this message on the same page:

These files are no longer part of Gallery. They probably won’t cause any problems but it is a good idea to remove them to keep your install clean. Gallery can’t remove these files for you, but you can download and run this script in your gallery2 directory to delete them for you.

And the word “script” was a link to download a nice shell script which removed all the old files for me. Brilliant. And just when I thought the application couldn’t get any better. So if the Gallery development team are doing this, why can’t more app developers? In most cases it comes down to laziness and the cool factor; it’s just not cool or fun to do that kind of stuff.

Best Web Application Install Ever

Since I started administering web based services of one kind or another back in 1996, I’ve installed many web applications. And in most cases I’ve installed or upgraded many versions over time. And until today, every one of those installs has been a pain at best and a tediously, frustrating struggle at worst. Perhaps I’ve been installing the wrong apps or perhaps I’ve just had bad luck or haven’t followed the instructions (if there even was any). But it seems that’s just the way things are and for a system administrator responsible for web applications, that sucks. That’s not to say I’m not happy with most of the apps I’ve installed; for example, I’ve been very pleased for the most part with this blogg app, though I’ve definitely had some issues.

But I said ‘until today’ because this afternoon I installed Gallery, an open source, web based photo album organizer. In most cases, it’s not the actual install per se that causes the grief but the configuration step of the install process. At best, there is some sort of command line or curses based program (SquirrelMail has this) and at worst, such as is the case with Roller, you have to modify a config file and run the database scripts manually. Not so with Gallery. With Gallery, after I copied the files to the proper location on the web server, I then accessed the website to continue and set the application up via a web based wizard. Gallery prompted me for all the various configuration settings, including the database vendor (in my case, PostgreSQL) and database location and login details. Gallery then created and populated the DB tables and got the application to a point where it was fully functional, all within 10 steps or so. And when something couldn’t be done (perhaps a file needed to be created or have certain permissions), the current page instructed me on what needed to be done and wouldn’t let me proceed until it was. The entire process was fast, easy and very slick.

And because of the easy install, Gallery has restored my faith in open source programmers. Why? Because when developing applications, web based or not, programmers (especially junior and volunteer coders) typically like to avoid anything but coding the meat and potatoes of the application. That means that bug fixing, unit tests, install wizards, etc all get second-tier status and rarely get done. But in Gallery’s case, someone clearly cared somewhere along the line. And I can only hope that their upgrade process, when it comes time for me to do that, is equally as slick.