The Internet in Numbers
I’m not one to usually re-post videos, but this is a good one, and on topic 
JESS3 / The State of The Internet from JESS3 on Vimeo.
Java Weather API
I’ve also been spending quite a lot of time with a friend developing a Java Weather API, which is now almost nearing completion (85th revision and counting). I’ve been using the API as part of a larger system that monitors climate conditions in certain locations during the course of a year, collating the results which will later be used to generate graphs and other statistics.
The API is pretty simple, but certainly effective, and makes use of Google’s own weather API that’s provided as a SOAP service.
The API parses the XML feed given by Google, and constructs Weather objects that can then further be manipulated.
Here’s a simple example of how the API could be used:
try
{
Weather weather = WeatherStation.getWeather("wirral");
}
catch(WeatherStationException wse)
{} // Bad!
weather.getConditions(); //Returns a string, such as "Fog", "Partly Cloudy"
weather.getTemperature(); //Default is to return result in degree celsius
As with my last post, we do fully intend to make these APIs available publicly at some point in the near future.
Java API for monitoring websites
I’ve spent the past couple of hours completing a first revision for a Java API that makes use of the excellent HtmlUnit libraries to enable monitoring of a web servers response time.
Here’s a simple example of how I’d implement the API:
String[] siteUrls = {"http://www.bingo.com",
"http://www.microsoft.com",
"http://www.google.com"};
for (String url : siteUrls)
{
Website test = new Website(url);
System.out.print("Site: ");
System.out.println(test.getSiteUrl());
for (int i = 0; i < 2; i++) {
System.out.println("\tTest " + i + ":");
System.out.println("\t\tCreated: " + test.getCreateDate());
System.out.println("\t\tResponse time: " + test.getResponseTime() + "ms");
System.out.println("\t\tTime of last refresh: " + test.getLastRefreshed());
}
System.out.println();
System.out.println("");
}
Which gives:
Site: http://www.bingo.com
Test 1:
Created: Tue Mar 02 20:29:29 GMT 2010
Response time: 149ms
Time of last refresh: Tue Mar 02 20:29:30 GMT 2010
Test 2:
Created: Tue Mar 02 20:29:29 GMT 2010
Response time: 100ms
Time of last refresh: Tue Mar 02 20:29:31 GMT 2010
Site: http://www.microsoft.com
Test 1:
Created: Tue Mar 02 20:29:31 GMT 2010
Response time: 507ms
Time of last refresh: Tue Mar 02 20:29:33 GMT 2010
Test 2:
Created: Tue Mar 02 20:29:31 GMT 2010
Response time: 361ms
Time of last refresh: Tue Mar 02 20:29:34 GMT 2010
Site: http://www.google.com
Test 1:
Created: Tue Mar 02 20:29:34 GMT 2010
Response time: 86ms
Time of last refresh: Tue Mar 02 20:29:35 GMT 2010
Test 2:
Created: Tue Mar 02 20:29:34 GMT 2010
Response time: 85ms
Time of last refresh: Tue Mar 02 20:29:35 GMT 2010
The idea is that I’ll use this API in a bigger project I’m working on, a simple OpenSource website monitoring/reporting suite.
Still a lot of work to do, leave a comment if you’d like to take a closer look at the API. I do intend to make a public SVN server at some point and start porting over some of these little projects.
Use a swap file instead!
I’ve only recently stumbled across this, but you can now (and probably have been able to since god knows what kernel 
) use a swap file instead of having to create a swap partition. There’s no performance impact, and it’s easy to setup/configure.
Enter the following at the CLI:
#> dd if=/dev/zero of=/opt/512MB.swap bs=1M count=512
#> mkswap /opt/512MB.swap
#> swapon /opt/512MB.swap
Obviously you can put the file wherever you want. The real benefit of utilising a swap file rather than a swap partition may already be obvious: if you want to increase your swap size, simply swapoff, create a new, bigger file – and swapon again.
Quick BASH tips
If like me you use history extensively, then you’ve probably been annoyed at the fact that it forgets that complicated command from six months previous due to its size limitations.
Increasing the size of your history is an easy change, simply edit ~/.bashrc and insert the following line:
export HISTSIZE=10000
(Set HISTSIZE=0 if you want to disable the history all together).
As is already documented over at the ‘Linux Quick Tips‘ section, perhaps one of the most useful BASH shortcuts is the CTRL-R command, that allows you to search through the command history.
Also, it’s probably a good idea to stop duplicate lines from being entered one after another and taking needless space, so uncomment the following line to leave it as shown:
export HISTCONTROL=ignoredups
If you’re privacy conscious – you can change that line to read:
export HISTCONTROL=ignoreboth
That (and indeed having HISTCONTROL=ignorespace) will mean that any lines starting with a space will be ignored.
Whilst you’re in that file, you can also fix one of my pet hates when using BASH on other people’s servers – auto-completion features for programs that can use them, i.e. apt-get. If you’re logged in as a normal user, auto-completion for apt-get et al is already enabled. But for root, it isn’t.
To enable auto-completion, scroll to the bottom of the file and uncomment the appropriate section so it looks like the following:
if [ -f /etc/bash_completion ]; then
. /etc/bash_completion
fi
Finally, once you’ve saved and quitted vi/nano/vim/pico etc, to apply all the changes in your current shell, you can respawn bash by doing:
#> bash
#>
Cyrus Squat File Problem
If you’re getting the following (non-)error in your log files (/var/log/mail.log), then there’s a quick way to fix it if you’re not interested in ignoring it:
Feb 21 13:57:53 zivi cyrus/imaps[4648]: SQUAT failed to open index file
Feb 21 13:57:53 zivi cyrus/imaps[4648]: SQUAT failed
The error is caused by a user doing a search within a mailbox, and a squatter file (aka a search index) is not available. Fix this by running the following as your cyrus user (typically ‘cyrus’):
#> squatter -r user.username
If you’re doing a tail -f of /var/log/mail.log you’ll see the indexes being created.
Et voila – no more errors:
Feb 21 14:10:11 zivi cyrus/imaps[5073]: SQUAT returned 277 messages
If you want to take it a step further, you can automate this by creating a cronjob. Or if you’re using a standard Cyrus installation, the last few lines of your /etc/cyrus.conf will have some useful settings commented out that will allow Cyrus to schedule the SQUAT operations automatically:
# reindex changed mailboxes (fulltext) approximately every other hour
#squatter_1 cmd=”/usr/bin/nice -n 19 /usr/sbin/squatter -s” period=120
# reindex all mailboxes (fulltext) daily
#squatter_a cmd=”/usr/sbin/squatter” at=0517
As hinted at the beginning of this entry, this is a sort of non-error in as much as it can be safely ignored, and squat indexes need only be created if their is an advantage to doing so (that is, searches are often carried out).
Apache and YSlow Optimisation
I’ve been working on a website for the past few weeks, and as I’m reaching the final stages I’ve started to look at loading times and performance etc. The site uses quite a lot of jQuery and other AJAX libraries, but I was still shocked when I realised that over 500kB was being used just in Javascript!
Whilst the vast majority of the population are probably now on broadband, even I can’t forgive the homepage of this website totalling almost a 1MB download when images and the rafts of CSS are included.
So I’ve been making good use of YSlow, the excellent performance debugger produced by Yahoo, which runs within Firefox (as part of the excellent Firebug suite).
My initial score was as bad as I thought, with the index page receiving a ‘Grade E’. My first port of call was to fix the GZip compression on my Apache box, which involved putting the following in a .htaccess file in the root web directory (/var/www on my Ubuntu server).
# Below uses mod_deflate to compress text files. Never compress binary files.
AddOutputFilterByType DEFLATE text/html text/plain text/css text/javascript text/xml image/svg+xml application/javascript application/x-javascript application/atom_xml application/rss+xml application/xml ap$
# Properly handle old browsers that do not support compression
BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch ^Mozilla/4\.0[678] no-gzip
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
# Explicitly exclude binary files from compression just in case
SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary
SetEnvIfNoCase Request_URI \.(?:exe|t?gz|zip|bz2|sit|rar)$ no-gzip dont-vary
SetEnvIfNoCase Request_URI \.pdf$ no-gzip dont-vary
SetEnvIfNoCase Request_URI \.avi$ no-gzip dont-vary
SetEnvIfNoCase Request_URI \.mov$ no-gzip dont-vary
SetEnvIfNoCase Request_URI \.mp3$ no-gzip dont-vary
SetEnvIfNoCase Request_URI \.mp4$ no-gzip dont-vary
SetEnvIfNoCase Request_URI \.rm$ no-gzip dont-vary
YSlow also complained that none of the page elements had expiry headers. Expiry headers are used by the browser in order know how long to cache an object for.
First of all, enable mod_expires:
#> a2enmod expires
And then put the following in your httpd.conf (/etc/apache2/httpd.conf):
ExpiresActive on
ExpiresDefault "access plus 1 year"
You should be able to see that the above sets the expiry headers for the most common file formats, and sets them for one year in the future. You can modify this value as per your requirements, but be warned that YSlow complains if it isn’t “too far in the future”.
Finally, YSlow was complaining about ETags. ETags are explained thoroughly here, which can be summarised with: the majority of browsers use the “Last modified” header tag to validate components, and as such, ETags are largely redundant and as such they’re best turned off.
To do so, first enable mod_headers:
#> a2enmod headers
And then place the following in your site config file (i.e. /etc/apache2/sites-enabled/sitename):
ServerAdmin lm@nothingbutreboots.com
ServerName nothingbutreboots.com
ServerAlias www.nothingbutreboots.com
DocumentRoot /var/www
Options FollowSymLinks
AllowOverride All
Header unset Etag
FileETag none
Finally, all that was left to do was follow the remaining tips offered by YSlow, namely merge all the JScripts into one file, the same for CSS. And then run all the images through the excellent Smush.It.
Et voila! Grade: A 
Google now offers there own performance analysis tool: Google Page Speed. Like YSlow, it tags on to the Firebug backend. Initial reports are mixed, but there’s a helpful comparison of both tools here.
Synergy @ symbol/sign problem
For anyone who uses Synergy as much as me, you’re probably sad to see that it is no longer maintained, which means bugs that have for long gone unanswered will never get fixed unless you start looking at the code.
Alas, a new fork has been created – synergy+, which can be found over at their Google code project page.
Anyway, if like me you’re still using Synergy2 to share your keyboard and mouse between your Windows and Linux systems, you’ve probably come across the bug whereby the @ symbol on your Linux system (presuming Windows is your ‘server’) is replaced by the Greek Omega letter (Ω). A bug report exists over at the Launchpad site for Synergy2 detailing this problem, and thankfully somebody has posted a quick fix/hack for it.
Simply execute the following at the CLI:
echo keycode 24 = q Q at at at at | xmodmap -
Quick Wireless Survey in Windows
I recently needed the MAC address for another AP in my house, and after downloading and installing NetStumbler, was dismayed to find that for some reason the Atheros 11n chipset in my MBP isn’t supported.
A quick scout around and I found this command, which makes use of the excellent netsh command within Windows (XP onwards only I think, and the syntax/parameters seem to change every release – this was tested in Windows 7 and is known working in Vista…):
netsh wlan show networks mode=bssid
Et voila! You should have a nicely detailed list of all the APs in your vicinity. Nice to not have to use a third party app for something simple within Windows!
Hacking the O2 Wireless Box II
After finally getting my hands on the now famous Linksys WRT54GL router, and flashing it with DD-WRT, I was left with a small problem: I had ADSL, not Cable, and I didn’t have an ADSL modem to hand, not even under my bed. No problem I thought, I’d just use my Netgear DG834G flashed with the DGTeam firmware and use that in bridge/modem mode, and so I did. At this point clouds started forming over head and my worst fears were confirmed, the Gods truly where conspiring against me, for the DG834G was only sync’ing at 18Mbps/812Kbps. [/drama]
Using the O2 Wireless Box II (if I ever have to mention the model number of a router again it will be too soon) I had previously achieved 22Mbps/1.2Mbps – and for any self-respecting geek that 4Mbps matters; a shortfall of that magnitude could do serious damage to my reputation on the touring circuit. After some quick research it was confirmed: the supposed ‘ADSL2+’ compatible chipset in the Netgear isn’t up to the task.
So I had only one option left – hack the O2 box. After half an hour or so of digging, I finally found an article that detailed the process (without the use of an .ini file, which I was hoping to avoid), and documented it well. I’ll run through the steps here in case that forum or that particular post ever goes ‘bye-bye’.