Trickle Your Data

Sending large amounts of data out over a consumer DSL connection can completely saturate your upload bandwidth for extended periods. Recently, I scripted the backup of all the data stored on my NSLU2 to Amazon's S3, and initially I had problems accessing other services online while the backup ran.

The simple network setup at home means that traffic shaping at the firewall is difficult, to say the least. As my NSLU2 runs Debian, I was able to throttle the traffic generated by the backup task using an open source application called trickle. It runs in userspace and doesn't require root privileges, allowing you to shape the network traffic of one or more applications. However, there are two caveats to be aware of when attempting to use trickle:

TCP Only

trickle is only able to shape TCP traffic generated by an application.

No Statically Compiled Binaries

trickle works by temporarily replacing the network functionality provided by the underlying operating system. This means that it will only work with applications that are dynamically linked to shared libraries and don't have all their functionality compiled in statically. It is possible to check whether a particular binary is dynamically linked by using the ldd command. So, for example:

alpha:~# ldd /usr/bin/apt-get |grep libc.so
libc.so.6 => /lib/libc.so.6 (0x40280000)
alpha:~#

If the command had not produced any output, it was statically compiled and cannot be used in conjunction with trickle.

Installing trickle on Debian was as easy as typing apt-get install trickle as root. Once installed, you can throttle an application very simply. For example:
trickle -u <upload limit in KB/s> -d <download limit in KB/s> <command>

You can omit either of the two arguments used here, if you just wish to limit the upload or download bandwidth available to an application, and there are further parameters that can be passed to trickle. There is also a trickled daemon that can be used to traffic shape several trickle sessions at a time.

After amending my backup script to make use of trickle, enough upload bandwidth was reserved to allow casual browsing online while it executed.

MySQL: Too Many Open Files

Recently, I experienced an issue with a MySQL instance running on a CentOS 5.4 host whereby applications were unable to contact the daemon. This was despite there being being no network issues and the operating system not reporting any excessive load. Restarting MySQL fixed the issue and allowed the applications to continue working. While investigating the root cause of the problem, I found that the /var/log/mysqld.log file contained the following messages:

100804 4:43:00 [ERROR] /usr/libexec/mysqld: Can't open file: './test_db/test_table.frm' (errno: 24)
100804 4:43:00 [ERROR] /usr/libexec/mysqld: Can't open file: './test_db/test_table.frm' (errno: 24)
100804 4:43:03 [ERROR] Error in accept: Too many open files

In order to prevent the problem occurring again, I increased the maximum number of file descriptors available to MySQL. First, I increased the maximum number of open file descriptors available to the mysql user account by adding the following two lines to the file /etc/security/limits.conf:

mysql soft nofile 8192
mysql hard nofile 8192

Next, I modified the configuration of the MySQL instance to explicitly set the maximum number of open files allowed by simply adding the following line to the /etc/my.cnf file:

open-files-limit=8192

This configuration directive has a default value of 0, which means MySQL will attempt to allocate enough file descriptors itself. However, as I knew exactly how many open files where available to the MySQL user (8192), I could safely use this as the value for open-files-limit.
A restart of MySQL applied the changes and resolved the issue.