Thursday, August 5, 2010

Trickle Your Data

Sending large amounts of data out over a consumer DSL connection can completely saturate your upload bandwidth for extended periods. Recently, I scripted the backup of all the data stored on my NSLU2 to Amazon's S3, and initially I had problems accessing other services online while the backup ran.

The simple network setup at home means that traffic shaping at the firewall is difficult, to say the least. As my NSLU2 runs Debian, I was able to throttle the traffic generated by the backup task using an open source application called trickle. It runs in userspace and doesn't require root privileges, allowing you to shape the network traffic of one or more applications. However, there are two caveats to be aware of when attempting to use trickle:
TCP Only

trickle is only able to shape TCP traffic generated by an application.

No Statically Compiled Binaries

trickle works by temporarily replacing the network functionality provided by the underlying operating system. This means that it will only work with applications that are dynamically linked to shared libraries and don't have all their functionality compiled in statically. It is possible to check whether a particular binary is dynamically linked by using the ldd command. So, for example:

alpha:~# ldd /usr/bin/apt-get |grep => /lib/ (0x40280000)

If the command had not produced any output, it was statically compiled and cannot be used in conjunction with trickle.

Installing trickle on Debian was as easy as typing apt-get install trickle as root. Once installed, you can throttle an application very simply. For example:
trickle -u <upload limit in KB/s> -d <download limit in KB/s> <command>

You can omit either of the two arguments used here, if you just wish to limit the upload or download bandwidth available to an application, and there are further parameters that can be passed to trickle. There is also a trickled daemon that can be used to traffic shape several trickle sessions at a time.

After amending my backup script to make use of trickle, enough upload bandwidth was reserved to allow casual browsing online while it executed.