My webserver is the same machine I use for everything else. So when someone starts downloading 10 files at
once at 200KB/s max per tcp session, my web browsing gets very slow. Unacceptable. With thttpd I could
easily set webserver wide bandwidth throttling, but it's SSI features leave much to be desired. So nginx.

Throttling bandwidth with nginx is hard. Limiting speeds by connection rate and max connections doesn't
really do the job. You either end up with high max connections with each having a low $limit_rate, or only
one or two max connections and a high $limit_rate. With the former single files take forever to download
with some http clients, while lots of cocurrent file downloads will behave 'appropriately'. With the later
single files download fine, but additional cocurrent downloads will go over the bandwidth limit. And if you
avoid that by setting the max connection limit low when pages with many files to download (say, 350+ images)
will take forever or 503.

______________________________________________________________________________________________________________
$ sudo /usr/bin/trickle -u 200 /usr/sbin/nginx # for testing, I made an /etc/init.d/trickle-nginx script too

My first throught was to try the userspace tool, trickle. But a little testing reminded me that the preloaded
library method that it uses won't be inherited by the worker processes and only the master will be rate
limited... and it isn't even sending data. So trickle doesn't work.

______________________________________________________________________________________________________________
$ sudo tc qdisc add dev eth0 root handle 1:0 htb default 10;
$ sudo tc class add dev eth0 parent 1:0 classid 1:10 htb rate 200kbps ceil 350kbps prio 0;
$ sudo iptables -A OUTPUT -t mangle -p tcp --sport 80 -j MARK --set-mark 10;
$ sudo tc filter add dev eth0 parent 1:0 prio 0 protocol ip handle 10 fw flowid 1:10;

I then tried to use the linux traffic control (tc) and iptables for this wherein iptables is marking the tcp
 packet headers so tc knows which to connections to throttle. --sport 80 seems like it'd only match packets 
being sent out by my webserver, but whenever I set these rules and max out the bandwidth with a large test 
file it slows down regular web browsing on the webhost machine too. It works, sure, but it causes the very 
thing I'm trying to avoid by throttling.

______________________________________________________________________________________________________________
Since I couldn't figure out an iptables mangle rule specific enough for just the webserver's outgoing packets
I'm turning to a third option: wondershaper. wondershaper just uses traffic controller (tc) too, but with more
finesse apparently. It won't just be shaping nginx traffic, though, and that's a major downside.