I guess there's a chance your bittorrent activity is choking your connection. But then you have a beefy line so that seems unlikely.
I wrote a little test for myself so that I can see exactly what the state of my connection is at any given time.
If you download this:
http://www.matusiak.eu/numerodix/blog/wp-content/uploads/networktest.sh
Then in the shell "source" the code (it's bash):
$ source networktest.sh
And now you can run the network test:
$ havenet
If you want to have this command around all the time you can do
$ echo "source ~/path/to/networktest.sh" >> ~/.bashrc