Is the peer to peer transfer enabled, and is it working well? Else, well, you'll see a warning on the downloader and a link to an help page.
If it's working fine but still slow, it's because the peer to peer part of the download system is screwed by the fact that you're like among the only 5 people on the planet who decided to download old patch X today
Try to find the patch you need
on wowwiki (Use at your own risk, even if it's usually decently safe. Just try to choose a website you trust a bit in the list).