There's a lot of possibilities here and it could be multiple things.
Is this a new setup, or did it start happening recently?
When the network slows way down, can you still access the web admin page at full speed? (http://192.168.0.1)
Does the ping time to the router change?
When I had bizarre delays like this, I also tried accessing the cable modem's web page to see what happens and pinged the modem through the router. If you can access your cable modem that might give some useful feedback as well.
That's what I did anyway to narrow down the chances that it might be the router instead of the network. Others may have additional ideas.
Just curious, what speed business cable speed did you get? Is anyone running a torrent? All PCs have anti-virus software installed?