Alright...Disregard that last. Its definitely the smartphone. Going into the router's wireless status I see it does connect at 1-5Mbps, whereas everything else does at 70+ usually, so I guess everything is held back by the slowest link.
When the phone is on 3G it doesn't seem to change anything, it only hurts performance when on Wifi.