I noticed on my DIR-655 rev. A4 a similar issue. When I went to check how fast the wifi clients were connected at I found clients connected at puny speeds like 1Mbps, 5Mbsp, 18Mbps, etc.
I did three things:
1) I set the router to use wireless g only (when the problem was evident it was set to g + n).
2) Clients were generally connecting at 54Mbps now but in some areas they would still drop to very low speeds. I then did a wifi heat map of the house using http://www.ekahau.com/products/heatmapper/overview.html and found the area where we used wifi the most was getting a poor wifi signal.
3) I had an old Netgear WGR614 v9 router lying around and in the area with the poor wifi signal we set that up as a wifi repeater and now all the wifi clients are connecting at 54Mbps or at worst 48Mbps. It isn't stellar speeds compared to newer routers that can do 300 or even 450Mbps but I don't notice any issues now, everything is much more snappy. Since we only have one client in the house that can even do wifi N I'm happy with getting 54Mbps.
You might consider doing a heatmap and possibly you are having a similar issue.
Also, you might grab netstumbler and install that on a laptop and check what other wifi networks are nearby and what channel they are using. Possibly your router is also using a heavily used channel in your neighborhood and you can move to a less congested channel in the router settings.