I have used a DIR 655 for over a year with no VoIP issues. Having worked for a VoIP startup, I have used this router successfully with numerous SIP termination providers (when running a local side B2BUA or Proxy), as well as with several different local SIP devices registered to various SIP providers and systems (Asterisk, SER, various SIP providers) -- other than the normal and infrequent transient issues related to BW exhaust, have had no issues. I used one Cisco 72XX phone (registered to a remote proxy in our data center) nearly daily for months with no issues. Also used this setup with Skype, Google Voice, etc with no issues.
You may just have a bad unit or the Firmware is different than what I am using. What Firmware are you on?