What happens is that playback totally freezes. With a high quality H264 (1920x1080 at >25MBit) playback freezes almost instantly, with lower quality playback starts and gets some way before freezing. It doesn't seem to restart even if left for several minutes, although the player is not crashed as it responds to the remote immediately.
Actually further experiments show that even at 12.5Mbit H264 will sometimes freeze. I have successfully played MPEG2 at 20Mbit and MP4 at 15Mbit so I don't think the problem is networking and I have used both wired and wireless (my wireless is a DAP-1522 on 5GHz with gigabit all the way from the server to the AP).
By the way, you need to get the word out that these extra containers work. The general word on the forums is that H264 has to be in an AVI file! I'd like to see a table on the web of containers vs supported codecs for the DSM-750.