Windows media server transcodes files on the fly, so should be able to play pretty much any video file that plays correctly on your PC.
The DNS 320 on the other hand doesn't really have to horsepower or codecs needed work this way, it simply streams the file itself over the network, so unfortunately it really depends on which formats your blu-ray player supports...
I could be totally off with this, but that's my understanding of how they work...
If your blu-ray player has a usb port, it might be worth sticking a selection of files onto a USB stick and playing them from that as a test - if they don't work that way, then you shouldn't really expect them to work over the network without first being transcoded.
Probably not the answer you were hoping for... Sorry!