Yes - I just tested an H.264 M4V file and works great. I am running Twonky Media Server v4.4.9 for Linux directly on my DNS-323 to my DSM-750 over the wired network connection. No comp necessary. This version had a packaged installation feature which was good for me as I am a Linux dummy. Once it is set-up you don't interface with Linux at all, rather, you use a web-interface to config your options. There might be a newer version - I haven't checked since mine aint broke so to speak.
I've been running for over a year and it is rock solid. I too was looking for a stand alone solution that I could leave with my family and know I wasn't likely going to get a tech support call from my wife when I was at work. I had too many of those with a PC server. I also have it all hooked up to switch over via a Logitech Harmony remote to further simplify.
I think it only cost me $40 for a perpetual license so I deemed it money well spent. I have used WMC, Tversity extensively and like Twonky the best - largely because I don't need a PC in between and it just WORKS! It doesn't have all the flashy bells and whistles but enough of them so that I can enjoy video, music and MP3 on my DSM-750. I am not sure how it handles the sorting of media but I elect to create custom folders as I prefer that.
To run this, you need a Linux capable NAS - like the DNS-323 that has UPnP media serving capability. I would go all D-link as it works and you have one single throat to choke for support.