So I was having the same issues with my DNS-325 as you were; I was unable to play my movies on my PS3 because they were .m4v files. My setup for some reason wouldn't even see any files on the NAS.
First of all, let me start off and say this. My machines are both Windows 7 Home Premium 64bit. I have a Motorola SB6580. The NAS is connected directly to the router via CAT5 cable. I have 2 PS3s and it works on both. One is an old 40GB that I have to use a gaming adapter and the other is a new slim with built in WiFi.
I do not know how the original setup from the CD went as I just set things up by going directly to the web management tool. Under the Management tab on the left side Device, make sure your Workgroup is set to the same as your home network. Generally windows Defaults to WORKGROUP but if you changed it, then you need to make sure the NAS is in the same workgroup as you have listed. I went in and created a folder on the NAS. I happened to name it "Videos." Under the management tab and Application Management, I went to UPnP AV Server on the left side of the screen. I enabled that and clicked on the ADD button and selected the "Videos" folder I just created. I copied all my movie files in .m4v format over to the videos directory on my NAS. A movie file in .mv4 is interchangeable with a .mp4 format. You will just have to rename your files.
To rename your files .mp4 instead of .m4v go to your control panel. From there it depends on how you have your views setup. If you have it viewed as Category (Windows Default) then click "Appearance and Personalization" and then "Folder Options." If you have it shown as Small or Large icons, you can just click "Folder Options." From here, click on the "View" tab. Under the advanced settings box, you will find "Hide extensions for known file types." If you uncheck that and click apply/ok, then you will see the file extentions. Go back to your NAS drive and into your folder with movies. Right click on them and click "Rename." Highlight the m4v and change it to mp4. It will ask you "If you change a file name extension, the file might become unusable. Are you sure you want to change it?" Click YES.
Now before my PS3 would see the files (and I'm not really sure why this is) but I had to go back into the Application Management --> UPnP AV Server --> Refresh All. When you go to the PS3 and you search for Media Servers (which by the way make sure you have Media Server turned on in your PS3 settings), you should be able to see and play all your movies. I also have to have the PS3 research for the Media Servers whenever I add a new movie to the folder. Slightly annoying but I got the thing working and that makes me happy.
I hope this helps you out. I know it was long winded but that is how I got mine working properly. Now I just wish I could figure out how to get the iTunes server working. I hear that is an issue with iTunes 10.x though. We'll see.