hi,
im assuming you have the av server (advanced -> upnp av server) enabled on your dns323.
make sure you click on the refresh button within this page. this will manually refresh the files, may take awhile depending on the number and size of files in your selected folder - im using the root as my folder for av stuff.
no idea how to set up the xbox but the ps3 is straight forward. in the settings, you need to enable media streaming/server. cant remember off the top of my head right now, but there is an option in the ps3 xmb settings to switch this to enabled. once that is enabled, you should see the dns323 or click search for media servers.
if you have already done that, and still having troubles then i suspect it is something to do with your network configuration. i tend to get dlna errors all the time on the ps3 but it rarely affects playback of files.