I have found RiverPast Video Cleaner Pro is good at converting video to something playable on the DSM-520 (like XVid). A lot of people have had success with AviDeMux (http://www.avidemux.org/) which is free.
When using Video Cleaner Pro, I generally extract the audio stream first, convert the video with Video Cleaner Pro, and then put it back together with AVIMux (different then AviDeMux). All videos I have converted this was have worked perfectly on the DSM-520.
Good luck,