That is similar to what I have done. I use MKVExtractGUI to extract the audio track out of the MKV file. I then convert the MKV to XVid AVI using RiverPast (video only checked).
I then use AVIMux to take the resulting XVid AVI and the audio tract into a new AVI with both audio and video.
It is a couple steps, but the result is a great looking and sounding AVI that will play on the DSM-520.
I find WMV 9 does not look as good as XVid.
It would be really nice to have MKV & H.264 native support in the DSM-520.
If you need any more info on the converting method I used, ask away.
Kolinb