I am not a Mac user, so cannot offer any first-hand advice, but the first couple of paragraphs of this page may help somewhat:
howto:mac_osx_users - DNS323Wiki
if you are not quite sure what you are supposed to do, please consider the following:
there are no straightforward procedures given to undo the stuff described in this wiki. it is important because some tricks work at once and some do not. some need just minor tinkering, some may make you tear your hair out, grow it back and then tear it out again (good example of this is the 'poor man's time machine' hack. my advice: don't bother).
restoring factory defaults does not help a bit if you have invaded too deep into your dns-323's inner workings.
the same goes for firmware reflashing/updating and drives formatting.
best help you can get is from a friend who has extensive linux experience. to get help here you need to be well articulated describing your problems. people here are nice and willing to help if they can.
overall, if you do not enjoy fixing things you have broken yourself, i would recommend installing better media server (i'm really happy with twonky), transmission torrent client and leaving the rest be.