I'm betting on the switch, of course there could be some flaw in the DNS-323 network adapter as well. I have run my DNS-323 and DNS-321 at gigabit with jumbo frames for some time, and I haven't seen any networking issues.
I DID have the speeds suddenly fall way down, and it turned out the switch had suddenly forgotten how to handle jumbo frames when it got warm! I also had a total failure with another switch, a couple of the ports on the switch died.
I'd have to try swapping ports on the switch first, then the whole switch next.