Plugging into the switch is a better choice in my house.  The router ports are not GBit.
Thanks for posting your specs.  Unfortunately I cannot lend any further assistance since I assume you would prefer good help.  I run a RAID1 and like you have a basic NAS setup.  I have NFS disabled.
For some reason, and it could be a false memory, but I thought I saw folks having problems with NFS and SMB running at the same time.  I'm probably wrong.  I'm sure that since I added this statement, someone will chime in and lay down some facts.
Also, did you format in EXT2 or EXT3.  Doubt it makes a difference but you never know.
Good luck...
-Joe
P.S. 5 people on this NAS?  is it working fast enough?