The issue occurs when I connect devices to my network. I have, for example, seen network failures when I connect my phone while streaming on my Roku. Turn off the phone's WiFi and it resolves itself. Phone, right? Nope. Power up my TouchPad and it does the same thing. Sometimes, the Roku is off, and it happens when my girlfriend's notebook is connected and I connect my notebook.
Even if it were the presence of routers nearby, the 802.11 standard allows for that. It even allows for a lot of them nearby, and .11n was written specifically with coexistence in mind (hence why it detects the presence of 20MHz APs and doesn't use 40MHz features around them).
I still have not had time to pin down the issue. The fact that .11g works perfectly fine tells me that it's something in either the chip or the firmware covering .11n functionality, and it needs to get fixed.