Well, after a year and a half, I think I've finally got the constant BSoD's I get when playing a game with my nVidia GTX 570 fixed.
First, I bit the bullet and used MSI Afterburner to underclock it to 650 MHz. I may not need to keep it that low, but I still got lockups with 690.
I also added a registry key. Via Mike's Technology and Finance Blog, you can set a key at HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\GraphicsDrivers called TdrLevel.
One of the complaints with Windows (or really any other operating system) is that the screen freezes from time to time. If the screen freezes for more than a few seconds, users are likely to hard reset the machine that they are working on. This seems natural, but in this case the system is still responsive. The graphics processing unit (GPU) is busy processing something (possibly a game, 3D render, or even Windows Aero) and is not actively refreshing the screen.
In Windows Vista SP1 and Windows Server 2008 SP1 Microsoft introduced a feature to help catch and correct this behavior using a feature called "Timeout Detection and Recovery (TDR)." The TDR feature works to identify whether the graphics processor is hung (the default timeout is 2 seconds), and if it is, it prepares to reset the graphics processor and the relevant part of the graphics stack. During this process, it tells the driver not to access the hardware or memory and gives it a short time for currently running threads to leave the driver. If the threads do not leave within the timeout, then the system bug checks with 0x116 VIDEO_TDR_FAILURE. The system can also bug check with VIDEO_TDR_FAILURE if a number of TDR events occur in a short period of time (the default is 5 TDRs in 1 minute). If the TDR is successful, then the user may receive a bubble that says "Display driver stopped responding and has recovered."
TdrLevel should be a REG_DWORD. I set it to 0, to disable checking for TDR entirely.
I'm not sure if that helped or not; I think the underclocking was the more important step (as when I set TdrLevel to 0 but didn't underclock, I still got a lockup). But TDR certainly sounds like something that matches my symptoms, as the lockups usually occur with graphical and audio sputtering -- indeed, sometimes I don't get a blue screen at all, the game just sputters to the point of unusability and the system becomes unresponsive.
At any rate, I'm cautiously optimistic; it looks like I've finally got this thing under control and can actually play games under Windows without constant crashes. I didn't notice any performance hit, either, but then again it's not like I'm trying to run Crysis 2. Walking Dead works fine with its settings maxed out, but you don't need a GTX 570 for that.
Now if I could only get OSX running stably with a 64-bit kernel.