About KB936971
A few years ago, back in the days of Visual Studio 2005, many users did complain about an nasty bug by which some modal dialogs allowed could not be closed under certain circumstances.
Due to a bug, VS IDE will end up not allowing you to close it (nor doing any other task), if you accidentally clicked outside the scope of a modal dialog window.
The easiest way to reproduce this bug is trying to add a reference, by References->Add Reference, and then, click somewhere else on ide's main window (like at the code window just behind the newly appeared modal dialog).
If you are affeced by this bug, once you clicked outside's modal dialog space, you wont be able to click any more on modal's dialog space, nor will you be able to close it.
At this point, if you try to close VS it will show up an error window saying "Microsoft Visual Studio cannot shut down because a modal dialog is active".. Having as only option killing devenv.exe process, maybe loosing you work..
After a while, Microsoft released a hotfix addressing this issue, which if you are using VS2k5 you can find here.
KB936971 On Visual Studio 2008
The funny thing about this subject is that, ever since I installed Visual Studio 2008 I've been suffering this same issue.. I've been used to this issue, and so I tend to be very carefull when platying with modal dialogs within VS..
But today, dunno why, I've hitted the problem twice.. Which made me really mad.. mad enough to find a final solution.. ;)
After a few minutes of reading KB936971, it is clear that the problem lied on the msenv.dll used by VS 2k5.. And the hotfix released by microsoft upgraded such dll from to version 8.0.50727.879.
Knowing that VS2k8 depends on some components which depend on VS2k5's runtime.. I suspected from having an msenv.dll older than the one especified at KB936971.. So I found all over my harddisk for any msenv.dll named file.
The result gave me the following files:
- c:\Program Files\Common Files\Microsoft Shared\Help 8\msenv.dll
- c:\Program Files\Common Files\Microsoft Shared\Help 9\msenv.dll
- c:\Program Files\Common Files\Microsoft Shared\OFFICE12\VS Runtime\MSENV.DLL
- c:\Program Files\Common Files\Microsoft Shared\VSA\9.0\VsaEnv\msenv.dll
- c:\Program Files\Microsoft SQL Server\90\Tools\Binn\VSShell\Common7\IDE\msenv.dll
- c:\Program Files\Microsoft Visual Studio 8\Common7\IDE\msenv.dll
- c:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE\msenv.dll
- c:\Program Files\Common Files\Microsoft Shared\Help 8\msenv.dll
- c:\Program Files\Microsoft SQL Server\90\Tools\Binn\VSShell\Common7\IDE\msenv.dll
- c:\Program Files\Microsoft Visual Studio 8\Common7\IDE\msenv.dll
Ok.. at this point I knew the cause of my madness..
Howto Apply KB936971's hotfix on VS 2008
Once I knew that this dll was the one causing my problems I tried (obviusly) to apply the recomended hotfix (VS80sp1-KB936971-X86-ENU)..
However, such hotfix didnt want to install on my computer as the software it was designed for (VS2k5) was not installed on my computer either.. (obiosuly also..)
Well.. that's fine.. I'll have to find the way to manually apply this hotfix..
Extracting VS80sp1-KB936971-X86-ENU.exe contents..
First we need to extract the contained MSP file at VS80sp1-KB936971-X86-ENU.exe. This is easily done by executing:
VS80sp1-KB936971-X86-ENU.exe /extract OutDir
This will create a folder named "OutDir" which will contain the MSI Path file "VS80sp1-KB936971-X86-ENU.msp".
Extracting VS80sp1-KB936971-X86-ENU.msp contents..
Having worked with MSI/MSP files in the past, I already know this usefull utility called "msix" which will allow you to extract VS80sp1-KB936971-X86-ENU.msp file contents, by executing the following command:
MsiX.exe VS80sp1-KB936971-X86-ENU.msp /out files
By the way, remember to create a folder named "files" before executing msix.
Now should have extracted a few files under the "files" folder.. Most of them are internal MSI data.. but the one named "PCW_CAB_VS_10546" is a cabinet file which contains the patched dll files.
Using your favorite cab/zip/etc. extraction tool, decompress this cab file to obtain a file called "msenv_dll_4_____X86.3643236F_FC70_11D3_A536_0090278A1BB8".
This is the patched dll we were looking for..
Replacing old msenv.dll file..
Now you can substitute the old msenv.dll files we talked about before, just remember to backup them first.. just in case.. You know ;)
In particular, if you just want to fix VS and dont brother at all about SQL Tools, etc.. The only file you should replace it's c:\Program Files\Microsoft Visual Studio 8\Common7\IDE\msenv.dll
Once this dll's are replaced, you wont experience this problem again.. Try it, just open an VS solution, try yo add a reference.. And.. Woila! you can click outside of modal window's area with out deadlocking visual studio's main window!!!
