Blank Variable Watch, or OMF Errors

During a debugging session I faced a weird situation where the code compiled and ran just fine, yet some class members appeared blank at the watch window, and others showed -

CXX0033: error in OMF type information

Skip to the bottom line: there’s a compiler switch, /Yl, that’s specifically tailored to address this symptom. In my project, the issue was solved by adding  /YlSomeFunctionIUse to the stdafx.cpp compiler command line (in the project that defined, not consumed, the blank symbols).

The root cause seems to be a clash of two intended behaviors: (1) Debug info of defined symbols is embedded in the PCH object module (~= .obj file) itself, (2) when a source file refers to the PCH but does not directly use any of the functions defined in it, the PCH object module is dropped from that compilation unit altogether, and thus relevant debug info is lost.

Quoting the /Yl msdn page:

An error can occur when you store the precompiled header in a library, use the library to build an object module, and the source code does not refer to any of the functions the precompiled header file defines.

- so it might be that the symptom is revealed only when the PCH is used in a build of a static library.

This KB article says this behaviour is by design, which seems weird to me. I see no justification for dropping the debug info along with unused function definitions. I’d gladly pay the price of some PDB bloat, to avoid having to use arcane, hidden compiler switches just to be able to debug properly.

BTW, WTF is OMF?

It really stands for [Relocatable] Object Module Format, a relic of ~20 years (which goes to show how old this debugger code is, really). It’s an object-file format designed by Intel in the 70s, and used by MS prior to adoption of their own COFF flavour sometime in the early 90s. The spec is still around, but seems untouched since 1993. The industry probably gave up on the idea of using different vendors for different parts of the compiler-linker-loader tool chain, and so standardization efforts have halted since.

Three F-keys Gotchas

We recently did a small internal app, that had to use all 12 F-keys – which turned out to be surprisingly cumbersome. I hadn’t found this stuff concentrated on a single place, and certainly it could have saved me some trouble.

F1 Gotcha

Quote:

On Win32 systems, the operating system will generate the WM_HELP message when F1 is pressed.”

This isn’t much trouble if you don’t handle the message, as VK_F1 key message is still being sent. On MFC (and other) wizard generated apps, you might have to explicitly disable OnHelp on message maps:

BEGIN_MESSAGE_MAP(CMyWinApp, CWinApp)
…
  //ON_COMMAND(ID_HELP, CWinApp::OnHelp)   ! comment this
…
END_MESSAGE_MAP()

F10 Gotcha

Quote:

If the F10 key is pressed, the DefWindowProc function sets an internal flag. When DefWindowProc receives the WM_KEYUP message, the function checks whether the internal flag is set and, if so, sends a WM_SYSCOMMAND message to the top-level window. The WM_SYSCOMMAND parameter of the message is set to SC_KEYMENU.

So to get proper VK_F10 notifications, you can either bypass DefWindowProc completely – which is unfeasible, or handle specifically the WM_SYSCOMAND message. On MFC apps, that amounts to something like:

void MyWnd::OnSysCommand( UINT nID, LPARAM lParam )
{
  if(nID == SC_KEYMENU) // F10 pressed
    ::SendMessage(m_Child->GetHwnd(), WM_KEYDOWN, VK_F10, NULL);
    // The NULL in LPARAM is kinda sloppy. If you use key-message nuances, invest here a bit further.
  else
    __super::OnSysCommand(nID, lParam);
}

F12 Gotcha

This one is the most obscure – on some developer machines, pressing F12 seems to give a weird error message:

…This may be due to a corruption of the heap…  This may also be due to the user pressing F12…

To cut a long search short, this is a built in Win32 debugging feature. It can indeed be disabled through this registry key:

HKLM\Software\Microsoft\Windows NT\CurrentVersion\AeDebug\UserDebuggerHotkey

But contrary to what the connect page says, I wouldn’t advise to change it into just any nonzero value: this value is the scan code for the key that that would force a breakpoint!  If, for example, you change it to 0×9, you’d have the same problem while using the Tab key (on standard keyboards). I suggest setting the value to 0xFF – looking at some scan code tables around, I saw none that map it to an actual key.

Of course this issue would never manifest itself on customer machines, but solving it can make dev work much easier.

Afterthought

It seems an odd design choice to bake something as basic as F-keys handling so deep into the OS. I’m guessing this was done back when the wildest applications imaginable were word processors and spreadsheets, and the main design goals were to save what was perceived as boilerplate code rather than allow for flexibility. I still think modern app wizards should generate code that lets you opt-in, rather than opt out of these old key handling mechanisms.