Ofek's Visual C++ stuff

We recently had trouble reproducing in debug builds the exact numerical behavior of release builds.   This just happens  occasionally, and we’re used to accept it as some compiler black magic.  This time we dug a bit deeper – I’m pretty confident we understood all root causes.

1.    Different code paths

Here’s a very real code snippet:

VECTOR& VECTOR::Normalize()

{
#if defined( _DEBUG )
 D3DXVec3Normalize((D3DXVECTOR3*)this, (D3DXVECTOR3*)this);
#else
   float norm2 = x * x + y * y + z * z;
   __m128 val = _mm_load_ss( &norm2 );
   _mm_store_ss( &norm2,
       _mm_and_ps(
             _mm_cmplt_ss( _mm_set_ss( EPSILON ), val ), _mm_rsqrt_ss( val )
                 )       );

   x *= norm2;
   y *= norm2;
   z *= norm2;

#endif

return *this;
}

Not only are the execution results numerically different, the _DEBUG path consistently runs 30% faster..

2.    /arch

Release builds were compiled with /arch:SSE2, (project properties-> Configuration properties -> C/C++ -> Code Generation-> Enable enhanced instruction set) while debug builds weren’t.

This means release builds were aware of SSE instructions, and mostly favored them over FPU instructions. Even for scalar computations, SSE is usually faster – but by default it differs in computational precision (roughly, SSE: 32 bit, FPU: 80 bit).

3.    Floating point model

Release builds were compiled with /fp:fast, and debug builds with /fp:precise. (Project properties-> Configuration properties -> C/C++ -> Code Generation-> Floating point model).

That’s a heavier issue, and in some sense the root cause. A thorough survey is here.

In a nutshell – the C++ standard is strict about order of operations, and about points of intermediate rounding (from register precision to final stack precision). For example, an expression like ‘a+b+c’ must be evaluated as ‘(a+b)+c’.   a+(b+c) is usually different – since we’re working with finite precision floats:  to see why, try  a= 1.0,  b = c = 2^(-24)  (other examples at the link).

When you compile with /fp:fast, you explicitly say “dear compiler, I don’t care about this crap. When you see stuff like ‘a+b+c’, assume I don’t mind how you evaluate it – if I did, I’d put parenthesis”.   The compiler is free to make many optimization choices – a comprehensive list is at the link.

When we set /fp:precise to release builds, (after the previous 2 fixes), all remaining numerical differences vanish (well, at least all those I tested).  However in some scenarios I tested the performance penalty is tangible.

For now we set /fp:fast to all configurations (including debug) in all projects. Sadly, some numerical differences remain (the compiler does take different choices in optimized and non-optimized builds), but they seem 2 orders of magnitude smaller.

Bottom Line

After correcting for these factors, there is a much, much better chance of reproducing release-behaviour in debug builds. If there’s a behaviour you’re still unable to reproduce, try compiling both release & debug with /fp:precise and reproducing. This is the only official way of producing consistent results.

This feature is well documented, but yet from what I see – doesn’t get the usage it deserves. Here’s a quick, beginner-oriented rehash – if only to refer my teammates.

Problem and Immediate Solution

If you’re developing MFC apps, the way you’ll usually notice any leaks is by terminating your app and seeing the following in the output window:

Detected memory leaks!
Dumping objects ->
C:\myfile.cpp(20): {130} normal block at 0x00780E80, 64 bytes long.
Data: <      > CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD CD
Object dump complete.

the {130} is emphasized on purpose – it is the serial number of the leaking allocation.  What if you could count allocation occurrences, and break in exactly the leaky one?  You could not only get a complete stack trace for the leak, but even step and debug it!

Well, it might not be that easy.  If the allocation serial number is not consistent across multiple runs, it means the leaking memory is allocated in a threaded code portion.  In such cases, you’re probably better off resorting to other methods.

Now if the serial number is consistent, what allocations exactly does it count? In what API do you brake??  must you really set a breakpoint with a ‘hit count’ there?  If the number is high, it could easily get prohibitively slow.

Happily, there is an easy solution to the second set of worries. Early in your code, call –

_CrtSetBreakAlloc(130)

Mid-depth Dive

The _CrtSetBreakAlloc trick is an iceberg-tip of some heavier CRT machinery. It is entirely operated for you if you use MFC, but you can use it yourself in non-MFC apps (although you should watch out for some issues).

The dump that starts it all is the output of _CrtDumpMemoryLeaks.  You can call it directly in your app – but if you’re down to a binary search for the leak , you  might prefer _CrtMemDumpAllObjectsSince for more fine-grained control of the allocations you dump. Maybe more on that in a future post.

The allocation counter resides in the undocumented _heap_alloc_dbg, which channels calls from all C allocators – namely new, malloc, and their brethren (_malloc_dbg, aligned /offset flavours, etc.).  This is a CRT apparatus, so naturally it won’t catch direct OS allocations (HeapAlloc, VirtualAlloc et. al.), not to mention COM allocators (SysAllocString, CoTaskMemAlloc etc.).

When you do set an allocation breakpoint, you’d (naturally) break in the counting function itself, _heap_alloc_dbg.  Your own code probably resides a good 4-5 stack levels below – go there and debug away.

Bonus

You can do all the above at runtime, from the debugger. The documentation for VS2005 says you can put –

{,,msvcr71d.dll}_crtBreakAlloc

in the watch window, and modify its contents directly (say, to 130). However, the documentation is wrong in 3 places.

1. For VS2005, the correct dll version is msvcr80d.dll. This seems to have been fixed for the VS2008 and VS2010 pages.
2. When using the context operator, you must use decorated symbol names – which amounts to adding another underscore.
3. The value is an int*, and for some reason my debugger fails to deduce it himself.

All in all, just type at the watch window –

(int*){,,msvcr80d.dll}__crtBreakAlloc

And debug happily ever after.

MS maintains two separate debugger product lines – one (supposedly owned by the smaller team) being WinDbg and the other shipped with Visual Studio. There’s an ever-going online flame for superiority between fans of both debuggers (sample 1, sample 2).

By and far WinDbg is considered the ‘serious’ brother, and partially for a good reason.  There are important domains where WinDbg is the only option – notably kernel mode debugging. It is also said to be easier on remote debugging (which I’ve tried only with VS, and admittedly had some trouble).  However, I’ve too often seen claims for WinDbg superiority that are just based on poor VS skills – I’ll try and dedicate some posts to show VS parallels of common WinDbg capabilities.  I currently believe if you do user mode debugging with source code (even in production builds), there’s no real reason for you to step back to the command-line UI of WinDbg. I’d be happy to hear if anyone has different experience.

So, starting off with memory search. This is a relatively known capability of WinDbg, but it’s apparently much less known it is in fact available in VS.

You can search through a memory range for a particular pattern (e.g., a string) or one of several patterns. You do this by typing at the Immediate window (Ctrl+Alt+I while debugging), and the complete syntax is here.

A notable shortcoming of VS is that it does not implement the ‘L?’ address-range WinDbg syntax, so a single search is limited to a memory range of 256M.  On the few occasions where I needed this functionality, this wasn’t a major issue.