Discovering Which Projects Depend on One

I am working with several large-ish (100+ project) solutions – and at this scale, dependency management is a very real issue. While you can easily view (and set) the dependencies of a project by viewing its references, there is no obvious tool to answer the reverse question: which projects depend on a given one?

Obviously a hack is in order. Enter project dependencies – project references predecessor. Both appear in the project context menu (from the solution explorer):

In a nutshell, dependencies are stored per-solution while references are stored per-project, as they should. But that’s beside the point here. The point is: the dependencies display is smart enough to keep you from forming cyclic dependencies. When you click ‘Project Dependencies…’ you’d see something like this:

The checked boxes indicate projects that the current one (selected in the top combo) includes either in its references or dependencies. The greyed out boxes (marked in a red rectangle here) indicate projects that include the current one in a similar manner. Indeed, if you try and check a greyed out box – thereby adding it to the current project dependencies – you get:

So there you have it: the list of greyed out boxes is a poor man’s answer to the question – which projects depend on the current one.

Note two limitations:

  1. These dependencies are both direct and indirect. Distinguishing these still requires some manual extra work.
  2. This hack applies only linker dependencies among projects, and is blind to dependency by header file inclusion. Generally speaking this amounts to dependency upon interfaces and not implementations (neglecting templates and other inlines), and so is a weaker form of dependency – but still one that might be of interest.

A few months ago I decided such hacks are no replacement for a proper tool, and started using CppDepend. It is not perfect, but I’m growing to like it. Maybe more on that in a future post – but in the meantime this hack should be useful to anyone working in large solutions like mine.

VS2012 Migration #3: autoexp and NoStepInto Replacements

In the past I blogged quite a few times about two immensely useful albeit mostly-unofficial debugger features: watch modification via autoexp.dat, and step-into modification via NoStepInto registry key. A long while ago I raised two suggestions at MS UserVoice, to invest in making these two semi-hacks into documented, supported features. The first suggestion got some traction, and is officially implemented in VS2012. The 2nd suggestion went mostly ignored – but nevertheless, there’s a new and better – though still undocumented – way to skip functions while stepping.

NatVis files

The Natvis (native-visualizers) file format is the shiny new replacement for autoexp.dat. It is well documented, and although still quite rough around the edges – bugs are accepted and treated, which means that for the first time it is actually supported. The new apparatus comes with several design advantages:

  1. It seems to be better isolated and not to crash the IDE so much,
  2. New visualizer debugging facilities are built in,
  3. Separate customized visualizers can be kept in separate files, allowing easier sharing (e.g., library writers can now share distribute .natvis files with their libraries).
  4. Natvis files can be placed at per-user locations.

It isn’t that much fun rehashing the syntax – being official and all – but I will include here a custom mfc-containers natvis, similar to the autoexp section I shared a while back

<?xml version="1.0" encoding="utf-8"?>
<AutoVisualizer xmlns="http://schemas.microsoft.com/vstudio/debugger/natvis/2010">
  <!--from afxwin.h -->
  <Type Name="CArray&lt;*,*&gt;">
    <AlternativeType Name="CObArray"></AlternativeType>
    <AlternativeType Name="CByteArray"></AlternativeType>
    <AlternativeType Name="CDWordArray"></AlternativeType>
    <AlternativeType Name="CPtrArray"></AlternativeType>
    <AlternativeType Name="CStringArray"></AlternativeType>
    <AlternativeType Name="CWordArray"></AlternativeType>
    <AlternativeType Name="CUIntArray"></AlternativeType>
    <AlternativeType Name="CTypedPtrArray&lt;*,*&gt;"></AlternativeType>
    <DisplayString>{{size = {m_nSize}}}</DisplayString>
    <Expand>
      <Item Name="[size]">m_nSize</Item>
      <Item Name="[capacity]">m_nMaxSize</Item>
      <ArrayItems>
        <Size>m_nSize</Size>
        <ValuePointer>m_pData</ValuePointer>
      </ArrayItems>
    </Expand>
  </Type>

  <Type Name="CList&lt;*,*&gt;">
    <AlternativeType Name="CObList"></AlternativeType>
    <AlternativeType Name="CPtrList"></AlternativeType>
    <AlternativeType Name="CStringList"></AlternativeType>
    <AlternativeType Name="CTypedPtrList&lt;*,*&gt;"></AlternativeType>
    <DisplayString>{{Count = {m_nCount}}}</DisplayString>
    <Expand>
      <Item Name="Count">m_nCount</Item>
      <LinkedListItems>
        <Size>m_nCount</Size>
        <HeadPointer>m_pNodeHead</HeadPointer>
        <NextPointer>pNext</NextPointer>
        <ValueNode>data</ValueNode>
      </LinkedListItems>
    </Expand>
  </Type>
  
  <Type Name="CMap&lt;*,*,*,*&gt;::CAssoc">
    <AlternativeType Name="CMapPtrToWord::CAssoc"></AlternativeType>
    <AlternativeType Name="CMapPtrToPtr::CAssoc"></AlternativeType>
    <AlternativeType Name="CMapStringToOb::CAssoc"></AlternativeType>
    <AlternativeType Name="CMapStringToPtr::CAssoc"></AlternativeType>
    <AlternativeType Name="CMapStringToString::CAssoc"></AlternativeType>
    <AlternativeType Name="CMapWordToOb::CAssoc"></AlternativeType>
    <AlternativeType Name="CMapWordToPtr::CAssoc"></AlternativeType>
    <AlternativeType Name="CTypedPtrMap&lt;*,*,*&gt;::CAssoc"></AlternativeType>
    <DisplayString>{{key={key}, value={value}}}</DisplayString>
  </Type>

  <Type Name="CMap&lt;*,*,*,*&gt;">
    <AlternativeType Name="CMapPtrToWord"></AlternativeType>
    <AlternativeType Name="CMapPtrToPtr"></AlternativeType>
    <AlternativeType Name="CMapStringToOb"></AlternativeType>
    <AlternativeType Name="CMapStringToPtr"></AlternativeType>
    <AlternativeType Name="CMapStringToString"></AlternativeType>
    <AlternativeType Name="CMapWordToOb"></AlternativeType>
    <AlternativeType Name="CMapWordToPtr"></AlternativeType>
    <AlternativeType Name="CTypedPtrMap&lt;*,*,*&gt;"></AlternativeType>
    <DisplayString Condition="(m_nHashTableSize &gt;= 0 &amp;&amp; m_nHashTableSize &lt;= 65535">{{size={m_nHashTableSize}}}</DisplayString>
    <Expand>
      <Item Name="num bins">m_nHashTableSize</Item>
      <ArrayItems>
        <Size>m_nHashTableSize</Size>
        <ValuePointer>m_pHashTable</ValuePointer>
      </ArrayItems>
    </Expand>
  </Type>

  <Type Name="CMap&lt;*,*,*,*&gt;">
    <AlternativeType Name="CMapPtrToWord"></AlternativeType>
    <AlternativeType Name="CMapPtrToPtr"></AlternativeType>
    <AlternativeType Name="CMapStringToOb"></AlternativeType>
    <AlternativeType Name="CMapStringToPtr"></AlternativeType>
    <AlternativeType Name="CMapStringToString"></AlternativeType>
    <AlternativeType Name="CMapWordToOb"></AlternativeType>
    <AlternativeType Name="CMapWordToPtr"></AlternativeType>
    <AlternativeType Name="CTypedPtrMap&lt;*,*,*&gt;"></AlternativeType>
    <DisplayString>{Hash table too large!}</DisplayString>
  </Type>
  

  <Type Name="ATL::CAtlMap&lt;*,*,*,*&gt;">
    <AlternativeType Name="ATL::CMapToInterface&lt;*,*,*&gt;"/>
    <AlternativeType Name="ATL::CMapToAutoPtr&lt;*,*,*&gt;"/>
    <DisplayString>{{Count = {m_nElements}}}</DisplayString>
    <Expand>
      <Item Name="Count">m_nElements</Item>
      <ArrayItems>
        <Size>m_nBins</Size>
        <ValuePointer>m_ppBins</ValuePointer>
      </ArrayItems>
    </Expand>
  </Type>
  <Type Name="ATL::CAtlMap&lt;*,*,*,*&gt;::CNode">
    <DisplayString Condition="this==0">Empty bucket</DisplayString>
    <DisplayString Condition="this!=0">Hash table bucket</DisplayString>
  </Type>
</AutoVisualizer>

Visualizing Map is a bit tricky, and I didn’t take the time yet to look deep into it – but the file is hopefully useful as it is. To use, just save the text as, say, MfcContainers.natvis, either under %VSINSTALLDIR%\Common7\Packages\Debugger\Visualizers (requires admin access), or under %USERPROFILE%\My Documents\Visual Studio 2012\Visualizers\ .

NatStepFilter Files

- are the new and improved substitute for the NoStepInto registry key. While there are some online hints and traces, the natstepfilter spec is yet to be introduced into MSDN – or even the VC++ team blog. For now you can watch the format specification, along with some good comments, at the %VSINSTALLDIR%\Xml\Schemas\natstepfilter.xsd near you, or even better – inspect a small sample at %VSINSTALLDIR%\Common7\Packages\Debugger\Visualizers\default.natstepfilter.

The default.natstepfilter is implemented by Stephen T. Lavavej, and is very far from complete – both because of regex limitations and because of decisions not to set non-overridable limitations on users:

“Adding something to the default natstepfilter is a very aggressive move, because I don’t believe there’s an easy way for users to undo it (hacking the file requires admin access), and it may be surprising when the debugger just decides to skip stuff.”

I can think of several ways for users to override .natstepfilter directives (never mind stepping-into via assembly, how about setting a plain breakpoint it the function you wish to step into?) – and so I don’t agree with that decision. Still I hope the default rules would improve alongside the documentation. We mostly avoid STL, so I had no need to customize .natstepfilter’s yet – I’ll be sure to share such customizations if I do go there.

Caveat

Both improvements, natvis and natstepfilter files, do not work for debugging native/managed mixed code, which sadly renders them unusable for most of our code. While this behavior is documented – I would hardly say it is ‘by design’. It does seem to irritate many others, so there is hope – as Brad Sullivan writes that MS are-

“… working on making everything just work in a future release of Visual Studio.”

VS2012 Migration #1: “The Project ‘’ has been renamed’ and Other Errors on Build

About half a year ago I started experimenting with VS2012. It was not a smooth migration by any means, and I finally got around to recording online some of the lessons I learnt along the way.

One of the first things that greeted me was a slew of weird and uninformative error messages, whenever a build was tried: ‘The project ” has been renamed’ on sln build, ‘a build is already in progress’ on individual project build, and ‘Object reference not set to an instance of an object’ – well, just occasionally.

Others come across similar issues as well, but I believe more can be said than what was already noted. Bottom line, the issue is with native references to projects not contained in the current solution.

VS2010 did agree to try and build with such external references, and if a binary copy of them was available where VS2010 could find it – either the build would succeed or fail with surprising errors (e.g., due to debug/release version mismatch). VS2012 refuses to try and build in such a case, which is a very good thing. I only wish the error messages were more informative.

Moreover, VS2010 didn’t show the external references in the references tab. Not only does VS2012 show the missing projects, it (probably unintentionally) makes it easy to distinguish them by displaying them as the explicit path (as stored in the referencing vcxproj file), and not the referenced project name:

Thus, the solution to these error messages is to just scan the solution’s projects’ references , find the missing ones by looking for vcxproj suffixes, and either remove them (if they are redundant) or add them to the solution. If your solution is very large, it might be faster to isolate the offending projects by unloading part – say, half – of the projects at a time, and see if a build attempt still raises errors.

I really wish the reference UI (and dare I say, even the underlying mechanism) for native projects was as clear as for managed projects. But guess we’ll have to do with such hacks for now.

Entry Point Not Found, and other DLL Loading Problems

Occasionally I come across DLL load problems:

The verbosity of the error messages varies greatly. In their raw form these include at least the DLL name, but as various frameworks come into play (for the error message above, it’s .net) – native exceptions are caught and re-thrown, and more often than not helpful information is lost on the way.

Turns out there’s a built in way to get verbose windows-loader output: the Show Loader Snaps flag. The easiest way to mark it is with the gflags utility, bundled with debugging tools for windows:

Under the hood, it merely adds a FLG_SHOW_LDR_SNAPS flag (0×00000002), to a DWORD value in the relevant IFEO registry key. This in turn causes Windows Loader to set the _ShowSnaps variable in the ntdll copy specific to the named process.

And now, behold the new and shiny loader trace (dumped to the debugger output window):

…    

2724:245c @ 11813487 – LdrpFindOrMapDll – RETURN: Status: 0×00000000

2724:245c @ 11813487 – LdrpLoadImportModule – RETURN: Status: 0×00000000

2724:245c @ 11813487 – LdrpLoadImportModule – RETURN: Status: 0×00000000

2724:245c @ 11813487 – LdrpLoadImportModule – RETURN: Status: 0×00000000

2724:245c @ 11813487 – LdrpSnapThunk – WARNING: Hint index 0x70a for procedure “?Revert@CStreamMemory@@UAGJXZ” in DLL “YaddaYadda.dll” is invalid

2724:245c @ 11813487 – LdrpSnapThunk – ERROR: Procedure “?Revert@CStreamMemory@@UAGJXZ” could not be located in DLL “YaddaYadda.dll”

First-chance exception at 0x77321d32 (ntdll.dll) in Strategist.exe: 0xC0000139: Entry Point Not Found.

Bam! There’s the offending DLL and the offending imported function, right there in the debugger.

Like many other useful features – it is documented, but very low on discoverability. Which is a fancy way of saying you can find it only if you already know exactly what you are looking for. I personally got around to it after digging around in ntdll assembly (just like Matt Pietrek, 14 years ago), trying to get to a string containing the name of an offending DLL.

The windows-copycat-opensource ReactOS source gives a nice view of the internal usage of this flag – called ShowSnaps in their source. The ‘snapping’ verb in this context refers to one of the actions performed by the loader: after rebasing the loaded DLL in the loading process memory space, the DLL’s exported function addresses are updated and must be copied to the importing process (or other dll) Import Address Table. This – in this context – is called snapping, and that’s where the extra tracing is hooked.

Forcing Construction of Global Objects in Static Libraries

Suppose you have a global object whose constructor does useful stuff – say, registration somewhere or initialization of global resources. Suppose further this object isn’t directly accessed anywhere – you just need the functionality in its ctor (which might trigger other functionality eventually. All is fine, until we add the last assumption: suppose this object lies in a static library.

This seems to be a long lasting pain, ultimately arising from the old (‘broken’? let’s just say ‘outdated’) C++ compiler-linker model. The way the linker works is by repeatedly searching for implementations of yet-unresolved referenced symbols, and including only the obj files with such implementations – thereby dropping entirely obj files with no external references, such as the one containing the global object whose ctor you need to run.

To make things concrete, take the following toy example:


//main.cpp
#include <tchar.h>

int _tmain(int argc, _TCHAR* argv[])
{
	return 0;
}

//GlobalInLib.cpp – compile as static lib
#include <stdio.h>
#include <tchar.h>

struct UsefulCtor
{
	UsefulCtor()  { _tprintf(_T("ThereIsNoSpoon")); }
};

UsefulCtor MyGlobalObj;

Under normal linkage, MyGlobalObj would be ignored. You can verify this either by putting a breakpoint in its constructor and see that it is never hit, or inspecting the output console window and see that it is empty.

<Aside> An interesting discussion arose a while ago in MS forums on whether this behavior violates the standard. Here, einros writes:

The C++ standard, section 3.7.1, specifies:

“If an object of static storage duration has initialization or a destructor with side effects, it shall not be eliminated
even if it appears to be unused, [...]“

But MS’ Holder Grund clarifies –

[Your quote of the standard] only holds if the corresponding translation unit is part of the program. In my definition and the one of at least four major toolchain implementators, it is not.

</Aside>

Enter ‘Use Library Dependency Inputs’.

This arcane combo box in the project references dialog has the sole documented effect of enabling incremental linking for static libs, but the interesting part is how it does it:

When this property is set to Yes, the project system links in the .obj files for .libs produced by dependent projects, thus enabling incremental linking.

And indeed, setting this option to True causes construction of MyGlobalObj in the example above.

Turns out you can force construction of globals in static libs after all.

 

 

Addendum: Only after writing this post did I come across this excellent 2005->2012 thread, which mentions this setting as a solution. Still, this effect of the linker is all but undocumented, and qualifies as deserving-more-web-presence.