Visual Studio Projects that Just Keep Rebuilding, or: How Quantum Mechanics Mess Up Your Build

A lot has already been said on it, (no, seriously, a lot) and yet some root causes are not yet covered. All that follows holds for C++ projects, C#, VB, and everything MSBuild in general.

The symptom

Normally if you build your solution, change nothing and immediately try to build again – nothing happens, as you’d expect. But occasionally some projects are rebuilt despite not being modified. If by bad luck these projects are high up the dependency tree – many other projects are re-built, and irks galore abound.

First step: diagnose

Set MSBuild’s build verbosity to ‘diagnostic’, under Tools / Options / Projects and solutions / Build and Run:

Then build. The first lines of the output should hold the reason that VS thinks the project is out of date. Typical stated reasons are –

—— Up-To-Date check: Project: Whateva, Configuration: Release Win32 ——
Project not up to date because build input ‘F:\sources\Shiny.h’ is missing.

Project ‘Whateva’ is not up to date. Project item ‘Shiny.html’ has ‘Copy to Output Directory’ attribute set to ‘Copy always’.

Project ‘Whateva’ is not up to date. CopyLocal reference source ‘F:\folder1\blah.dll’ is more recent than ‘F:\folder2\blah.dll’.

Project ‘Whateva’ is not up to date. Input file ‘F:\sources\yadda.csproj’ is modified after output file ‘F:\bin\yadda.pdb’.

Project ‘Whateva’ is not up to date. Last build was with unsaved files.

In my personal experience the most common reason is the first one: a file that is included in a project but is missing on disk. Treating this state as out-of-date is a weird design decision (I’d say VS should refuse to build), but that’s just the way it is. Anyway, more can – and should – be said about file dates.

File Dates: More than meets the eye

Oddly, sometime MSBuild’s claim that file1 is newer than file2 persists after a build re-copies file1 over file2. Sometimes manually copying file1 over file2 helps, and sometimes not.

The question whether file1 is newer than file2 is not as straightforward as it might seem. NTFS has two associated times: created time and modified time (never mind ‘last access time’ now). The full rules of how these dates react to a copy/move are somewhat involved but the key piece in this context is:

If you copy a file from D:\NTFS to D:\NTFS\SUB, it keeps the same modified date and time but changes the created date and time to the current date and time. [In particular, making the created date later than the modified date]

So the date that is preserved across copies is modified date – but MSBuild compares created dates to determine whether a file should be copied. I sincerely wish it wouldn’t – but wait, the creation date in the copy can only grow newer than the source, and this shouldn’t call for a re-copy anyway, should it?

Enter NTFS Tunneling.

This Windows feature is beyond esoteric, so don’t feel bad if it doesn’t ring any bell. In a nutshell:

When a name is removed from a directory (rename or delete), its short/long name pair and creation time are saved in a cache, keyed by the name that was removed. When a name is added to a directory (rename or create), the cache is searched to see if there is information to restore. The cache is effective per instance of a directory. If a directory is deleted, the cache for it is removed.

Simply put, when you copy a file to a location where it previously existed, its original created date is resurrected – regardless of the created date of the actual source file.

<Sigh/>.

The name ‘tunneling’ derives from a quantum mechanics phenomena (hence the clickbait post title) where particles emerge in seemingly impossible locations. The original motivation for this design is irrelevant for decades now (probably since MS-DOS), and it is most probably left around as a compatibility constraint. Even better, on my own computer it seems the lifetime of the cache (advertised to be 15 seconds) is really infinite, and modifying it via the MaximumTunnelEntryAgeInSeconds registry key isn’t working.

<Double sigh/>.

Solutions

You can shut down tunneling altogether by setting HKLM\SYSTEM\CurrentControlSet\Control\FileSystem\MaximumTunnelEntries to 0, as shown in the KB article.

If for whatever reason you prefer not to mess with arcane NTFS registry keys, you can do the following: rename your bin folder (the one holding the created-date cache) to a temporary name, create a new bin folder and copy the previous bin contents to it. This seemingly no-op clears the tunneling cache and in my experience rids of the last bogus out-of-date checks.

These redundant re-builds are pretty much gone for me now. Please do tell me in the comments if it worked for you – and more importantly, if it didn’t.

Advertisement
This entry was posted in Uncategorized. Bookmark the permalink.

7 Responses to Visual Studio Projects that Just Keep Rebuilding, or: How Quantum Mechanics Mess Up Your Build

  1. Ivan says:

    Thanks Ofek, build verbosity was good enough for me (VS2015CE). This really saved my sanity, thanks a lot.

  2. Emil Wojak says:

    I changed the name of one of my header files without updating the VS project. It kept rebuilding because the header file was missing under its old filename. Log verbosity enlightened me. Thank you!

  3. Pingback: Visual Studio 2017 builds all project because of "missing" PDB file | Question and Answer

  4. Sam says:

    Still having a lot of trouble with the last case “Project ‘Whateva’ is not up to date. Last build was with unsaved files.” Wish the verbose logs were a bit clearer to read and determine what is causing the problem.

  5. Anonymous says:

    Sam- when you build from Visual Studio, msbuild receives the unsaved versions of files from Studio. In this case it will need to rebuild. Adding nuget packages and changing project properties can edit the csproj file, in which case you won’t see the * on the editor for unsaved files. Use File>Save All.

    • Ofek Shilon says:

      VS saves all automatically before running a build. I, like Sam, routinely encounter rebuild glitches that are unsolved by verbose msbuild logging.

  6. Kerem Ispirli says:

    The following bit might need an update:
    “So the date that is preserved across copies is modified date – but MSBuild compares created dates to determine whether a file should be copied.”

    I have a C++ project that keeps re-linking and for every other linking MSBuild log shows this message:
    “Source compilation required: input D:\PATH\TO\PROJECT\SOMECLASS.OBJ is newer than output D:\PATH\TO\PROJECT\PROJECTNAME.PDB.”
    While the creation date of .PDB file is up-to-date, the creation date of .OBJ file seems to be resurrected from the Tunnel Cache; it’s a week older. According to your article, this warning should be other way around.

    But when I compare the ModifiedDate’s of the files, I see that the .OBJ file has been written more than 200ms later than .PDB file; so it fits with MSBuild’s warning, it doesn’t make sense and it seems to be my actual problem.

    Anyway, it looks like a new GetOutOfDateItems task specific for C++ was introduced with VS 2017, which decides according to tlog files: https://docs.microsoft.com/en-us/visualstudio/msbuild/getoutofdateitems-task?view=vs-2017

    Kind regards,
    Kerem

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s