Friday, April 3, 2015

Adventures (and frustrations) in NuGet package creation

I'm frustrated.

And disappointed.

Frustrated and disappointed. But more knowledgeable.

I spent a good part of today trying to get our CI build to create our NuGet packages so the resulting packages list their dependencies without developers having to manually edit the .nuspec file unless they have to. Unfortunately, I've failed, but I know a little more about how to interact with our TeamCity installation and the nuget pack command.

What I learned was that our build process invokes the nuget pack command which creates the package. Reviewing the documentation for that command I read about the IncludeReferencedProjects option which is described as doing the following:
Include referenced projects either as dependencies or as part of the package. If a referenced project has a corresponding nuspec file that has the same name as the project, then that referenced project is added as a dependency. Otherwise, the referenced project is added as part of the package.
"This is great!" I thought, "The solution I'm currently working on has project references that met that description. I won't have to explicitly declare the dependency."

What I was trying to do

Maybe a brief note about the solution is in order. It's for a WebAPI and has several projects in it, not all of which are published as NuGet packages. The projects are:
  • MyApp 
  • MyApp.Application
  • MyApp.Api
  • MyApp.Api.WebHost
  • MyApp.Repository
With the exception of MyApp.Api they each result in a NuGet package.  Mainly because even though we're moving to have all calls go through the MyApp.Api.WebHost which is the package that's automatically deployed to the web servers, there are some older projects that continue to need access to MyApp, MyApp.Application and MyApp.Repository.

As you would guess, there are project references between some of the projects. For example, MyApp.Api.WebHost references MyApp, MyApp.Application and MyApp.Repository. What we've been doing is modifying the web host project's .nuspec file to declare the dependencies on MyApp, MyApp.Application and MyApp.Repository.

What I tried

Given this configuration I thought the IncludeReferencedProjects option would allow us to only declare package dependencies that existed outside the solution.

So I sat with a co-worker who is very knowledgeable about the TeamCity templates we have. He showed me the how to include that option on the build step which creates the NuGet package. We copied the template to a temporary one to test with, added the option, associated the MyApp build to that template, modified the .nuspec file to remove the explicit package dependency declarations, committed the code, and sat back to see the results.

It didn't work.

The resulting nupkg file doesn't list the dependencies. We then modified the way nuget was run to turn up the verbosity (-Verbosity detailed) and ran it again to see if we learned anything useful. All we saw in our build log was [pack] Dependencies: None. Then my co-worker noticed in the pack command examples that in addition to running nuget pack against the .nuspec file you can also run it against the .csproj file. So we tried that. That change resulted in a number of failing builds because the .csproj files had references to Visual Studio 10 and we're using VS2013 - doesn't make a difference to MSBuild because it knows what version it is, and it didn't matter to nuget pack when we targeted the .nuspec file (presumably because it didn't need it) but when we target the .csproj file it broke. So we changed the project files so:
<VisualStudioVersion Condition="'$(VisualStudioVersion)' == ''">10.0</VisualStudioVersion>
became
<VisualStudioVersion Condition="'$(VisualStudioVersion)' == ''">12.0</VisualStudioVersion>

and
<Import Project="$(MSBuildExtensionsPath32)\Microsoft\VisualStudio\v10.0\WebApplications\Microsoft.WebApplication.targets" Condition="false" />
became
<Import Project="$(MSBuildExtensionsPath32)\Microsoft\VisualStudio\v12.0\WebApplications\Microsoft.WebApplication.targets" Condition="false" />

Neither of these changes fixed the problem. We were continuing to see an error that read:
[pack] Microsoft.Build.Exceptions.InvalidProjectFileException: The imported project "C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v10.0\WebApplications\Microsoft.WebApplication.targets" was not found. Confirm that the path in the declaration is correct, and that the file exists on disk.  [the path to my .csproj file]
[pack]    at Microsoft.Build.Shared.ProjectErrorUtilities.ThrowInvalidProject(String errorSubCategoryResourceName, IElementLocation elementLocation, String resourceName, Object[] args)
[pack]    at Microsoft.Build.Shared.ProjectErrorUtilities.ThrowInvalidProject(IElementLocation elementLocation, String resourceName, Object arg0)
[pack]    at Microsoft.Build.Evaluation.Evaluator`4.ExpandAndLoadImports(String directoryOfImportingFile, String importExpressionEscaped, ProjectImportElement importElement)
[pack]    at Microsoft.Build.Evaluation.Evaluator`4.EvaluateImportElement(String directoryOfImportingFile, ProjectImportElement importElement)
[pack]    at Microsoft.Build.Evaluation.Evaluator`4.PerformDepthFirstPass(ProjectRootElement currentProjectOrImport)
[pack]    at Microsoft.Build.Evaluation.Evaluator`4.Evaluate()
[pack]    at Microsoft.Build.Evaluation.Evaluator`4.Evaluate(IEvaluatorData`4 data, ProjectRootElement root, ProjectLoadSettings loadSettings, Int32 maxNodeCount, PropertyDictionary`1 environmentProperties, ILoggingService loggingService, IItemFactory`2 itemFactory, IToolsetProvider toolsetProvider, ProjectRootElementCache projectRootElementCache, BuildEventContext buildEventContext, ProjectInstance projectInstanceIfAnyForDebuggerOnly)
[pack]    at Microsoft.Build.Evaluation.Project.ReevaluateIfNecessary(ILoggingService loggingServiceForEvaluation)
[pack]    at Microsoft.Build.Evaluation.Project.Initialize(IDictionary`2 globalProperties, String toolsVersion, String subToolsetVersion, ProjectLoadSettings loadSettings)
[pack]    at Microsoft.Build.Evaluation.Project..ctor(String projectFile, IDictionary`2 globalProperties, String toolsVersion, String subToolsetVersion, ProjectCollection projectCollection, ProjectLoadSettings loadSettings)
[pack]    at NuGet.Commands.PackCommand.BuildFromProjectFile(String path)
[pack]    at NuGet.Commands.PackCommand.BuildPackage(String path)
[pack]    at NuGet.Commands.PackCommand.ExecuteCommand()
[pack]    at NuGet.Commands.Command.Execute()
[pack]    at NuGet.Program.Main(String[] args)
[pack] Process exited with code 1


For some reason neither of us understood, the project file was still looking for version 10, even though the file on the build server had version 12 specified. We decided to change the way nuget pack was being called to include the property assignment of the VisualStudioVersion (-Properties Configuration=Release;VisualStudioVersion=12.0).

This got us a successful build and the resulting nupkg file started listing our dependencies as expected.

So the IncludeReferencedProjects option only appears to work when nuget pack targets the project file.

Short lived victory lap

After high fives all around we set about changing the real template for our CI builds. Once that was done we kicked off builds for the projects which used that template. What we found was that even with the changes we made related to the VisualStudioVersion, in both the properties passed from the nuget pack command and the project files, some of the builds kept failing because the version being sought was v10 not v12.

Observations before reverting

What we noticed was the problem seemed to occur when project A.1 was being packed but project A.1 referenced A.2 and the import line that was causing the exception was in project A.2.  For example, during the pack of project MyApp.Account.Api.WebHost the exception line was
[pack] Microsoft.Build.Exceptions.InvalidProjectFileException: The imported project "C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v10.0\WebApplications\Microsoft.WebApplication.targets" was not found. Confirm that the path in the declaration is correct, and that the file exists on disk.  F:\Agent\Work\3b114de58491d1e6\src\MyApp.Account.Api\MyApp.Account.Api.csproj

At this point we got stumped because all the project files had the hard coded v12.0 value. Somehow the import is being given the wrong path. We can't find the cause of it and after a half day we had to move on.

Appeal for advice

If anyone knows how we might successfully revisit this and work past the problems; either by telling us why the IncludeReferencedProjects option wasn't working from the .nuspec file or explaining what we have to do related to the VisualStudioVersion, (or something else we're not seeing that you know of) it would be appreciated. Tweet me.