Building CORE_ROOT is broken

[BruceForstall]

src\tests\build.cmd x64 checked generatelayoutonly fails with

c:\gh\runtime\artifacts\bin\coreclr\windows.x64.Checked\build\Microsoft.NETCore.Native.Publish.targets(77,5): error : The PrivateSdkAssemblies ItemGroup is required for _ComputeAssembliesToCompileToNative [c:\gh\runtime\src\coreclr\tools\aot\crossgen2\crossgen2_publish.csproj]

full log:

c:\gh\runtime>src\tests\build.cmd x64 checked generatelayoutonly
BUILDTEST: Starting Build at 10:19:41.63
BUILDTEST: Commencing CoreCLR test build
BUILDTEST: Common MSBuild args: "/p:TargetOS=windows" "/p:Configuration=Checked" "/p:TargetArchitecture=x64" "/p:RuntimeFlavor=coreclr" /nologo /verbosity:minimal /clp:Summary /maxcpucount
**********************************************************************
** Visual Studio 2022 Developer Command Prompt v17.12.0-pre.3.0
** Copyright (c) 2022 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'
BUILDTEST: powershell -NoProfile -ExecutionPolicy ByPass -NoLogo -Command "c:\gh\runtime\eng\common\msbuild.ps1"   c:\gh\runtime\src\tests\build.proj -warnAsError:0 /t:TestBuild /nodeReuse:false  /p:RestoreDefaultOptimizationDataPackage=false /p:PortableBuild=true  /p:UsePartialNGENOptimization=false /maxcpucount '/flp:Verbosity=normal;LogFile="c:\gh\runtime\artifacts\log\TestBuild_windows__x64__Checked.log"' '/flp1:WarningsOnly;LogFile="c:\gh\runtime\artifacts\log\TestBuild_windows__x64__Checked.wrn"' '/flp2:ErrorsOnly;LogFile="c:\gh\runtime\artifacts\log\TestBuild_windows__x64__Checked.err"' '/bl:"c:\gh\runtime\artifacts\log\TestBuild_windows__x64__Checked.binlog"'  "/p:TargetOS=windows" "/p:Configuration=Checked" "/p:TargetArchitecture=x64" "/p:RuntimeFlavor=coreclr" /nologo /verbosity:minimal /clp:Summary /maxcpucount

  [10:19:48.56] Restoring all packages...

    Determining projects to restore...
    Restored c:\gh\runtime\src\tests\Common\test_dependencies_fs\test_dependencies.fsproj (in 1.69 sec).

    Determining projects to restore...
    Restored c:\gh\runtime\src\tests\Common\test_dependencies\test_dependencies.csproj (in 178 ms).

    Determining projects to restore...
    Restored c:\gh\runtime\src\tests\Common\CoreCLRTestLibrary\CoreCLRTestLibrary.csproj (in 298 ms).

    Determining projects to restore...
    Restored c:\gh\runtime\src\tests\Common\GenerateHWIntrinsicTests\GenerateHWIntrinsicTests_Arm.csproj (in 170 ms).

    Determining projects to restore...
    Restored c:\gh\runtime\src\tests\Common\GenerateHWIntrinsicTests\GenerateHWIntrinsicTests_General.csproj (in 165 ms).

    Determining projects to restore...
    Restored c:\gh\runtime\src\tests\Common\GenerateHWIntrinsicTests\GenerateHWIntrinsicTests_X86.csproj (in 163 ms).

    Determining projects to restore...
    Restored c:\gh\runtime\src\tests\Common\XUnitLogChecker\XUnitLogChecker.csproj (in 247 ms).

    Determining projects to restore...
    Restored c:\gh\runtime\src\tests\Common\XUnitWrapperGenerator\XUnitWrapperGenerator.csproj (in 608 ms).

    Determining projects to restore...
    Restored c:\gh\runtime\src\tests\Common\XUnitWrapperLibrary\XUnitWrapperLibrary.csproj (in 159 ms).

    Determining projects to restore...
    Restored c:\gh\runtime\src\tests\Common\XHarnessRunnerLibrary\XHarnessRunnerLibrary.csproj (in 211 ms).
    1 of 2 projects are up-to-date for restore.

    Determining projects to restore...
    Restored c:\gh\runtime\src\tests\Common\external\external.csproj (in 370 ms).

    Determining projects to restore...
    Restored c:\gh\runtime\src\tests\Common\ilasm\ilasm.ilproj (in 199 ms).
  [10:20:29.07] Restoring all packages...Done.
  Building Targeting Pack
  external ->
  Determining projects to restore...
  Restored c:\gh\runtime\src\tools\illink\src\linker\ref\Mono.Linker.csproj (in 220 ms).
  Restored c:\gh\runtime\src\coreclr\tools\aot\crossgen2\crossgen2_publish.csproj (in 220 ms).
  Restored c:\gh\runtime\src\coreclr\tools\aot\ILCompiler.ReadyToRun\ILCompiler.ReadyToRun.csproj (in 227 ms).
  Restored c:\gh\runtime\src\coreclr\tools\aot\ILCompiler.Diagnostics\ILCompiler.Diagnostics.csproj (in 220 ms).
  Restored c:\gh\runtime\src\installer\pkg\sfx\Microsoft.NETCore.App\Microsoft.NETCore.App.Crossgen2.sfxproj (in 220 ms).
  Restored c:\gh\runtime\src\coreclr\tools\aot\ILCompiler.TypeSystem\ILCompiler.TypeSystem.csproj (in 249 ms).
  Restored c:\gh\runtime\src\tools\illink\src\linker\Mono.Linker.csproj (in 249 ms).
  Restored c:\gh\runtime\src\coreclr\tools\aot\ILCompiler.DependencyAnalysisFramework\ILCompiler.DependencyAnalysisFramework.csproj (in 249 ms).
  Restored c:\gh\runtime\src\tools\illink\src\ILLink.RoslynAnalyzer\ILLink.RoslynAnalyzer.csproj (in 290 ms).
  Restored c:\gh\runtime\src\libraries\System.Runtime.InteropServices\gen\DownlevelLibraryImportGenerator\DownlevelLibraryImportGenerator.csproj (in 290 ms).
  Restored c:\gh\runtime\src\tools\illink\src\ILLink.CodeFix\ILLink.CodeFixProvider.csproj (in 295 ms).
  Restored c:\gh\runtime\src\libraries\System.Runtime.InteropServices\gen\Microsoft.Interop.SourceGeneration\Microsoft.Interop.SourceGeneration.csproj (in 297 ms).
  Restored c:\gh\runtime\src\tools\illink\src\ILLink.Tasks\ILLink.Tasks.csproj (in 428 ms).
  ILLink.RoslynAnalyzer -> c:\gh\runtime\artifacts\bin\ILLink.RoslynAnalyzer\x64\Release\netstandard2.0\ILLink.RoslynAnalyzer.dll
  ILLink.CodeFixProvider -> c:\gh\runtime\artifacts\bin\ILLink.CodeFixProvider\x64\Release\netstandard2.0\ILLink.CodeFixProvider.dll
  Mono.Linker -> c:\gh\runtime\artifacts\bin\Mono.Linker\ref\x64\Release\net9.0\illink.dll
  Mono.Linker -> c:\gh\runtime\artifacts\bin\Mono.Linker\x64\Release\net9.0\win-x64\illink.dll
  ILLink.Tasks -> c:\gh\runtime\artifacts\bin\ILLink.Tasks\x64\Release\net9.0\win-x64\ILLink.Tasks.dll
  Mono.Linker -> c:\gh\runtime\artifacts\bin\Mono.Linker\x64\Release\net9.0\illink.dll
  ILLink.Tasks -> c:\gh\runtime\artifacts\bin\ILLink.Tasks\x64\Release\net9.0\ILLink.Tasks.dll
  ILCompiler.DependencyAnalysisFramework -> c:\gh\runtime\artifacts\bin\ILCompiler.DependencyAnalysisFramework\x64\Checked\ILCompiler.DependencyAnalysisFramework.dll
  ILCompiler.TypeSystem -> c:\gh\runtime\artifacts\bin\ILCompiler.TypeSystem\x64\Checked\ILCompiler.TypeSystem.dll
  ILCompiler.Diagnostics -> c:\gh\runtime\artifacts\bin\ILCompiler.Diagnostics\x64\Checked\ILCompiler.Diagnostics.dll
  ILCompiler.ReadyToRun -> c:\gh\runtime\artifacts\bin\ILCompiler.ReadyToRun\x64\Checked\ILCompiler.ReadyToRun.dll
  crossgen2_publish -> c:\gh\runtime\artifacts\bin\crossgen2_publish\x64\Checked\crossgen2.dll
c:\gh\runtime\artifacts\bin\coreclr\windows.x64.Checked\build\Microsoft.NETCore.Native.Publish.targets(77,5): error : The PrivateSdkAssemblies ItemGroup is required for _ComputeAssembliesToCompileToNative [c:\gh\runtime\src\corec
lr\tools\aot\crossgen2\crossgen2_publish.csproj]
  Determining projects to restore...
  All projects are up-to-date for restore.

Build FAILED.

c:\gh\runtime\artifacts\bin\coreclr\windows.x64.Checked\build\Microsoft.NETCore.Native.Publish.targets(77,5): error : The PrivateSdkAssemblies ItemGroup is required for _ComputeAssembliesToCompileToNative [c:\gh\runtime\src\corec
lr\tools\aot\crossgen2\crossgen2_publish.csproj]
    0 Warning(s)
    1 Error(s)

Time Elapsed 00:00:59.19
Build failed with exit code 1. Check errors above.

Note that this is documented by https://github.com/dotnet/runtime/blob/main/docs/workflow/testing/coreclr/testing.md#building-the-core_root

This was possibly broken by #106965?

@am11 @jkotas

[dotnet-policy-service]

Tagging subscribers to this area: @hoyosjs
See info in area-owners.md if you want to be subscribed.

[BruceForstall]

@agocke

[BruceForstall]

I see some CI jobs use /p:UsePublishedCrossgen2=false which causes the script to succeed (and maybe does less work, so might be faster, which is good).

[am11]

Those CI legs are separate SPMI related mostly, because they don't require crossgen2. This is passing in CI apparently.

[ivdiazsa]

Trying to reproduce it on the latest runtime. Just yesterday, I was able to generate the CORE_ROOT just fine with a few days old clone and it worked, at least on Linux.

[agocke]

Tried this locally using the exact instructions at https://github.com/dotnet/runtime/blob/main/docs/workflow/testing/coreclr/testing.md#building-the-core_root, can't repro.

[jkotas]

@BruceForstall This error means that your build does not have the live native AOT compiler that is needed by src\tests\build.cmd now. What are the command that you use to build the runtime?

[ivdiazsa]

Was able to reproduce, but it seems to be an expected failure. The trigger to see the error is to not build the clr.aot subset, which I assume is a requirement to generate the CORE_ROOT as per @jkotas's comment.

For reference, I used these commands to trigger the error:

./build.sh --subset host.native+clr.runtime+clr.corelib+clr.nativecorelib+clr.tools+clr.iltools+clr.alljits+clr.spmi+libs --configuration Release --runtimeConfiguration Checked

./src/tests/build.sh -x64 -checked -generatelayoutonly

[jkotas]

Yes, it is a new requirement after #106965.

There is a separate discussion we can have on whether it is a good idea to depend on live aot toolchain or whether it would be better to depend on the aot toolchain from the SDK (#106965 (comment) and following comments).

[BruceForstall]

Ok, so there are two solutions:

1. If building using subsets, include the clr.aot subset.
2. invoke src\tests\build.cmd generatelayoutonly with /p:UsePublishedCrossgen2=false (I don't understand the implications, but it also makes generatelayoutonly much faster)

Another solution might be using what Jan suggests above.

[VSadov]

I am hitting the same problem.

C:\Program Files\dotnet\sdk\9.0.100-rc.1.24452.12\Microsoft.Common.CurrentVersion.targets(5322,5): error MSB3030: Could not copy the file "E:\dotnet004\runtime\artifacts\obj\coreclr\crossg
en2_publish\windows.x64.Checked\apphost.exe" because it was not found. [E:\dotnet004\runtime\src\coreclr\tools\aot\crossgen2\crossgen2_publish.csproj]

As much as I try to read into the messages above, I am not sure I can see what the correct solution is.

I see that I could use /p:UsePublishedCrossgen2=false, with unknown implications.
Or I could do . . . what ?

For reference. The way that no longer works is:

build.cmd clr+libs+libs.tests -rc checked -lc release & src\tests\build.cmd checked

I've tried also doing

build.cmd clr.aot -rc checked -lc release

it does not seem to help.