Spurious x86_64 Windows CI failures due to OOM

[ecstatic-morse]

A MSVC x86_64 bors job has failed on two unrelated PRs (#60026, #66170) with the same error:

2019-11-12T18:01:21.3370489Z memory allocation of 4294967304 bytes failed[RUSTC-TIMING] hex test:false 3.438
2019-11-12T18:01:21.3472541Z error: could not compile `hex`.
2019-11-12T18:01:22.8915944Z [RUSTC-TIMING] glob test:false 1.984
2019-11-12T18:01:22.9032050Z error: build failed
2019-11-12T18:01:22.9081592Z command did not execute successfully: "D:\\a\\1\\s\\build\\x86_64-pc-windows-msvc\\stage0\\bin\\cargo.exe" "build" "-Zconfig-profile" "--target" "x86_64-pc-windows-msvc" "-Zbinary-dep-depinfo" "-j" "2" "--release" "--locked" "--color" "always" "--manifest-path" "D:\\a\\1\\s\\src/tools/cargo\\Cargo.toml" "--features" "rustc-workspace-hack/all-static" "--message-format" "json-render-diagnostics"
2019-11-12T18:01:22.9082199Z expected success, got: exit code: 101
2019-11-12T18:01:22.9082199Z expected success, got: exit code: 101
2019-11-12T18:01:23.0062715Z failed to run: D:\a\1\s\build\bootstrap\debug\bootstrap dist

In both cases, cargo build attempted to allocate several GB of memory while compiling the hex crate. One of these failures was in the dist-x86_64-msvc job, and the other was in x86_64-msvc-cargo.

This failure appears to be spurious: a previous version of #60026 passed the same job.

[Mark-Simulacrum]

I wonder if we could get a backtrace or something from this allocation -- it seems... suspicious that we're only sometimes OOMing.

It's also interesting to note that this is almost exactly 4 GB in a single allocation, which is surprising generally (hex is not a big crate -- 300 lines or so, with comments etc).

[ecstatic-morse]

On my PR it was a different (almost) power of two:

memory allocation of 2147483656 bytes failed

edit
In both cases, the size of the allocation is 8 bytes less than a power of two.

[ollie27]

So I minimized this to:

pub fn from_hex() -> [u8; 42949672960] {
    loop {}
}

on a recent nightly gives:

memory allocation of 42949672960 bytes failed

It looks like the compiler is trying to allocate enough space to store a value of the return type. As long as I use a large enough value it reproduces reliably and it isn't specific to Windows: playground.

I bisected this to 57a5f92 and I'm guessing #66216 (cc @wesleywiser) is the root cause.

[wesleywiser]

Yes, that seems likely. I will take a look at this tonight.