Support worker termination by collecting sum stats on exit.

[jdmarshall]

This is the last PR required for the next semver-major release of the Prometheus Client. It addresses #803, via code changes, exposing additional lifecycle events for managing worker lifetimes, and documentation changes to warn of corner cases.

This change creates a retained mode for defunct workers where all of their final telemetry for 'sum' metrics is recorded in the aggregator process and aggregated down with each subsequent terminated worker. It also uses a call and response
system so that the worker can ensure that the flushed data was acknowledged by the collector before exiting.

As part of this work, most of the 'thenables' have been removed from the clusterMetrics() code path, in favor of async/await which should also reduce the surface area of incompatibility with third party Promise libraries. And in general the code complexity of clusterMetrics has been reduced close to the essential complexity of this code, to satisfy Kernighan's Law with respect to spotting bugs in the implementation.

Pains are taken to make sure that the workers and historicalMetrics fields are only ever read or written on the same tick, so that any shutdown operations that overlap with scraping don't smash into each other and cause data artifacts. You either get a scrape because the workers all replied before shutting down, or you get a transient timeout error that corrects itself on the subsequent collection.

The collector thread only collects 'sum' metric types - a dead process cannot participate in average, max, or min calculations any longer and those outliers shouldn't be retained once the process exits. However counts and histograms must be kept in aggregate to prevent sawtooth patterns in the collected data due to elision of previously-reported data.

This PR also reworks a number of tests that relied on delay() to instead watch the appropriate EventEmitters to watch for events that are being emitted asynchronously. There are a couple of cases where tests have around a 5-10% failure rate due to timing issues and if you run the suite over and over again, you will see false positives intermittently. Most of these should now be corrected.

Because this is trying to keep two implementations in lockstep, it is unfortunately again a rather large PR, but hopefully not difficult to comprehend.

[krajorama]

First pass.

The general approach makes sense to me, but will need another pass to go into the details.

[krajorama]

Seems to me these are just function calls, not promises, so no await needed?

Suggested change

	historicMetrics = await Registry.aggregate(metrics).getMetricsAsArray();
	historicMetrics = Registry.aggregate(metrics).getMetricsAsArray();

[jdmarshall]

Hmm. That’s odd. One is sync and the rest are async. However did we manage that?

But I should double check that collect() is being called in the worker because I bet I missed one.

And this needs to be fixed in both.

Also there's an unhandledRejectionError surface area here that I missed. This code evolved during testing and I think I ran out of steam before it was finished.

[jdmarshall]

Looks like maybe I didn't quite harden all of the race conditions in the tests.

Oh, that test. Ugh.

Edit: Looks like I missed one of the old tests. Fixed.

[krajorama]

lgtm, llm claims some flakiness/cross pollination in test only - I'll open a PR for that afterwards, I need to understand it myself.

Please resolve the remaining small typo/fixe and I can approve.

[krajorama]

LGTM, I've also done a simple smoke test locally. should be good for RC

[jdmarshall]

Done.

What this PR has taught be is that we need a more robust way to simulate workers in single-threaded unit tests. Or we need integration tests for this part of the code. I'm not sure which. But I only thought of better solutions to the problem of handling the lifecycle tests, some of which are in evidence. But they are still not good ones, let alone great ones. They simply suck less.

This code needs additional coverage going forward. And maybe that should be excavating some more blocks of functional code and moving them to Util.