Broadcast Kernel Fusion

[ejmeitz]

This PR is implements the automatic fusion of broadcast expressions with unary & binary operations. Really any intrinsic you could use in a CUDA kernel.

y .= sin.(x)
z = x .+ y .* z
w = x .+ y .+ 2.0f0

# Would be nice to support:
f(x,y) -> x + y
z .= 2.0f0 .* f.(x, y)

Goals

• Support broadcast expressions with arbitrarily many input/output NDArrays that leverage any binary/unary op and may be combined with some number of scalar arguments.
• Some kind of fall back logic if we cannot fuse to just use the un-fused kernel and emit a warning.
• Support broadcasting of arbitrary user defined functions. There is not even an un-fused code path right now. This might "just work" as CUDA.jl should handle this for us.

High Level Implementation

• Generate key from broadcast expression, check cache.
• If not in cache: Generate PTX for the broadcast kernel using CUDA.jl by spoofing the NDArrays as CuDeviceArrays
• Launch the LoadPTXTask and RunPTXTask legate tasks defined in cuda.cpp with the registered kernel

This PR does not aim to support fusion of functions. That will be a future PR.