Get result heimdall task

[Will Graham (wlggraham)]

Description

Some Heimdall payloads do not produce a result, but the current Caterpillar Heimdall Task implementation always hits the /result API. This causes issues with certain payloads, particularly SQL DDL.

This PR introduces a new field GetResult, which determines whether or not we want to hit the /result endpoint after completion of a job. This is intentionally different from the ReturnResult which has different behaviors depending on how the heimdall plugin was written. For example, Trino ALWAYS returns a result, regardless of whether or not ReturnResult is used. Because of this, having a separate indicator for how we poll the heimdall API is essential.

To test this, I tested the following pipeline:

  - name: query_with_result
    type: heimdall
    fail_on_error: true
    get_result: false
    job:
      command_criteria:
        - type:sparksql-eks
      cluster_criteria:
        - data:prod
      context:
        query: SHOW databases;
        return_result: true

  - name: echo
    type: echo
    only_data: true 

Both with get_result and without it.

Types of changes

• Docs change / refactoring / dependency upgrade
• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist

• My code follows the code style of this project.
• My change requires a change to the documentation and I have updated the documentation accordingly.
• I have added tests to cover my changes.