[AIMIGRAPHX-1100] Add no-rebuild callback for verify - #5067
Draft
eddieliao wants to merge 10 commits into
Draft
Conversation
eddieliao
added
enhancement
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #5067 +/- ##
===========================================
+ Coverage 92.89% 93.26% +0.37%
===========================================
Files 603 623 +20
Lines 32448 32969 +521
===========================================
+ Hits 30140 30747 +607
+ Misses 2308 2222 -86 🚀 New features to boost your workflow:
|
Regressions detected 🔴 * No develop baseline was found for this PR's branch point; compared against the latest available develop run instead. |
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a --no-rebuild verification mode to the MIGraphX driver that performs a single reference run and a single target run while tracing instruction outputs by debug symbol, enabling layer-wise divergence reporting without recompiling for --reduce/--bisect workflows.
Changes:
- Adds
verify_options::no_rebuildand wires--no-rebuildinto the driver CLI. - Introduces a trace-based
verify_callbackthat captures reference outputs by debug symbol and compares them against target outputs. - Updates verify flows (
verify_program, reduced, bisected) to use the new layer-wise compare path when--no-rebuildis enabled.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| src/driver/verify.hpp | Declares verify_callback and exposes layer-wise compare support from the driver verify interface. |
| src/driver/verify.cpp | Implements trace-based capture/compare, baseline error attribution, and integrates --no-rebuild into verify/reduce/bisect modes. |
| src/driver/verify_options.hpp | Adds no_rebuild option to the verify options struct. |
| src/driver/main.cpp | Adds --no-rebuild CLI flag and help text for verify. |
Comment on lines
+296
to
+298
| // Last-writer-wins on the symbol keeps the fused op's final value, not an interior one. | ||
| target_outputs[symbol] = {output.copy(), ins->name(), order, std::move(inputs)}; | ||
| }; |
Comment on lines
+451
to
+462
| bool any_failed = false; | ||
| for(const auto& [symbol, lr] : vcb->results) | ||
| { | ||
| if(not lr.passed) | ||
| { | ||
| any_failed = true; | ||
| log::error() << "FAILED at " << lr.symbol << " (" << lr.op << ")"; | ||
| } | ||
| } | ||
| if(not any_failed) | ||
| log::info() << "MIGraphX verification passed successfully."; | ||
| return; |
Comment on lines
+49
to
+53
| // Captures ref outputs by debug symbol and compares each target op at its terminal symbol. | ||
| struct verify_callback | ||
| { | ||
| using trace_function = std::function<void(instruction_ref, const argument&)>; | ||
|
|
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.
Suppressed comments (4)
src/driver/verify.cpp:177
- Returning
nulloptfor every lens mismatch hides actual target shape regressions. If a symbol's target output has the wrong dimensions, that layer is omitted fromresults; other layers can still pass and the command reports overall success. Distinguish inherited-symbol intermediates from the terminal output, and record a failed shape comparison when the terminal symbol has no shape-compatible target result.
// Reshapes and slices inherit the symbol; only the op whose lens match produced it.
if(not shape::same_lens(ref.output.get_shape(), output.get_shape()))
return nullopt;
src/driver/verify.cpp:304
- This helper can return successfully with zero comparisons. For example, a constant-only model has debug symbols on literals/return but neither is inspectable, and a
--compiled-modelwithout corresponding symbols also leavesresultsempty. Both callers then treatfailures().empty()as verification success. Reject an empty result set instead of reporting a pass.
log::info() << "Layers compared: " << vcb.results.size();
return vcb;
src/program.cpp:733
substituteis a public callback, so its return value is an external API boundary. This debug-only assertion disappears in release builds, allowing an incompatible argument to reach downstream operators. Validate the replacement shape at runtime and throw a clear error.
assert(sub->get_shape() == result.get_shape());
src/include/migraphx/execution_environment.hpp:42
- The new substitution execution path and layer-wise verifier have no regression tests in this PR, although adjacent evaluation callback behavior is covered in
test/trace_eval_test.cppandtest/api/test_trace_callback.cpp. Add tests proving a replacement is consumed by downstream instructions and covering no-match, shape-mismatch, and multi-output debug-symbol cases; these cases currently permit false passes or incorrect substitutions.
// Replaces an instruction's result, so later instructions read the returned value instead.
std::function<optional<argument>(instruction_ref, const argument&)> substitute = nullptr;
Comment on lines
+143
to
+144
| for(const auto& symbol : ins->get_debug_symbols()) | ||
| ref_outputs[symbol] = {buf, order}; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Technical Details
Changelog Category
Add a
CHANGELOG.mdentry for any option other thanNot ApplicableFollow the LLVM AI Tool Use Policy for contributions using AI.