Skip to content

Fixed the Studio view test comparing goldens at the wrong offset - #184

Merged
fdesbiens merged 1 commit into
eclipse-threadx:devfrom
fdesbiens:fix/studio-view-golden-comparison
Aug 27, 2026
Merged

Fixed the Studio view test comparing goldens at the wrong offset#184
fdesbiens merged 1 commit into
eclipse-threadx:devfrom
fdesbiens:fix/studio-view-golden-comparison

Conversation

@fdesbiens

Copy link
Copy Markdown
Contributor

Fixes the 29 failures in GUIX Studio View Test on #183.

What was wrong

compare_file() locates the line where comparison should start, so the banner GUIX Studio writes into every generated file — which carries a revision string and a generation timestamp — is skipped rather than compared. It looked that line up correctly in the first file, then looked it up in the first file again for the second:

    for line in list_2:
        if compare_start_string in line:
            start_row_2 = list_1.index(line)     # ← list_1

So the golden was sliced at the generated file's offset. While both files happened to reach #include on the same line the mistake was invisible, which is why it survived since the tests were added in #84.

The offsets diverged when 6ba13dbd ("Added missing copyright headers") added a ten-line MIT licence header to all 111 golden .c and .h files. GUIX Studio does not emit that header, so every golden now reaches its first #include ten lines later than the file it is compared against. The golden was therefore sliced ten lines early, and banner text got compared against source.

Blast radius, and why it looked like scattered flakiness

Every one of the **107 .c/.h comparisons failed — the whole set, not a sample. They roll up into the 29 reported test failures because a test stops at its first mismatching file.

The .csv, .xliff and .xml comparisons all passed. Those wrappers pass skip_line rather than compare_start_string, so they never reach this code. That split is exactly what the failure log shows.

The reported mismatch is identical in all 29 — a generated #include line against a golden banner comment. Confirmed arithmetically: in every case the golden line reported sits exactly ten lines before that golden's own first #include.

Golden Reported idx Its #include Delta
generic_16bpp_resources.c 13 23 10
generic_16bpp_resources.h 16 26 10
generic_16bpp_specifications.c 14 24 10
generic_16bpp_specifications.h 16 26 10

compare_output_file() is the only caller that passes compare_start_string, so this change cannot affect the xliff, xml or plain-file comparisons.

Verification

Run locally against a Studio built from this tree. Seven suites that fail in CI now pass with no mismatches at all, each gaining exactly the tests it had been failing:

Suite CI (broken) Local (fixed)
Font 9 / 1 10 / 0
Multi-Themes 18 / 1 19 / 0
Project Import 10 / 2 12 / 0
Trigger Edit 6 / 1 7 / 0
Trigger Target Rename 1 / 1 2 / 0
Bidi Text 3 / 1 4 / 0
Widget Name 3 / 1 4 / 0

Project Import compares a generated specifications file, so this also shows the screen flow prototype block added by #173 does not disturb these goldens — that block is only emitted for projects using Screen Flow, and these do not.

No golden file is regenerated

The goldens still carry GUIX Studio Revision 6.1.12.0 and a 2022 timestamp, and still name Azure RTOS rather than Eclipse ThreadX. But the banner is precisely what the comparison is meant to skip, so none of that needs touching to make the suite correct. Worth refreshing one day; not needed here, and not something to do while a release is waiting.

Context

This workflow had never actually executed before 2026-08-27. Every earlier run was cancelled after queueing 24 hours for the retired windows-2019 image, or sat awaiting approval on a fork pull request — including the run for the v6.5.1 release itself. #174 gave it a runner again; this is the first result it has ever produced, and the defect it found is in the test harness rather than in GUIX or in Studio.

The GUI automation itself is fine on a hosted runner: roughly 900 assertions passed, including the CRC32 canvas checks and 246 copy/paste-between-projects tests.

Note on validation

studio_view_test.yml triggers only on pull requests targeting master, so this change is not exercised by its own pull request into dev. It is validated by the release pull request that carries dev to master (#183), which re-runs the suite once this lands.

compare_file() locates the line where comparison should start, so that the
banner GUIX Studio writes into every generated file -- which carries a revision
string and a generation timestamp -- is skipped rather than compared. It looked
that line up correctly in the first file and then looked it up in the first file
again for the second:

    for line in list_2:
        if compare_start_string in line:
            start_row_2 = list_1.index(line)

So the golden was sliced at the *generated* file's offset. While both files
happened to reach "#include" on the same line the mistake was invisible, which
is why it survived since the tests were added in eclipse-threadx#84.

The offsets diverged when 6ba13db added a ten-line MIT licence header to all
111 golden .c and .h files. GUIX Studio does not emit that header, so every
golden now reaches its first #include ten lines later than the file it is
compared against. The golden was therefore sliced ten lines early and compared
banner text against source.

That made every one of the 107 .c and .h comparisons in the suite fail -- all 29
reported test failures -- while the .csv, .xliff and .xml comparisons passed.
Those use skip_line instead of compare_start_string and never reach this code,
which is exactly the split the failure log shows.

The reported mismatch is the same in all 29: a generated "#include" line against
a golden banner comment. Confirmed arithmetically -- in every case the golden
line reported sits exactly ten lines before that golden's own first #include:

    generic_16bpp_resources.c        reported idx 13, #include at 23, delta 10
    generic_16bpp_resources.h        reported idx 16, #include at 26, delta 10
    generic_16bpp_specifications.c   reported idx 14, #include at 24, delta 10
    generic_16bpp_specifications.h   reported idx 16, #include at 26, delta 10

compare_output_file() is the only caller that passes compare_start_string, so
the change cannot affect the xliff, xml or plain-file comparisons.

Verified by running the suite locally against a Studio built from this tree.
Seven suites that fail in CI now pass with no mismatches at all, each gaining
exactly the tests it had been failing:

                              CI (broken)   local (fixed)
    Font                       9 / 1         10 / 0
    Multi-Themes              18 / 1         19 / 0
    Project Import            10 / 2         12 / 0
    Trigger Edit               6 / 1          7 / 0
    Trigger Target Rename      1 / 1          2 / 0
    Bidi Text                  3 / 1          4 / 0
    Widget Name                3 / 1          4 / 0

Project Import compares a generated specifications file, so this also shows that
the screen flow prototype block added by eclipse-threadx#173 does not disturb these goldens:
that block is only emitted for projects that use Screen Flow, and these do not.

No golden file is regenerated. The goldens still carry "GUIX Studio Revision
6.1.12.0" and a 2022 timestamp in their banners, and still name Azure RTOS
rather than Eclipse ThreadX, but the banner is what the comparison is supposed
to skip -- so none of that needs to be touched to make the suite correct.

This workflow had never actually executed before 2026-08-27. Every earlier run
was cancelled after queueing 24 hours for the retired windows-2019 image, or
sat awaiting approval on a fork pull request, including the run for the v6.5.1
release itself. eclipse-threadx#174 gave it a runner again; this is the first result it has
ever produced, and the defect it found is in the test harness rather than in
GUIX or in Studio.

Note that studio_view_test.yml triggers only on pull requests targeting master,
so this change is not exercised by its own pull request into dev. It is
validated by the release pull request that carries dev to master.

Assisted-by: Claude Code (Opus 5)

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@fdesbiens
fdesbiens merged commit e2d28f5 into eclipse-threadx:dev Aug 27, 2026
3 checks passed
@fdesbiens
fdesbiens deleted the fix/studio-view-golden-comparison branch August 27, 2026 20:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant