-
Notifications
You must be signed in to change notification settings - Fork 13
[TRUNK-17757] Document Test History tab on Flaky Tests test detail page #481
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -42,15 +42,14 @@ Test Deletion & History | |||||
|
|
||||||
| ### Test case details | ||||||
|
|
||||||
| <figure><picture><source srcset="../.gitbook/assets/flaky-tests-failure-details-dark.png" media="(prefers-color-scheme: dark)"><img src="../.gitbook/assets/flaky-tests-failure-details-light.png" alt=""></picture><figcaption></figcaption></figure> | ||||||
|
|
||||||
| You can _click_ on any of the test cases listed on the Flaky Test dashboard to access the test case’s details. On a test's details page, you can find: | ||||||
| You can _click_ on any of the test cases listed on the Flaky Test dashboard to access the test case’s details. The test details page uses a tabbed layout with the following tabs: | ||||||
|
|
||||||
| * The test's current status | ||||||
| * Visualizations and a timeline detailing the test's health history | ||||||
| * A table of unique failure types for this test | ||||||
| * **Summary** — Run result charts and failure types grouped by unique failure reason. | ||||||
| * **Test History** — A searchable, paginated table of every individual test run with filtering and a detail panel. | ||||||
| * **Monitors** — Detection monitors configured for this test (visible when the detection engine is enabled). | ||||||
| * **Events** — Detection events and status changes for this test (visible when the detection engine is enabled). | ||||||
|
|
||||||
| This is in addition to information like ticket status and the current codeowner. | ||||||
| In addition to the tabbed content, the test details page shows the test’s current status, ticket status, and codeowner information. | ||||||
|
|
||||||
| ### **Code owners** | ||||||
|
|
||||||
|
|
@@ -60,30 +59,77 @@ If you have a codeowners file configured in your repos, you will see who owns ea | |||||
|
|
||||||
| This information will also be provided when creating a ticket with the [Jira integration](ticketing-integrations/jira-integration.md) or [webhooks](webhooks/). | ||||||
|
|
||||||
| ### **Failure types** | ||||||
| ### Summary tab | ||||||
|
|
||||||
| <figure><picture><source srcset="../.gitbook/assets/flaky-tests-failure-details-dark.png" media="(prefers-color-scheme: dark)"><img src="../.gitbook/assets/flaky-tests-failure-details-light.png" alt=""></picture><figcaption></figcaption></figure> | ||||||
|
|
||||||
| The Summary tab shows an overview of the test’s recent run results and groups past failures by unique failure type. | ||||||
|
|
||||||
| #### **Failure types** | ||||||
|
|
||||||
| <figure><picture><source srcset="../.gitbook/assets/unique-failure-reason-dark.png" media="(prefers-color-scheme: dark)"><img src="../.gitbook/assets/unique-failure-reason-light.png" alt=""></picture><figcaption></figcaption></figure> | ||||||
|
|
||||||
| The Failure Types table shows the history of past test runs grouped by unique failure types. | ||||||
|
|
||||||
| The Failure Type is a summary of the stack trace of the test run. You can click on the failure type to see a list of test runs labeled by branch, PR, Author, CI Job link, duration, and time. | ||||||
|
|
||||||
| ### Failure details | ||||||
| #### Failure details | ||||||
|
|
||||||
| You can click on any of these test runs to see the detailed stack trace: | ||||||
|
|
||||||
| <figure><picture><source srcset="../.gitbook/assets/run-details-dark.png" media="(prefers-color-scheme: dark)"><img src="../.gitbook/assets/run-details-light.png" alt=""></picture><figcaption></figcaption></figure> | ||||||
|
|
||||||
| You can flip through the stack traces of similar failures across different test runs by clicking the left and right arrow buttons. You can also see other similar failures on this and other tests. | ||||||
|
|
||||||
| #### Go to the CI job logs | ||||||
| ##### Go to the CI job logs | ||||||
|
|
||||||
| If you want to see full logging of the original CI job for an individual test failure, you can click **Logs** in the expanded failure details panel to go to the job's page in your CI provider. | ||||||
| If you want to see full logging of the original CI job for an individual test failure, you can click **Logs** in the expanded failure details panel to go to the job’s page in your CI provider. | ||||||
|
|
||||||
| <figure><picture><source srcset="../.gitbook/assets/failure-logs-dark.png" media="(prefers-color-scheme: dark)"><img src="../.gitbook/assets/failure-logs-light.png" alt=""></picture><figcaption></figcaption></figure> | ||||||
|
|
||||||
| ### **Test history** | ||||||
| ### Test History tab | ||||||
|
|
||||||
| The Test History tab gives you full visibility into every individual run of a test. Use it to investigate patterns across branches, find specific failing runs, and drill into error details. | ||||||
|
|
||||||
| #### Daily runs chart | ||||||
|
|
||||||
| A stacked bar chart at the top of the tab shows daily test run counts broken down by result: | ||||||
|
|
||||||
| * **Green** — Pass | ||||||
| * **Red** — Fail | ||||||
| * **Blue** — Quarantined | ||||||
| * **Gray** — Skipped | ||||||
|
|
||||||
| You can click and drag on the chart to select a date range, which filters the table below to only show runs from the selected days. Click the **Reset** button or the clear icon on the chart to remove the date range selection. | ||||||
|
|
||||||
| #### Filters | ||||||
|
|
||||||
| A filter bar below the chart lets you narrow down the runs table: | ||||||
|
|
||||||
| <table><thead><tr><th width="188">Filter</th><th>Description</th></tr></thead><tbody><tr><td>Result</td><td>Toggle between <strong>All</strong>, <strong>Pass</strong>, <strong>Fail</strong>, and <strong>Quarantined</strong> to show only runs with a specific outcome.</td></tr><tr><td>SHA</td><td>Filter by commit hash. Matches runs whose SHA starts with the entered text.</td></tr><tr><td>Branch</td><td>Filter by branch name. Matches runs on branches containing the entered text.</td></tr></tbody></table> | ||||||
|
|
||||||
| All filters combine using AND logic, so you can use them together to find specific runs. For example, select **Fail** and enter a branch name to see all failures on that branch. Click the **Reset** button to clear all active filters. | ||||||
|
|
||||||
| Filter state is saved in the URL, so you can share or bookmark a filtered view. | ||||||
|
|
||||||
| #### Runs table | ||||||
|
|
||||||
| The runs table displays a paginated list of individual test runs (25 per page) with the following columns: | ||||||
|
|
||||||
| <table><thead><tr><th width="188">Column</th><th>Description</th></tr></thead><tbody><tr><td>Timestamp</td><td>When the test ran, displayed in UTC.</td></tr><tr><td>Duration</td><td>How long the test took to execute.</td></tr><tr><td>Branch</td><td>The branch context. Shows "Mergequeue testing #NNN" for merge queue runs, "#NNN" for pull request runs, or the branch name for other runs.</td></tr><tr><td>Commit</td><td>The first 7 characters of the commit SHA.</td></tr></tbody></table> | ||||||
|
|
||||||
| Each row has a colored left border indicating the result: green for pass, red for fail, blue for quarantined, and orange for error. | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The daily runs chart legend above (lines ~93–101) includes Gray = Skipped, but gray/skipped is missing from this row border color list. Consider adding it for consistency:
Suggested change
|
||||||
|
|
||||||
| #### Run detail panel | ||||||
|
|
||||||
| Click any row in the runs table to open a detail panel on the right side of the page. The panel shows: | ||||||
|
|
||||||
| * **Run header** — Timestamp, result badge, and duration. | ||||||
| * **Source control** — Links to the commit, pull request, and branch. | ||||||
| * **Error details** — For failed or errored runs, the error message and stack trace. | ||||||
|
|
||||||
| ### Status history | ||||||
|
|
||||||
| <figure><picture><source srcset="../.gitbook/assets/test-history-dark.png" media="(prefers-color-scheme: dark)"><img src="../.gitbook/assets/test-history-light.png" alt=""></picture><figcaption></figcaption></figure> | ||||||
|
|
||||||
| Tests may transition between flaky, broken, and healthy states multiple times over their lifetime. You can see previous changes in the detected health status of a test in Test History, as well as an explanation for why it was detected to have a new state. | ||||||
| Tests may transition between flaky, broken, and healthy states multiple times over their lifetime. You can see previous changes in the detected health status of a test, as well as an explanation for why it was detected to have a new state. | ||||||
| Original file line number | Diff line number | Diff line change | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -201,7 +201,7 @@ When a manual override is active, a banner shows who set it and when. | |||||||||
|
|
||||||||||
| <figure><img src="../.gitbook/assets/qurantine-individual-tests-revert.png" alt=""><figcaption></figcaption></figure> | ||||||||||
|
|
||||||||||
| To review a history of all quarantine changes on a test, use the **Quarantine Events** filter within the **Test History** section. This will show every override, setting change, and comment, along with the author and timestamp for each entry. | ||||||||||
| To review a history of all quarantine changes on a test, check the **Events** tab on the test details page. You can also use the **Quarantined** result filter in the **Test History** tab to see all quarantined runs. These views show every override, setting change, and comment, along with the author and timestamp for each entry. | ||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The last sentence "These views show every override, setting change, and comment..." conflates two tabs that show different things: the Events tab shows overrides/setting changes/comments, while the Test History tab (filtered to Quarantined) shows runs. Consider splitting for clarity:
Suggested change
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The closing sentence "These views show every override, setting change, and comment..." still conflates the two tabs: the Events tab shows overrides/setting changes/comments, while the Test History tab only shows runs. Applying the cleaner split from the previous suggestion:
Suggested change
|
||||||||||
|
|
||||||||||
| ### Tracking quarantined jobs in the dashboard | ||||||||||
|
|
||||||||||
|
|
||||||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The chart color legend above (lines 98–101) lists Gray = Skipped, but this row-border description mentions orange for error without including gray/skipped. The two color references are inconsistent — readers may wonder why the row borders and chart use different color sets.
Consider aligning them, e.g.: