Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 59 additions & 13 deletions flaky-tests/dashboard.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,15 +42,14 @@ Test Deletion & History

### Test case details

<figure><picture><source srcset="../.gitbook/assets/flaky-tests-failure-details-dark.png" media="(prefers-color-scheme: dark)"><img src="../.gitbook/assets/flaky-tests-failure-details-light.png" alt=""></picture><figcaption></figcaption></figure>

You can _click_ on any of the test cases listed on the Flaky Test dashboard to access the test case’s details. On a test's details page, you can find:
You can _click_ on any of the test cases listed on the Flaky Test dashboard to access the test case’s details. The test details page uses a tabbed layout with the following tabs:

* The test's current status
* Visualizations and a timeline detailing the test's health history
* A table of unique failure types for this test
* **Summary** — Run result charts and failure types grouped by unique failure reason.
* **Test History** — A searchable, paginated table of every individual test run with filtering and a detail panel.
* **Monitors** — Detection monitors configured for this test (visible when the detection engine is enabled).
* **Events** — Detection events and status changes for this test (visible when the detection engine is enabled).

This is in addition to information like ticket status and the current codeowner.
In addition to the tabbed content, the test details page shows the test’s current status, ticket status, and codeowner information.

### **Code owners**

Expand All @@ -60,30 +59,77 @@ If you have a codeowners file configured in your repos, you will see who owns ea

This information will also be provided when creating a ticket with the [Jira integration](ticketing-integrations/jira-integration.md) or [webhooks](webhooks/).

### **Failure types**
### Summary tab

<figure><picture><source srcset="../.gitbook/assets/flaky-tests-failure-details-dark.png" media="(prefers-color-scheme: dark)"><img src="../.gitbook/assets/flaky-tests-failure-details-light.png" alt=""></picture><figcaption></figcaption></figure>

The Summary tab shows an overview of the test’s recent run results and groups past failures by unique failure type.

#### **Failure types**

<figure><picture><source srcset="../.gitbook/assets/unique-failure-reason-dark.png" media="(prefers-color-scheme: dark)"><img src="../.gitbook/assets/unique-failure-reason-light.png" alt=""></picture><figcaption></figcaption></figure>

The Failure Types table shows the history of past test runs grouped by unique failure types.

The Failure Type is a summary of the stack trace of the test run. You can click on the failure type to see a list of test runs labeled by branch, PR, Author, CI Job link, duration, and time.

### Failure details
#### Failure details

You can click on any of these test runs to see the detailed stack trace:

<figure><picture><source srcset="../.gitbook/assets/run-details-dark.png" media="(prefers-color-scheme: dark)"><img src="../.gitbook/assets/run-details-light.png" alt=""></picture><figcaption></figcaption></figure>

You can flip through the stack traces of similar failures across different test runs by clicking the left and right arrow buttons. You can also see other similar failures on this and other tests.

#### Go to the CI job logs
##### Go to the CI job logs

If you want to see full logging of the original CI job for an individual test failure, you can click **Logs** in the expanded failure details panel to go to the job's page in your CI provider.
If you want to see full logging of the original CI job for an individual test failure, you can click **Logs** in the expanded failure details panel to go to the jobs page in your CI provider.

<figure><picture><source srcset="../.gitbook/assets/failure-logs-dark.png" media="(prefers-color-scheme: dark)"><img src="../.gitbook/assets/failure-logs-light.png" alt=""></picture><figcaption></figcaption></figure>

### **Test history**
### Test History tab

The Test History tab gives you full visibility into every individual run of a test. Use it to investigate patterns across branches, find specific failing runs, and drill into error details.

#### Daily runs chart

A stacked bar chart at the top of the tab shows daily test run counts broken down by result:

* **Green** — Pass
* **Red** — Fail
* **Blue** — Quarantined
* **Gray** — Skipped

You can click and drag on the chart to select a date range, which filters the table below to only show runs from the selected days. Click the **Reset** button or the clear icon on the chart to remove the date range selection.

#### Filters

A filter bar below the chart lets you narrow down the runs table:

<table><thead><tr><th width="188">Filter</th><th>Description</th></tr></thead><tbody><tr><td>Result</td><td>Toggle between <strong>All</strong>, <strong>Pass</strong>, <strong>Fail</strong>, and <strong>Quarantined</strong> to show only runs with a specific outcome.</td></tr><tr><td>SHA</td><td>Filter by commit hash. Matches runs whose SHA starts with the entered text.</td></tr><tr><td>Branch</td><td>Filter by branch name. Matches runs on branches containing the entered text.</td></tr></tbody></table>

All filters combine using AND logic, so you can use them together to find specific runs. For example, select **Fail** and enter a branch name to see all failures on that branch. Click the **Reset** button to clear all active filters.

Filter state is saved in the URL, so you can share or bookmark a filtered view.

#### Runs table

The runs table displays a paginated list of individual test runs (25 per page) with the following columns:

<table><thead><tr><th width="188">Column</th><th>Description</th></tr></thead><tbody><tr><td>Timestamp</td><td>When the test ran, displayed in UTC.</td></tr><tr><td>Duration</td><td>How long the test took to execute.</td></tr><tr><td>Branch</td><td>The branch context. Shows "Mergequeue testing #NNN" for merge queue runs, "#NNN" for pull request runs, or the branch name for other runs.</td></tr><tr><td>Commit</td><td>The first 7 characters of the commit SHA.</td></tr></tbody></table>

Each row has a colored left border indicating the result: green for pass, red for fail, blue for quarantined, and orange for error.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The chart color legend above (lines 98–101) lists Gray = Skipped, but this row-border description mentions orange for error without including gray/skipped. The two color references are inconsistent — readers may wonder why the row borders and chart use different color sets.

Consider aligning them, e.g.:

Suggested change
Each row has a colored left border indicating the result: green for pass, red for fail, blue for quarantined, and orange for error.
Each row has a colored left border indicating the result: green for pass, red for fail, blue for quarantined, gray for skipped, and orange for error.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The daily runs chart legend above (lines ~93–101) includes Gray = Skipped, but gray/skipped is missing from this row border color list. Consider adding it for consistency:

Suggested change
Each row has a colored left border indicating the result: green for pass, red for fail, blue for quarantined, and orange for error.
Each row has a colored left border indicating the result: green for pass, red for fail, blue for quarantined, gray for skipped, and orange for error.


#### Run detail panel

Click any row in the runs table to open a detail panel on the right side of the page. The panel shows:

* **Run header** — Timestamp, result badge, and duration.
* **Source control** — Links to the commit, pull request, and branch.
* **Error details** — For failed or errored runs, the error message and stack trace.

### Status history

<figure><picture><source srcset="../.gitbook/assets/test-history-dark.png" media="(prefers-color-scheme: dark)"><img src="../.gitbook/assets/test-history-light.png" alt=""></picture><figcaption></figcaption></figure>

Tests may transition between flaky, broken, and healthy states multiple times over their lifetime. You can see previous changes in the detected health status of a test in Test History, as well as an explanation for why it was detected to have a new state.
Tests may transition between flaky, broken, and healthy states multiple times over their lifetime. You can see previous changes in the detected health status of a test, as well as an explanation for why it was detected to have a new state.
2 changes: 1 addition & 1 deletion flaky-tests/quarantining.md
Original file line number Diff line number Diff line change
Expand Up @@ -201,7 +201,7 @@ When a manual override is active, a banner shows who set it and when.

<figure><img src="../.gitbook/assets/qurantine-individual-tests-revert.png" alt=""><figcaption></figcaption></figure>

To review a history of all quarantine changes on a test, use the **Quarantine Events** filter within the **Test History** section. This will show every override, setting change, and comment, along with the author and timestamp for each entry.
To review a history of all quarantine changes on a test, check the **Events** tab on the test details page. You can also use the **Quarantined** result filter in the **Test History** tab to see all quarantined runs. These views show every override, setting change, and comment, along with the author and timestamp for each entry.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The last sentence "These views show every override, setting change, and comment..." conflates two tabs that show different things: the Events tab shows overrides/setting changes/comments, while the Test History tab (filtered to Quarantined) shows runs. Consider splitting for clarity:

Suggested change
To review a history of all quarantine changes on a test, check the **Events** tab on the test details page. You can also use the **Quarantined** result filter in the **Test History** tab to see all quarantined runs. These views show every override, setting change, and comment, along with the author and timestamp for each entry.
To review a history of all quarantine changes on a test, check the **Events** tab on the test details page. You can also use the **Quarantined** result filter in the **Test History** tab to see all quarantined runs. The Events tab shows every override, setting change, and comment, along with the author and timestamp for each entry.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The closing sentence "These views show every override, setting change, and comment..." still conflates the two tabs: the Events tab shows overrides/setting changes/comments, while the Test History tab only shows runs. Applying the cleaner split from the previous suggestion:

Suggested change
To review a history of all quarantine changes on a test, check the **Events** tab on the test details page. You can also use the **Quarantined** result filter in the **Test History** tab to see all quarantined runs. These views show every override, setting change, and comment, along with the author and timestamp for each entry.
To review a history of all quarantine changes on a test, check the **Events** tab on the test details page. You can also use the **Quarantined** result filter in the **Test History** tab to see all quarantined runs. The Events tab shows every override, setting change, and comment, along with the author and timestamp for each entry.


### Tracking quarantined jobs in the dashboard

Expand Down
Loading