Add `--tag-filters` option to filter tests using `execution_tag` marker by nprizal · Pull Request #83 · buildkite/test-collector-python

nprizal · 2026-01-22T02:39:57Z

Test Engine Tags can be applied to a test using @pytest.mark.execution_tag(“key”, “value”). However, pytest built-in -m option doesn’t support filtering tests using marker arguments. It can filter tests that have any execution_tag markers, but it can't filter tests that have execution_tag marker with specific key and value.

This PR adds new custom options --tag-filters to filter tests based on execution_tag'skey & value.

# test_foobar.py
import pytest

@pytest.mark.execution_tag("color", "red")
def test_foo()
 assert True

@pytest.mark.execution_tag("color", "blue")
def test_bar()
  assert True

Then to filter tests with execution tag

$ pytest --tag-filters "color:blue" -v
=========================== test session starts ============================
platform darwin -- Python 3.13.2, pytest-9.0.2, pluggy-1.6.0
rootdir: /my/project
plugins: buildkite-test-collector-1.2.0
collected 2 items / 1 deselected / 1 selected

test_foobar.py::test_bar PASSED                                       [100%]

===================== 1 passed, 1 deselected in 0.1s =======================

nprizal · 2026-01-22T03:55:06Z

src/buildkite_test_collector/pytest_plugin/__init__.py

    if plugin:
        api = API(os.environ)
-        numprocesses = config.getoption("numprocesses")
+        numprocesses = config.getoption("numprocesses", None)


One of the tests I added failed in CI because numprocesses option doesn't exists in the subprocess environment. I added None as default to prevent error being raised when evaluation that option.

Add a new --tag-filters CLI flag that allows filtering tests by execution tags. This is useful for selectively running subsets of tests. Currently only Pytest is supported via markers [1]. When specified, the tag filter is passed to pytest via the -m flag (e.g., pytest -m "test_execution"). We will replace the default marker filter with a custom filter option implemented in the Python test collector [2] Changes: - Add --tag-filters flag and BUILDKITE_TEST_ENGINE_TAG_FILTERS env var - Add TagFilters field to Config struct - Update Pytest runner to use -m flag when tag filters are specified. (This will be replaced with a custom filter) - Add test coverage for marker-based filtering for pytest Example usage: bktec run --tag-filters "slow" BUILDKITE_TEST_ENGINE_TAG_FILTERS="not integration" bktec run [1] https://docs.pytest.org/en/7.1.x/example/markers.html [2] buildkite/test-collector-python#83

gchan · 2026-01-25T22:23:41Z

src/buildkite_test_collector/pytest_plugin/__init__.py

        dest="mergejson",
        help='merge json output with existing file, if it exists'
    )
+    group.addoption(


Additional context
https://docs.pytest.org/en/7.1.x/example/markers.html#custom-marker-and-command-line-option-to-control-test-runs

gchan · 2026-01-25T22:41:13Z

tests/buildkite_test_collector/test_integration.py

+
+from pathlib import Path
+
+def test_add_tag_to_execution_data(tmp_path, fake_env):


Nice job backfilling a test

gchan

LGTM.

I had thoughts about : as the separator between a tag key and value. My suggestion was = but it doesn't matter as both characters are not permitted characters in a key.

The integration test proves test examples can be filtered/collected from one file. There is no reason for me to believe this doesn't extend to multiple files with similar behaviour to the built in marker selector EDIT: I can confirm it works as expected with multiple files!

Add a new --tag-filters CLI flag that allows filtering tests by execution tags. This is useful for selectively running subsets of tests. Currently only Pytest is supported via markers [1]. When specified, the tag filter is passed to pytest via the -m flag (e.g., pytest -m "test_execution"). We will replace the default marker filter with a custom filter option implemented in the Python test collector [2] Changes: - Add --tag-filters flag and BUILDKITE_TEST_ENGINE_TAG_FILTERS env var - Add TagFilters field to Config struct - Update Pytest runner to use -m flag when tag filters are specified. (This will be replaced with a custom filter) - Add test coverage for marker-based filtering for pytest Example usage: bktec run --tag-filters "slow" BUILDKITE_TEST_ENGINE_TAG_FILTERS="not integration" bktec run [1] https://docs.pytest.org/en/7.1.x/example/markers.html [2] buildkite/test-collector-python#83

gchan · 2026-01-26T02:23:44Z

I've integrated this change with bktec and tested it works correctly. It's good to merge! 👍

Ref: buildkite/test-engine-client#426

nprizal · 2026-01-28T21:33:22Z

I had thoughts about : as the separator between a tag key and value. My suggestion was = but it doesn't matter as both characters are not permitted characters in a key.

The reason I pick : is to be consistent with the tag filter in the UI.

nprizal force-pushed the te-5135-add-custom-option-to-filter-tests-by-execution-tag-in-pytest branch from af0becc to 07121b3 Compare January 22, 2026 02:47

nprizal added 2 commits January 22, 2026 15:52

Add integration test for execution_tag marker

40ea70a

Add --tag-filters option

bf552fa

nprizal force-pushed the te-5135-add-custom-option-to-filter-tests-by-execution-tag-in-pytest branch from 07121b3 to bf552fa Compare January 22, 2026 02:52

nprizal added 2 commits January 22, 2026 16:12

debug subprocess

bc265f0

Set default when getting numprocess option

c64ec8e

nprizal commented Jan 22, 2026

View reviewed changes

nprizal changed the title ~~Add --tag-filters option to filter tests using execution_tag marker~~ Add --tag-filters option to filter tests using execution_tag marker Jan 22, 2026

nprizal self-assigned this Jan 22, 2026

nprizal requested a review from gchan January 22, 2026 04:23

nprizal marked this pull request as ready for review January 22, 2026 04:23

gchan mentioned this pull request Jan 23, 2026

[TE-5134] Add --tag-filters option to filter tests by markers buildkite/test-engine-client#426

Merged

gchan reviewed Jan 25, 2026

View reviewed changes

gchan approved these changes Jan 25, 2026

View reviewed changes

gchan merged commit b064b24 into main Jan 26, 2026
10 checks passed

gchan deleted the te-5135-add-custom-option-to-filter-tests-by-execution-tag-in-pytest branch January 26, 2026 03:05

This was referenced Jan 26, 2026

fix: report teardown failures as failed tests #85

Closed

fix: report teardown failures as failed tests #86

Merged

[release] Version 1.3.0 #87

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `--tag-filters` option to filter tests using `execution_tag` marker#83

Add `--tag-filters` option to filter tests using `execution_tag` marker#83
gchan merged 4 commits intomainfrom
te-5135-add-custom-option-to-filter-tests-by-execution-tag-in-pytest

nprizal commented Jan 22, 2026 •

edited

Loading

Uh oh!

nprizal Jan 22, 2026 •

edited

Loading

Uh oh!

gchan Jan 25, 2026

Uh oh!

gchan Jan 25, 2026

Uh oh!

gchan left a comment •

edited

Loading

Uh oh!

gchan commented Jan 26, 2026

Uh oh!

Uh oh!

nprizal commented Jan 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		from pathlib import Path

		def test_add_tag_to_execution_data(tmp_path, fake_env):

Conversation

nprizal commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nprizal Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gchan Jan 25, 2026

Choose a reason for hiding this comment

Uh oh!

gchan Jan 25, 2026

Choose a reason for hiding this comment

Uh oh!

gchan left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gchan commented Jan 26, 2026

Uh oh!

Uh oh!

nprizal commented Jan 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

nprizal commented Jan 22, 2026 •

edited

Loading

nprizal Jan 22, 2026 •

edited

Loading

gchan left a comment •

edited

Loading