Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/version_scanner.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ jobs:
# Uses -o to output a detailed, raw CSV to a file
# Uses --stdout to print a slim, easier to parse summary to the GitHub Actions UI
# Uses --soft-fail to temporarily limit causing CI/CD failures during the migration to full operation.
python scripts/version_scanner/version_scanner.py -d python -v 3.7 --stdout -o version_scanner_output.csv --soft-fail
python scripts/version_scanner/version_scanner.py --matrix-file scripts/version_scanner/matrix.yaml --package-file scripts/version_scanner/example-list-non-generated-packages.txt --stdout -o version_scanner_output.csv --soft-fail
- name: Upload CSV Results
if: always()
Expand Down
28 changes: 24 additions & 4 deletions scripts/version_scanner/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,12 +45,32 @@ pip install -r scripts/version_scanner/requirements.txt

The scanner uses a YAML configuration file (`regex_config.yaml`) to define rules and regex patterns.

## Ignoring Directories
## Matrix File Format

You can create a `.scannerignore` file in the directory you are scanning (usually the repo root) to list directories to skip, one per line.
When using `--matrix-file`, you must provide a YAML file specifying dependencies and versions.

## Known Issues & Future Investigations
- **Binary Ignores in `.scannerignore`**: Recursive wildcard ignores (e.g., `*.jpg`) currently do not effectively ignore deeply nested binary files. The scanner logic should be investigated to support robust globbing or full-path suffix matching.
### Example
```yaml
python:
- "3.10"
- "3.11"
protobuf: "4.25.8"
```

> [!IMPORTANT]
> **Versions must be specified as quoted strings** (e.g., `"3.10"`, not `3.10`). This prevents YAML parsers from converting them to floats (which would truncate `3.10` to `3.1`).

## Ignoring Directories and Files

In order to ignore files OR entire directories, you can add ignore patterns to the `.scannerignore` file located in the same directory as the script (`scripts/version_scanner/.scannerignore`). Ignore patterns should be added one per line.

### Features
- **Case-insensitive**: All patterns are matched case-insensitively.
- **Globbing**: Supports standard shell globbing patterns (e.g., `*.jpg`, `test_*`).
- **Subpaths**: You can specify subpaths (e.g., `packages/pkg_a/.nox`).
- **Root Anchoring**: Patterns starting with a slash `/` are anchored to the root of the scan (e.g., `/packages` ignores the `packages` directory at root, but not `some/other/packages`).

---

---

Expand Down
31 changes: 31 additions & 0 deletions scripts/version_scanner/example-list-non-generated-packages.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
packages/bigframes
packages/bigquery-magics
packages/db-dtypes
packages/django-google-spanner
packages/gapic-generator
packages/google-api-core
# packages/google-api-python-client # non-monorepo, ignore for now.
packages/google-auth
packages/google-auth-httplib2
packages/google-auth-oauthlib
packages/google-cloud-bigquery
packages/pandas-gbq
packages/google-cloud-bigtable
packages/google-cloud-core
packages/google-crc32c
packages/google-cloud-datastore
packages/google-cloud-dns
packages/google-cloud-documentai-toolbox
packages/google-cloud-error-reporting
packages/google-cloud-firestore
packages/google-cloud-logging
packages/google-cloud-ndb
packages/google-cloud-pubsub
packages/google-cloud-runtimeconfig
packages/google-cloud-spanner
packages/google-cloud-storage
packages/google-cloud-testutils
packages/google-resumable-media
packages/proto-plus
packages/sqlalchemy-bigquery
packages/sqlalchemy-spanner
4 changes: 4 additions & 0 deletions scripts/version_scanner/matrix.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
python:
- "3.7"
- "3.8"
- "3.9"
6 changes: 0 additions & 6 deletions scripts/version_scanner/small_package_list.txt

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,8 @@ def test_integration_scan(tmp_path):
"-v", "3.7",
"-p", data_dir,
"--config", config_path,
"-o", "scanner_report.csv"
"-o", "scanner_report.csv",
"--soft-fail"
]

result = subprocess.run(cmd, cwd=tmp_path, capture_output=True, text=True, check=True)
Expand Down
Loading
Loading