feat: extensible transforms, process registry, temporal resampling, and reprojection (CLIM-679)

[turban]

Summary

This PR consolidates three areas of new functionality — all following the same plugin pattern:

Extensible transforms pipeline

• Dataset YAMLs declare a transforms: list of dotted-path functions applied during zarr build (after download, before writing)
• Built-in transforms live in climate_api/transforms/; custom transforms can be loaded from any importable package under plugins_dir
• Built-in transforms: convert_units, deaccumulate_era5, reproject_to_instance_crs

Reprojection transform

• reproject_to_instance_crs (rioxarray-backed) reprojects source data to the instance CRS configured in climate-api.yaml
• No-op when source CRS already matches the instance CRS — WGS84 instances incur zero overhead
• Wired into chirps3, era5_land (temperature and precipitation), and worldpop dataset templates
• Adds rioxarray>=0.17 as an explicit dependency

Process registry with plugin support

• Pluggable process registry backed by YAML files — same pattern as the dataset registry
• Built-in processes in climate_api/data/processes/; custom processes via plugins_dir/processes/
• Each process YAML declares an execution_function dotted path; POST /processes/{id}/execution dispatches generically — no hardcoded process-id checks

Temporal resampling (first built-in process)

• Aggregate a source dataset to a coarser temporal resolution using pandas frequency aliases (1D, W-MON, MS, etc.)
• Supported methods: mean, sum, min, max
• Produces a derived dataset stored as a new zarr alongside the source

Plugin pattern (consistent across datasets, transforms, processes)

Extension point	Built-in location	Plugin override location	Reference mechanism
Datasets	climate_api/data/datasets/	plugins_dir/datasets/	YAML with ingestion.function dotted path
Transforms	climate_api/transforms/	any importable package under plugins_dir	dotted path in dataset YAML transforms: list
Processes	climate_api/data/processes/	plugins_dir/processes/	YAML with execution_function dotted path

Key files changed

File	Change
climate_api/transforms/__init__.py	Transform registry + built-in exports
climate_api/transforms/reproject.py	reproject_to_instance_crs — rioxarray reprojection
climate_api/transforms/unit_conversion.py	convert_units — unit conversion via metpy
climate_api/transforms/deaccumulate.py	deaccumulate_era5 — ERA5 accumulation fix
climate_api/data_registry/services/processes.py	Process registry (list, get, plugin load)
climate_api/processing/resample.py	Core temporal resampling logic
climate_api/processing/routes.py	Generic process dispatch via registry
climate_api/processing/services.py	execute_resample entry point
climate_api/data/processes/resample.yaml	Built-in resample process definition
climate_api/data/datasets/*.yaml	Transform lists added to all built-in datasets
pyproject.toml	Added rioxarray>=0.17

Test plan

• make lint — clean (ruff, mypy, pyright)
• pytest — all tests pass (transforms, reproject, process registry, resampling, routes)
• POST /processes/resample/execution with valid request returns 200
• POST /processes/resample/execution with invalid method/frequency returns 400
• POST /processes/unknown/execution returns 404
• Process registry loads built-in resample from data/processes/resample.yaml
• Custom process YAML in plugins_dir/processes/ merges with built-ins
• reproject_to_instance_crs no-ops when source CRS matches instance CRS
• reproject_to_instance_crs calls rio.reproject with the correct target CRS

---

Supersedes #63. Incorporates #86 (transforms pipeline) and #93 (reprojection transform).

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds a pluggable process registry (YAML-backed, plugin-overridable) and a generic processing execution route, introducing temporal resampling as the first built-in process and extending the transforms pipeline with built-in transforms (unit conversion, deaccumulation, reprojection). Also expands time-period handling to support weekly ISO week strings across sync and coverage.

Changes:

• Introduces process registry (+ plugin override support) and POST /processes/{process_id}/execution generic dispatcher.
• Implements resampling materialization workflow (derived Zarr artifacts) and adds weekly period parsing/normalization.
• Adds transforms pipeline + built-in transforms, updates dataset YAMLs to use dotted-path transforms, and adds comprehensive tests.

Reviewed changes

Copilot reviewed 30 out of 32 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
tests/test_transforms_reproject.py	Adds tests for the reprojection transform behavior and rioxarray integration.
tests/test_transforms.py	Adds tests for unit conversion, deaccumulation, and the dotted-path transforms pipeline.
tests/test_shared_time.py	Extends coverage for weekly period normalization/parsing.
tests/test_processing_routes.py	Adds route-level tests for /processes/resample/execution behavior and error cases.
tests/test_processing_resample.py	Adds extensive tests for resample materialization, edge-period dropping, reuse/overwrite behavior, and publishing.
tests/test_process_registry.py	Tests built-in + plugin process registry loading and override behavior.
tests/test_datasets_sync.py	Updates sync tests for weekly support and refines unsupported period-type expectations.
tests/test_dataset_registry.py	Updates registry tests to reflect new ingestion function strings and removes some ingestion-validation tests.
pyproject.toml	Adds rioxarray dependency for .rio accessor support.
climate_api/transforms/unit_conversion.py	Implements built-in unit conversion transform (scale + offset).
climate_api/transforms/reproject.py	Implements built-in reprojection transform to instance CRS.
climate_api/transforms/deaccumulate.py	Implements built-in ERA5 deaccumulation transform.
climate_api/transforms/init.py	Exposes built-in transforms for dotted-path references.
climate_api/shared/time.py	Adds weekly period support and weekly handling in numpy datetime conversions.
climate_api/publications/services.py	Refactors managed dataset id generation into a reusable function.
climate_api/processing/services.py	Adds execution function for resample with validation and response formatting.
climate_api/processing/schemas.py	Adds Pydantic request/response schemas and supported methods constant.
climate_api/processing/routes.py	Adds generic process execution endpoint dispatched via process registry.
climate_api/processing/resample.py	Implements derived resampling materialization, artifact reuse, completeness checks, and Zarr writing.
climate_api/processing/init.py	Adds processing package module.
climate_api/main.py	Registers processing routes in the FastAPI app.
climate_api/ingestions/sync_engine.py	Adds weekly period arithmetic for sync planning.
climate_api/ingestions/services.py	Adds helper to store locally materialized Zarr artifacts and supports weekly default end.
climate_api/data_registry/services/processes.py	Adds YAML-backed process registry with plugin merging and dotted-path execution loading.
climate_api/data_manager/services/downloader.py	Adds transforms execution hook (_run_transforms) into dataset build flow.
climate_api/data_accessor/services/accessor.py	Normalizes period-string scalars when computing coverage.
climate_api/data/processes/resample.yaml	Registers the built-in resample process and its execution function.
climate_api/data/datasets/worldpop.yaml	Adds reprojection transform to dataset definition.
climate_api/data/datasets/era5_land.yaml	Switches preprocessing to transforms pipeline and adjusts display range after unit conversion.
climate_api/data/datasets/chirps3.yaml	Adds reprojection transform to dataset definition.
.gitignore	Ignores derived data directory (data/derived).