Add Person 3 ML training and evaluation pipeline

.github/workflows/deploy.yaml

@@ -12,6 +12,9 @@ concurrency:
   group: "${{ github.ref }}-${{ github.head_ref }}-${{ github.workflow }}"
   cancel-in-progress: false
 
+permissions:
+  contents: read
+
 jobs:
   build:
     name: Build package

.github/workflows/docs.yml

@@ -14,6 +14,8 @@ concurrency:
 jobs:
   docs:
     runs-on: ubuntu-latest
+    permissions:
+      contents: write
 
     steps:
     - uses: actions/checkout@v6

.github/workflows/lint.yml

@@ -6,6 +6,9 @@ on:
   pull_request:
     branches: [ main, develop ]
 
+permissions:
+  contents: read
+
 jobs:
   lint:
     runs-on: ubuntu-latest

.github/workflows/tests.yml

@@ -8,6 +8,9 @@ on:
   schedule:
     - cron: '0 0 * * *'
 
+permissions:
+  contents: read
+
 jobs:
   build:
 

Plans/person3-ml-workplan-2026-07-13.md

@@ -0,0 +1,173 @@
+# Person 3 Work Plan (ML Training & Evaluation)
+
+**Owner:** Chenou  
+**Date:** 2026-07-13  
+**Branch:** person3-ml-training-eval  
+**Target milestone:** Phase 3 complete by 2026-07-27
+
+---
+
+## Scope
+
+Build the end-to-end ML pipeline for ConfoState:
+
+1. Dataset assembly and split logic
+2. Baseline model training
+3. Evaluation + report generation
+4. Model artifact management / registry
+
+Primary deliverables:
+- confostate/data/datasets.py
+- confostate/models/baseline.py
+- confostate/models/train.py
+- confostate/models/evaluate.py
+- confostate/models/registry.py
+- tests for datasets/model training/evaluation
+- trained artifacts in data/models/
+- evaluation report in docs/reports/
+
+---
+
+## Dependencies and Risk Controls
+
+### Needed from Person 1 and 2
+- Stable annotations schema in data/annotations/leu_t_transporters.csv
+- Feature vectors for training rows (or at least a subset)
+
+### Risk if features are delayed
+- Use a synthetic feature table to unblock model/training/evaluation development.
+- Keep synthetic schema aligned with expected real feature names.
+- Add a quick integration check that can switch from synthetic to real features by changing only file paths.
+
+### Definition of done for integration
+- Train command runs successfully from raw tabular features + labels.
+- Evaluation command produces metrics and confusion matrix artifact.
+- Registry points to a reproducible model artifact and metadata.
+
+---
+
+## Week-by-Week Plan
+
+## Week 1 (2026-07-13 to 2026-07-19): Pipeline Skeleton + Baselines
+
+### Day 1: Data contract and dataset loader
+- Finalize expected columns:
+  - keys: pdb_id, family, conformation
+  - features: numeric columns only
+- Implement confostate/data/datasets.py:
+  - load_dataset(annotations_csv, features_csv)
+  - validate label and key integrity
+  - make_split(strategy="stratified", test_size=0.2, random_state=...)
+
+### Day 2: Baseline model module
+- Implement confostate/models/baseline.py:
+  - logistic regression baseline
+  - random forest baseline
+  - SVM baseline
+- Add uniform interface:
+  - get_model(name, random_state, class_weight)
+
+### Day 3: Training pipeline
+- Implement confostate/models/train.py:
+  - fit single model
+  - cross-validation score summary
+  - save artifact + metadata
+- Save outputs under data/models/{family}/{model_name}/
+
+### Day 4: Evaluation module
+- Implement confostate/models/evaluate.py:
+  - confusion matrix
+  - per-class precision/recall/F1
+  - macro and weighted scores
+  - optional ROC-AUC (when applicable)
+- Emit markdown report to docs/reports/
+
+### Day 5: Smoke tests + first run
+- Add tests for:
+  - dataset split stability
+  - model train/predict shape and class consistency
+  - evaluation output keys and file generation
+- Run first end-to-end experiment with available data or synthetic fallback
+
+## Week 2 (2026-07-20 to 2026-07-27): Hardening + Reporting + Handoff
+
+### Day 6-7: Hyperparameter tuning and comparison
+- Add small grid/random search for each baseline model
+- Compare by macro-F1 and balanced accuracy
+- Select default baseline for each family
+
+### Day 8: Registry and reproducibility
+- Implement confostate/models/registry.py:
+  - register_model(family, model_name, artifact_path, metrics, data_version)
+  - load_registered_model(family)
+- Include metadata fields:
+  - timestamp, git commit hash, feature schema hash, random_state
+
+### Day 9: Final report package
+- Produce docs/reports/phase3-baseline-report.md
+- Include:
+  - train/test split method
+  - model comparison table
+  - per-state metrics
+  - known limitations and next steps
+
+### Day 10: Handoff to Person 4 and Person 5
+- Share:
+  - top model artifacts
+  - feature importance-compatible model outputs
+  - stable prediction API contract for explainability and CLI
+
+---
+
+## Suggested File Interfaces
+
+### confostate/data/datasets.py
+- load_dataset(annotations_csv: str, features_csv: str, family: str | None = None)
+- train_test_split_dataset(df, label_col="conformation", test_size=0.2, random_state=42)
+
+### confostate/models/baseline.py
+- get_baseline_models(random_state=42) -> dict[str, estimator]
+- train_model(estimator, X_train, y_train)
+
+### confostate/models/train.py
+- run_training(config: dict) -> dict
+- save_model_artifact(model, out_dir, metadata)
+
+### confostate/models/evaluate.py
+- evaluate_model(model, X_test, y_test, labels=None) -> dict
+- write_evaluation_report(metrics: dict, output_path: str)
+
+### confostate/models/registry.py
+- register_model(...)
+- get_registered_model(family: str)
+
+---
+
+## Daily Execution Checklist
+
+- Confirm branch and pull latest main changes (if needed)
+- Implement one scoped unit of functionality
+- Add or update tests
+- Run local test subset
+- Commit with concise message
+- Post async update in team channel with blockers/dependencies
+
+---
+
+## Communication Cadence
+
+- Tuesday standup: report progress, current blocker, next 48h plan
+- Async updates: at least every 2 working days in #confostate-dev
+- Dependency syncs:
+  - with Person 2 on feature table schema
+  - with Person 4 on importance/explanation-ready outputs
+
+---
+
+## Exit Criteria (Phase 3 Complete)
+
+- End-to-end training command works on at least one family
+- >=2 baseline models compared and documented
+- Evaluation report generated with per-state metrics
+- Model registry points to reproducible artifacts
+- Person 4 receives model + outputs needed for explainability

Plans/person3-progress-memo-2026-07-13.md

@@ -0,0 +1,51 @@
+# Person 3 Progress Memo
+
+**Date:** 2026-07-13  
+**Owner:** Chenou  
+**Branch:** person3-ml-training-eval
+
+## Summary
+
+Implemented an initial, runnable scaffold for Person 3 (ML Training & Evaluation) so the team can start end-to-end model development before all upstream dependencies are finalized.
+
+## Completed Work
+
+1. Added detailed Person 3 execution plan and linked it from the 6-person team plan.
+2. Added dataset utilities for annotation/features merge, split generation, and X/y extraction.
+3. Added baseline model, training, evaluation, and registry modules under `confostate/models`.
+4. Added outline scripts for each Person 3 workstream task in `scripts/`.
+5. Updated package exports and dependencies to include model subpackages and `scikit-learn`.
+6. Documented script usage in `docs/USAGE.md`.
+
+## New/Updated Paths
+
+- Plans/person3-ml-workplan-2026-07-13.md
+- Plans/workplan-6person.md
+- confostate/data/datasets.py
+- confostate/models/__init__.py
+- confostate/models/baseline.py
+- confostate/models/train.py
+- confostate/models/evaluate.py
+- confostate/models/registry.py
+- confostate/__init__.py
+- confostate/data/__init__.py
+- scripts/p3_dataset_loader.py
+- scripts/p3_baseline_models.py
+- scripts/p3_training_pipeline.py
+- scripts/p3_evaluate_reporting.py
+- scripts/p3_model_registry.py
+- docs/USAGE.md
+- pyproject.toml
+
+## Notes
+
+- The scripts are intentionally outline-level and designed for iterative refinement.
+- Current workflow supports a synthetic feature table fallback if upstream feature extraction is delayed.
+- Preferred test environment for ConfoState runs:
+  `/nfs/homes5/Projects/SLC26/chenou/openff/UGM2025/workshops/OpenFF/micromamba_root/envs/ConfoState`
+
+## Immediate Next Steps
+
+1. Add a minimal synthetic features CSV fixture for smoke testing.
+2. Run one full train/eval/registry cycle and capture report artifacts.
+3. Add pytest smoke tests for dataset merge and training script execution.

Plans/workplan-6person.md

@@ -16,6 +16,7 @@ ConfoState development split into 6 parallel work streams, each led by one team
 
 - 2026-06-15 initial draft (AI generated)
 - 2026-06-29 annotated in group meeting
+- 2026-07-13 Person 3 execution plan added in Plans/person3-ml-workplan-2026-07-13.md
 
 ## General development notes
 
@@ -176,6 +177,8 @@ ConfoState development split into 6 parallel work streams, each led by one team
 
 **Assignee**: Chenou
 
+Detailed execution plan: Plans/person3-ml-workplan-2026-07-13.md
+
 ### Tasks
 
 1. **Build dataset loader**

confostate/__init__.py

@@ -4,5 +4,22 @@
 
 from confostate.data.loader import load_annotations
 from confostate.features import extract_features
+from confostate.models import (
+    evaluate_model,
+    get_baseline_models,
+    get_registered_model,
+    register_model,
+    run_training,
+    write_evaluation_report,
+)
 
-__all__ = ["load_annotations", "extract_features"]
+__all__ = [
+    "extract_features",
+    "evaluate_model",
+    "get_baseline_models",
+    "get_registered_model",
+    "load_annotations",
+    "register_model",
+    "run_training",
+    "write_evaluation_report",
+]

confostate/data/__init__.py

@@ -1 +1,16 @@
 """Data loading utilities for ConfoState."""
+
+from confostate.data.datasets import (
+    build_xy,
+    load_dataset,
+    train_test_split_dataset,
+)
+from confostate.data.loader import load_annotations, load_from_input_dir
+
+__all__ = [
+    "build_xy",
+    "load_annotations",
+    "load_dataset",
+    "load_from_input_dir",
+    "train_test_split_dataset",
+]