Skip to content

Implement AutoPopulate 2.0 #1290

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
dimitri-yatsenko wants to merge 56 commits into
base: pre/v2.0
Choose a base branch
from
Draft

Implement AutoPopulate 2.0 #1290

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
2be5f11
Introduce AttributeType system to replace AttributeAdapter
claude Dec 21, 2025
055c9c6
Update documentation for new AttributeType system
claude Dec 21, 2025
af9bd8d
Apply ruff-format fixes to AttributeType implementation
claude Dec 21, 2025
9bd37f6
Add DJBlobType and migration utilities for blob columns
claude Dec 21, 2025
c8d8a22
Clarify migration handles all blob type variants
claude Dec 21, 2025
61db015
Fix ruff linter errors: add migrate to __all__, remove unused import
claude Dec 21, 2025
78e0d1d
Remove serializes flag; longblob is now raw bytes
claude Dec 21, 2025
c173356
Remove unused blob imports from fetch.py and table.py
claude Dec 21, 2025
106f859
Update docs: use <djblob> for serialized data, longblob for raw bytes
claude Dec 21, 2025
e293fec
Merge branch 'claude/add-file-column-type-LtXQt' into claude/upgrade-…
dimitri-yatsenko Dec 21, 2025
d66f76e
Merge claude/add-file-column-type-LtXQt into upgrade-adapted-type
claude Dec 22, 2025
2f9b2be
Merge remote-tracking branch 'origin/claude/upgrade-adapted-type-1W3a…
claude Dec 22, 2025
9ad4830
Add Autopopulate 2.0 specification document
claude Dec 22, 2025
df94fcc
Add foreign-key-only primary key constraint to spec
claude Dec 22, 2025
9110515
Remove FK constraints from jobs tables for performance
claude Dec 22, 2025
4637708
Add table drop/alter behavior and schema.jobs list API
claude Dec 22, 2025
68d876d
Clarify ignore status is manual, not automatic transition
claude Dec 22, 2025
f0b7cd8
Simplify job reset mechanism and migration path
claude Dec 22, 2025
6b986ae
Simplify job reservation: no locking, rely on make() transaction
claude Dec 22, 2025
1d4b7b4
Merge remote-tracking branch 'origin/pre/v2.0' into claude/upgrade-ad…
claude Dec 22, 2025
8900fea
Clarify per-key reservation flow in populate()
claude Dec 22, 2025
1f56102
Merge pre/v2.0 into spec-issue-1243
claude Dec 22, 2025
7c22b6d
Update state diagram to Mermaid, consolidate scheduling into refresh()
claude Dec 22, 2025
3018b8f
Add (none)->ignore transition, simplify reserve description
claude Dec 22, 2025
7eda583
Add success->pending transition via refresh()
claude Dec 22, 2025
bab7e10
Use explicit (none) state in Mermaid diagram
claude Dec 22, 2025
586effa
Simplify diagram notation, remove clear_completed()
claude Dec 22, 2025
5b1e3e8
Refine jobs spec: priority, delete, populate logic
claude Dec 22, 2025
2e0a3d9
Clarify stale vs orphaned job terminology
claude Dec 22, 2025
77c7cf5
Remove FK-only PK requirement, add hazard analysis
claude Dec 23, 2025
86e21f4
Clarify conflict resolution and add pre-partitioning pattern
claude Dec 23, 2025
314ad0a
Fix incorrect statement about deleting reserved jobs
claude Dec 23, 2025
61cc759
Use relative delay (seconds) instead of absolute scheduled_time
claude Dec 23, 2025
7b11d65
Clarify that only make() errors are logged as error status
claude Dec 23, 2025
086de07
Implement Autopopulate 2.0 job system
claude Dec 23, 2025
53bd28d
Drop jobs table when auto-populated table is dropped
claude Dec 23, 2025
428c572
Add tests for Autopopulate 2.0 jobs system
claude Dec 23, 2025
e89e064
Fix ruff linting errors and reformat
claude Dec 23, 2025
0f98b18
Remove legacy schema-wide jobs system
claude Dec 23, 2025
956fa27
Rename jobs_v2.py to jobs.py
claude Dec 23, 2025
608020a
Improve jobs.py: use update1, djblob, cleaner f-string
claude Dec 23, 2025
8430e2a
Simplify reserve() to use update1
claude Dec 23, 2025
34c302a
Use update1 in complete() method
claude Dec 23, 2025
e0d6fd9
Simplify: use self.proj() for jobs table projections
claude Dec 23, 2025
83b7f49
Simplify ignore(): only insert new records, cannot convert existing
claude Dec 23, 2025
080b6c0
Use insert1 in _insert_job_with_status instead of explicit SQL
claude Dec 23, 2025
84ba4b7
Remove AutoPopulate._job_key - no longer needed
claude Dec 23, 2025
6ef2de7
Remove AutoPopulate.target property
claude Dec 23, 2025
55d7f32
Remove legacy _make_tuples callback support - use self.make exclusively
claude Dec 23, 2025
7b28c64
Eliminate _jobs_to_do method
claude Dec 23, 2025
d28fa7c
Simplify jobs variable usage in populate()
claude Dec 23, 2025
7d595fb
Inline _get_pending_jobs into populate()
claude Dec 23, 2025
0a5f3a9
Remove order parameter and consolidate limit/max_calls
claude Dec 23, 2025
8091225
Merge remote-tracking branch 'origin/claude/add-file-column-type-LtXQ…
claude Dec 23, 2025
bab4db8
Merge claude/upgrade-adapted-type-1W3ap
claude Dec 23, 2025
4a43f50
Merge pre/v2.0
claude Dec 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/src/compute/key-source.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ definition = """
-> Recording
---
sample_rate : float
eeg_data : longblob
eeg_data : <djblob>
"""
key_source = Recording & 'recording_type = "EEG"'
```
4 changes: 2 additions & 2 deletions docs/src/compute/make.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,7 @@ class ImageAnalysis(dj.Computed):
# Complex image analysis results
-> Image
---
analysis_result : longblob
analysis_result : <djblob>
processing_time : float
"""

Expand Down Expand Up @@ -188,7 +188,7 @@ class ImageAnalysis(dj.Computed):
# Complex image analysis results
-> Image
---
analysis_result : longblob
analysis_result : <djblob>
processing_time : float
"""

Expand Down
6 changes: 3 additions & 3 deletions docs/src/compute/populate.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ class FilteredImage(dj.Computed):
# Filtered image
-> Image
---
filtered_image : longblob
filtered_image : <djblob>
"""

def make(self, key):
Expand Down Expand Up @@ -196,7 +196,7 @@ class ImageAnalysis(dj.Computed):
# Complex image analysis results
-> Image
---
analysis_result : longblob
analysis_result : <djblob>
processing_time : float
"""

Expand Down Expand Up @@ -230,7 +230,7 @@ class ImageAnalysis(dj.Computed):
# Complex image analysis results
-> Image
---
analysis_result : longblob
analysis_result : <djblob>
processing_time : float
"""

Expand Down
Loading
Loading