Ls/async engine singleton#254
Closed
cmttt wants to merge 8 commits into
Closed
Conversation
Adds an asyncio-native osprey worker as a sibling package to the gevent worker. Includes coordinator input stream, sinks (rules, output), pigeon client + interceptors, etcd-backed sources provider, executor, and engine. Coordinator gains a bidirectional stream and pigeon module. Subsequent tuning to match the gevent baseline: - Move recompile off the event loop; force gc.collect after engine swap - Match gevent's compile CPU profile (and later remove the gc.collect hack) - Empirically widen pause-input-stream jitter (final: 0-600s) - Preserve WatchMux dedup state across watcher iteration
Engine: AST/grammar tweaks, executor and execution-context refinements, UDF helpers, stdlib UDF updates (categories, experiments, json_data, rules, string), external service utils. Worker: udf_register, osprey_engine, sources_provider (incl. skipping engine recompile on no-op etcd sources events), output/rules sinks. Plus repo housekeeping: CODEOWNERS, PR template, dev workflow doc, and a small UI polish in FeatureSelectModal.
- _mux.py / test_watcher_mux.py: preserve dedup state correctly across watcher iteration. - watcherd_impl.py: downgrade etcd watcher reconnect logs from exception to warning (these reconnects are routine, not errors). - action.proto + generated pb2/pb2.pyi: add GuildBatchMLScore action.
add and register count regex matches udf
* Bootstrap async UDFs through first-party plugin hook Move the async-stdlib MXLookup override from a hardcoded list inside bootstrap_async_udfs into a small first-party pluggy plugin module (stdlib_udfs/_async_stdlib_plugin.py). The plugin registers async-native stdlib replacements through the same register_udfs hook used by discord_osprey_async_plugins. _deduplicate_udfs already handles override-by-class-name, so MXLookup (async) automatically shadows MXLookup (sync) — no special path. Adding a new async stdlib override now means: drop the class file in stdlib_udfs/ and append it to _async_stdlib_plugin.register_udfs(). No edits to bootstrap_async_udfs required. * Tests for async UDF bootstrap behavior Lock down that bootstrap_async_udfs: - resolves MXLookup to the async-native class (not sync stdlib) - doesn't leak SyncMXLookup into the registry - preserves non-overridden stdlib UDFs (JsonData, Rule) - registers the first-party async-stdlib plugin via the same hook used by third-party plugins - emits AsyncMXLookup through the register_udfs hook
Mirrors the gevent osprey.worker.lib.singletons.ENGINE pattern so migrating services (smite_ui_api, smite_rules_api, webhook_sink, bulk_label_sink) can get a lazily-initialized AsyncOspreyEngine via ENGINE.instance(). Also adds get_feature_name_to_entity_type_mapping and get_post_execution_feature_name_to_value_type_mapping on AsyncOspreyEngine, mirroring the gevent OspreyEngine methods of the same names.
cmttt
requested review from
a team,
EXBreder,
ayubun,
haileyok and
vinaysrao1
as code owners
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Checklist
uv run ruff check .passes (no unused imports or other lint errors)uv tool run fawltydeps --check-unused --pyenv .venvpasses (no unused dependencies)CHANGELOG.mdwith my changes, if applicable