Dev#100
Open
NicolasKieffer wants to merge 49 commits into
Open
Conversation
feat: add Snapshot Timelines
feat: add user logs in the editorial manager PDF processing
feat: add Integration & E2E tests + CI integration
feat: improve logging system
fix: fix docker issue
fix: manage missing item.value in genshare data
Add an `options.no_cache` boolean to /processPDF. When it resolves to true the request fully bypasses caching: the snapshot-api cache layer is skipped (no canonical read, no seed, no cache-ref) and genshare-service is signalled to skip its own caches via the `config` form field. Access is enforced per-user through the existing option validator: genshare.options.no_cache uses available/default (e.g. admin [true,false] vs. others [false]). filterOptions now also validates boolean options (previously string-only), and no_cache is stripped from the options forwarded to genshare since it is a control flag, not an analysis option. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
feat(cache): add per-request no_cache option to bypass caching
Add a per-version `editorialPolicyOptions` mapping in conf/genshare.json that resolves each editorial_policy to the suggested_das_generation and funding_statement flags, forwarded to genshare-service in the request options. Server-authoritative: any client-supplied values are dropped. Falls back to a `default` entry, then all-false. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
feat(genshare): map editorial_policy to feature flags
Update package-lock.json to patched, in-range dependency versions via `npm audit fix` (no --force). No package.json range changes. Reduces audit findings 22 -> 4. Remaining issues require major upgrades (sqlite3 6, uuid, googleapis) or a library replacement and are handled separately. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolve the remaining moderate advisories (gaxios/googleapis-common/uuid).
Both are majors. Verified: google.auth.GoogleAuth + sheets v4 surface used
by src/utils/googleSheets.js still present; uuid named import { v4 } (used
in scripts/manage_users.js) works. npm audit now reports 0 vulnerabilities.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fix/npm audit safe fixes
- eslint-plugin-node@11's resolver doesn't understand uuid's package "exports" map (introduced in uuid 9, we're now on 14), so node/no-missing-require false-positived 'uuid is not found' in scripts/manage_users.js and failed CI. Allow uuid explicitly on the rule. - Bump actions/checkout and actions/setup-node v4 -> v5 in both workflows to clear the Node 20 runner deprecation warnings. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the deprecated eslint-plugin-node@11 (unmaintained since ~2021) with the maintained fork eslint-plugin-n@17 (peer eslint>=8.23, satisfied by the existing 8.57.1). eslint-plugin-n resolves package "exports" maps, so the uuid 'is not found' false-positive is fixed at the root and the allowModules: ['uuid'] workaround from the previous commit is removed. - .eslintrc.js: node/ -> n/ (extends, plugins, rules); drop the workaround - src/utils/s3Storage.js: inline eslint-disable node/ -> n/ Verified: npm run lint exits 0, uuid resolves without allowModules, 95/95 jest tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fix/npm audit safe fixes
activeGenShareGraphValue was initialized from the raw request (data.options?.editorial_policy || ''), so when a caller omitted editorial_policy it stayed empty even though filterOptions applies the default (e.g. TFOD) that is actually sent to GenShare. The post-response graph check then compared '' against the returned graph and logged a misleading WARN: "[GenShare] Graph values don't match: ( - TFOD)". Refresh activeGenShareGraphValue from the resolved filteredOptions.editorial_policy right after filterOptions runs. No functional change (the value is overwritten with the GenShare-returned graph immediately after the check); this only makes the comparison meaningful and removes the false warning. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
fix(genshare): avoid spurious 'Graph values don't match' warning
Separate two concerns that were both riding on availableFields/ restrictedFields: - availableFields/restrictedFields keep their original purpose: data-access (security) boundaries — what a user is allowed to see. Still enforced everywhere, including the snapshot-reports view. - returnedFields (new) is a presentation-only filter: an ordered strict whitelist that both selects and orders the fields returned in the API response, for client apps that cannot filter/sort the payload themselves. returnedFields is applied to the API response but intentionally skipped for the snapshot-reports view (filterAndSortResponseForUser now takes an applyReturnedFields option, false in snapshotReportsController). This fixes reports showing "N/A" for fields (e.g. reasoning_summary) that a user excluded from their own app's response but is still allowed to see. returnedFields can only narrow an already access-checked set — it can never re-introduce a restricted field. Defaults to empty, so existing users and all other response paths are unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GET /snapshot-reports/:requestId/genshare now accepts an apply_returned_fields boolean query parameter so both views are requestable from one endpoint: - omitted / anything but "true" (default): bypass returnedFields, return the available/restricted-filtered data — the snapshot-reports view, which must show everything the user is allowed to see. - apply_returned_fields=true: also apply the user's presentation-only returnedFields whitelist, reproducing the exact client-facing response. Defaulting to bypass keeps the reports app correct with no change on its side and lets snapshot-api deploy independently without re-introducing the "N/A" regression. The chosen mode is surfaced as meta.returned_fields_applied. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
returnedFields is an ordered whitelist that already both selects and sorts the response, and server-side sorting is only needed for the one client that can't order the array itself. fieldOrder is therefore redundant, so remove it: - genshareManager: drop sortResponseData() and its call; returnedFields is now the single source of server-side ordering (applied to the client response, bypassed for the snapshot-reports view). - refresh_user_logs: order CSV columns by returnedFields instead of fieldOrder, stripping the GenShare suffix so it matches the cleaned field names. Since row values are emitted in returnedFields order, this keeps columns and values aligned (more robust than the previous fieldOrder-based ordering). Users with neither setting are unaffected: responses keep GenShare's native order and logs fall back to alphabetical columns. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rewrite the README "Response Filtering and Sorting" section to reflect the new model: - separate data-access (availableFields/restrictedFields, security, enforced everywhere) from response shaping (returnedFields, presentation, client response only); - document returnedFields as an ordered strict whitelist that both selects and sorts, replacing the removed fieldOrder; - document the apply_returned_fields query parameter on the snapshot-reports endpoint and the meta.returned_fields_applied flag; - update config examples and implementation-detail references. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…haping Feat/returned fields response shaping
Let the snapshot-s3-manager admin UI edit the complete user config instead of only rateLimit/genshare/reports: - getUsers/getUser now return every field except the protected credentials (token, client_secret) via a shared toSafeUser helper, so role, googleSheets and any future keys are visible to the admin UI. - updateUserComplete (PUT /users/:userId) does a full-document replace, stripping token/client_secret/id from the payload and re-injecting the stored secrets so they are preserved and never editable over the API. - Add userManager.replaceUser for whole-object overwrites (updateUser keeps its shallow-merge semantics for the genshare/reports PATCH routes). - Add genshare.options (editorial_policy, no_cache) to the manage_users.js user creation script and refresh its stale 'default' version defaults to 'latest' to match the current schema. Fix stale reads after a write: configWatcher reloaded its in-memory copy only asynchronously (fs.watch + debounce), so a read right after a write returned old data. Add configWatcher.reloadConfig for a synchronous re-read and call it from updateUser/replaceUser after writing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
feat(users): support full user config editing and genshare.options
uuid v14 dropped CommonJS support, breaking manage_users.js with ERR_REQUIRE_ESM. Node 20's built-in crypto.randomUUID() replaces it, so the uuid package is removed entirely. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.