Database Packaging by isc-dchui · Pull Request #1121 · intersystems/ipm

isc-dchui · 2026-04-13T19:41:06Z

Description

Resolves #986

Overview

The feature adds a new lifecycle class, %IPM.Lifecycle.Database, that sits alongside the existing %IPM.Lifecycle.Module. It inherits from %IPM.Lifecycle.Base directly — not from Module — because Module has PACKAGING as a [Final] parameter and can't be subclassed to change it.
Two new commands are exposed in %IPM.Main:
- package-database → forces %IPM.Lifecycle.Database, routes to the Package phase
- publish-database → same, but routes to Publish (which calls Package first, then uploads to registry)
The -swap-db modifier is added to load/install/update to authorize the DB swap during install.

Key Details

Package = IRIS.DAT + metadata, not source. The .tgz contains the binary IRIS.DAT, an enriched module.xml (with database and a SHA-256 ), a deps/ directory of dependency manifests, and any non-compiled resource files (wheels, FileCopy, CPF). Nothing is recompiled at install time.
Remap-before-dismount during packaging. When building the package, the namespace is remapped to a fresh empty temp DB before the old DB is dismounted. This avoids any window where the namespace has no mounted routines DB.
Generated resources need special handling. Classes with Generated="true" are exported from the source DB to XML before the remap, then imported into the temp DB after. Once the old DB is dismounted, compiled generated classes are inaccessible.
Install skips Reload and Compile entirely. IsInstallContext() checks two conditions: Packaging=database AND IRIS.DAT is present in the module root. If both are true, %Reload sets SkipInvokes=1 and returns, %Compile is a no-op, and %Activate performs the swap. Pre-Activate elements are suppressed because the packaged classes don't exist until the DB is mounted.
Swap is a rename, not a copy. PerformDatabaseSwap dismounts the existing DB, renames the current IRIS.DAT to IRIS__.DAT as a backup, renames the packaged IRIS.DAT into place, and remounts. Renames are atomic and avoid copying multi-GB files.
Rollback is best-effort. On any failure after the swap starts, RollbackDatabaseSwap attempts dismount → delete new IRIS.DAT → rename backup back → remount. It never throws — it logs warnings and continues since it's called from a catch block.
Update step seeding on fresh install. %Activate calls HandleAllUpdateSteps(..., seedOnly=1) on a fresh install so a later update only runs steps introduced since that version. Skipped when params("Update")=1 (an actual upgrade).
One database package per namespace. ValidateBeforeSwap checks %IPM_Storage.ModuleItem for any other module with Packaging='database' and rejects the install if one exists.
Python deps are on by default for package-database. Unlike package (which defaults off), package-database defaults ExportPythonDependencies=1. Both requirements.txt-based deps (resolved to wheels via ExportPythonDependencies) and explicit resources are staged into the .tgz. PythonWheel install at runtime fires during Initialize phase, which has no Database.cls override.
Non-compiled resources are handled by resource processor hooks, not %Activate. FileCopy, WebApplication, and CPF processors run via OnBeforePhase/OnAfterPhase at the Module level before the lifecycle method is called. Overriding %Activate in Database.cls doesn't suppress them.
publish-database forces the lifecycle class. The handler sets tCommandInfo("data","Lifecycle") = "%IPM.Lifecycle.Database", then re-routes to the existing publish command handler. Base.%Publish calls ..Package(.pParams, 0) first (which populates ..Payload), then uploads to the registry. No Database.cls override of %Publish is needed.
dependencies.xml is actually a deps/ directory of per-dependency IRIS export XML files, each loadable with $system.OBJ.Load. RegisterDependencyMetadata loads them at install time to populate IPM storage — the code is already in the swapped-in DB, so only the IPM metadata records need registering. This is much easier to handle instead of using a fragile XSLT for a giant dependencies.xml
ValidateIPMNotInRoutinesDB is called at two points: immediately in ShellInternal (fail-fast before any work) and again inside %Package (defense-in-depth). It checks %Library.RoutineMgr.IsMapped("%IPM.Main.CLS").
The XSLT transform (InjectDatabasePackagingTransform XData) injects and into module.xml. The checksum placeholder REPLACECHECKSUM is substituted in ObjectScript before the stylesheet is compiled, since XSLT 1.0 has no parameterized element content.
When installing without swap-db, only installs source-packaged modules. If none exist, but database-packaged module(s) exist, informs user. When installing with swap-db, only installs database-packaged modules.
When publishing database-packaged modules to OCI registry, sets com.intersystems.ipm.packaging manifest annotation to "database" (for backwards compatibility, source packaged modules will not have this annotation) and use <module-version>_database__<IRIS-version> tag

Testing

All tests are in Test.PM.Integration.DatabasePackaging. The test infrastructure uses two shared lazy-initialized namespaces to avoid ~15 namespace create/teardown cycles per run:
- SharedNS1 (TESTDBPKGNS1): simple-db-module, module-with-invokes, module-with-dependencies
- SharedNS2 (TESTDBPKGNS2): all-resources-module, module-with-mixed-python, module-with-requirements; also has the zot ORAS registry configured
Per-test install namespaces are created fresh and torn down in OnAfterOneTest.

Test	What it covers
TestSimplePackageAndInstall	Package → verify IRIS.DAT + module.xml content (Packaging, checksum, SystemRequirements version) → install with `load -swap-db` → classes callable, module in list and history
TestPackageAndInstallWithDependencies	Package dep-module + main-with-deps; install main-with-deps → main class exists, dep appears in `list`
TestPackageAndInstallAllResourceTypes	Smoke test: module with class, include file, generated class, WebApp, FileCopy, PythonWheel packages and installs; include constant accessible, generated class callable, WebApp created
TestPackageAndInstallWithTestResources	`-include-test-resources` includes Scope=test (TestHelper) and UnitTest (Test) classes; default packaging excludes both
TestPackageWithUseCurrentDbFlag	`-use-current-db` completes without error and produces a package file
TestPackagingAndInstallValidation	Valid install succeeds; tampered IRIS.DAT (checksum mismatch), zero-sized IRIS.DAT, and missing `deps/` directory each fail validation
TestBackupAndRollback	Backup file created in routines DB directory after swap
TestUpgradeSourceToDatabase	Source-installed module upgraded via `update -path -swap-db`; Packaging changes to "database"; backup confirms DB swap occurred
TestUninstallDatabasePackage	Module removed from `list`; backup preserved; compiled classes gone from namespace
TestMixedPackaging	Database + source package coexist in same namespace; second database package in same namespace is rejected
TestPythonDependencyPackaging	module-with-mixed-python: explicit PythonWheel (lune) + requirements.txt wheel (pycparser) both included by default, excluded with `-no-export-python-deps`; lune importable after install. module-with-requirements: packaging wheel included by default, excluded with flag
TestInvokeBehaviorDuringDatabaseInstall	`<Invoke After="Compile">` skipped during database install; `<Invoke After="Activate">` runs; global markers confirm each
TestNonCompiledResourcesAppliedDuringInstall	CPF file staged in package; FileCopy target deleted before install, confirmed recreated after install — proves resource processor hooks fire independently of `%Activate` override
TestUpdateStepsWithDatabasePackaging	v1 fresh install seeds Step001 (not run); v2 upgrade runs Step002 once; Step001 still not run; backup count > 1 confirms second swap
TestPublishDatabaseWithORAS	publish-database + publish to zot; source install (no `-swap-db`) → Packaging=module; database install (`-swap-db`) → Packaging=database; main-with-deps install resolves dep as source; after unpublishing source tag, install without `-swap-db` fails with hint to use `-swap-db`
TestSystemRequirementsOverwrite	`package-database` overwrites a stale IRIS version in `<SystemRequirements>` with the current version

Not Yet Handled

Mirroring

Checklist

This branch has the latest changes from the main branch rebased or merged.
Changelog entry added.
Unit (zpm test -only) and integration tests (zpm verify -only) pass.
Style matches the style guide in the contributing guide.
Documentation has been/will be updated
- Source controlled docs, e.g. README.md, should be included in this PR and Wiki changes should be made after this PR is merged (add an extra issue for this if needed)
Pull request correctly renders in the "Preview" tab.

isc-jili

Looks great @isc-dchui ! I left some comments!

isc-jili · 2026-04-23T18:20:33Z

+}
+
+/// Remaps the namespace's routines to newDBName, then dismounts the old routines DB.
+/// Remap-before-dismount avoids a window where the namespace has no mounted routines DB.


Small nit: given that there is a nearly identical comment less than 10 lines down, suggest either removing that comment or removing this section from method description

isc-jili · 2026-04-23T20:21:01Z

+    // Resolve all module dirs before switching namespace (^UnitTestRoot only in original NS)
+    set depDir = ..GetModuleDir("db-packaging", "dependency-module")
+    set simpleDir = ..GetModuleDir("db-packaging", "simple-module")
+    set depsDir = ..GetModuleDir("db-packaging", "module-with-deps")


nit: suggest renaming depDir and depsDir to be more distinctive as currently their names are a bit confusing

isc-jili · 2026-04-23T20:28:18Z

+    // Extract and verify contents
+    set extractDir = ##class(%File).NormalizeDirectory(..PackageOutputDir _ "extract")
+    do ##class(%File).CreateDirectory(extractDir)
+    set sc = ##class(%IPM.General.Archive).Extract(packageFile, extractDir, .extractOutput)


Question: is there any info that needs to be checked in extractOutput? I don't see it being checked below. If it doesn't need to get checked, is there any value in storing it in a variable then?

isc-jili · 2026-04-27T19:59:21Z

+    set sc = ##class(%IPM.Main).Shell("package-database simple-db-module -path " _ ..PackageOutputDir)
+
+    // Restore original module.xml unconditionally before asserting, so a failure doesn't leave
+    // the shared NS in a broken state for subsequent tests


Question: if we want to restore module.xml unconditionally, would it be safer to have the following in a try/catch block in case the package-database fails?

Good call. Added a try/catch failsafe

isc-jili · 2026-04-27T20:05:47Z

+    set $namespace = installNS2
+
+    set sc = ##class(%IPM.Main).Shell("load " _ tamperedPackage _ " -swap-db")
+    do $$$AssertStatusNotOK(sc, "Tampered IRIS.DAT rejected by checksum validation")


Would it be more accurate to check for a substring of the expected error that is thrown as opposed to just the status not being OK? Since what if the status is not OK for a different reason than expected?

isc-jili · 2026-04-27T20:29:33Z

+        write:verbose !, "Computing SHA-256 checksum..."
+        set irisDAT = tempDBPath _ "IRIS.DAT"
+        set checksumHex = ..ComputeSHA256Hex(irisDAT)
+        write !, "  Checksum: ", checksumHex


Question: I'm noticing for print statements some are verbose and some are always printed. Just want to double check that this is the intent.
It seems that currently this is the state of print statements:

invoke warnings: always printed

checksum: always printed

package path: always printed

many progress steps: verbose-only

Right. I think the checksum and package path being always printed is helpful for the user in general

isc-jili · 2026-04-27T20:30:35Z

@@ -0,0 +1,1050 @@
+Include %syPrompt


Does anything in this class use %syPrompt?

The $$$SuccessResponse macro used to check the result of ##class(%Library.Prompt).GetYesNo() is defined in %syPrompt

isc-jili · 2026-04-27T20:31:02Z

+Method ExportGeneratedResources(
+	ns As %String,
+	includeTestResources As %Boolean,
+	ByRef params,


Nit: ExportGeneratedResources() takes ByRef params, but I don’t think the method actually uses it?

Good catch, removed it from the parameters

isc-jili · 2026-04-27T20:31:41Z

+/// Throws on any failure. Does not modify any state.
+Method ValidateBeforeSwap(
+	packageDir As %String,
+	ns As %String) [ Private ]


nit: param ns seems to be unused

Removed from parameters

isc-jili · 2026-04-27T20:36:34Z

+/// Dismounts the temp DB, remaps the namespace back to oldDBName, remounts it,
+/// and removes the temp database config entry.
+/// Logs warnings on partial failures but always attempts full restoration.
+ClassMethod RestoreRoutinesDB(


The new methods in this new class could all benefit from params documented in the method comments as well!

Added a bunch more documentation!

…e in one call

… and packaging property

isc-eneil · 2026-05-06T20:36:11Z


  build:
-    timeout-minutes: 45
+    timeout-minutes: 60


Just curious-- why the timeout-minutes changes?

isc-eneil · 2026-05-06T20:38:31Z

+<parameter name="module" required="true" description="Name of module to package as database" />
+<modifier name="path" aliases="p" dataAlias="Path" value="true" description="Output directory for the .tgz file." />
+<modifier name="use-current-db" dataAlias="UseCurrentDB" dataValue="1" description="Package the namespace's current routines DB instead of creating a fresh one." />
+<modifier name="include-test-resources" dataAlias="IncludeTestResources" dataValue="1" description="Include test resources (Scope=test or Scope=verify) in the package. Excluded by default." />


I still vote calling this -dev instead of -include-test-resources to keep the modifier names consistent across commands and minimize confusion, but I won't die on that hill

isc-eneil · 2026-05-10T21:31:03Z


 <command name="package" dataPrefix="D">
 <description>
 This command is an alias for `module-action module-name package`


Nit: update the description to indicate that source packaging is the default for the "package" command

isc-eneil · 2026-05-10T21:40:44Z

+                // Identical to "package": route to the Package phase using current lifecycle
+                set tCommandInfo = "package"
+                do ..RunOnePhase(.tCommandInfo)
+            } elseif (tCommandInfo = "package-database") {


@isc-dchui I remember that adding package-** commands was part of your original design, but I can't remember why we went with that instead of flags on the existing package command so that it continues to be considered a module-action rather than separate commands. I'm not sure I have a strong preference either way, design-wise, but I'm noticing that the flow through the Shell is different.

isc-eneil · 2026-07-20T20:52:21Z

        }
        set tModule.VersionString = ..Module.VersionString
-        if (..Module.Deployed) {
+        // Use the lifecycle's PACKAGING parameter to determine the publish type, not the module's


This is kind of confusing. Thoughts on changing the wording of this parameter to BuildPackaging or something like that?

isc-dchui requested review from isc-cborbonm, isc-egabhart, isc-eneil, isc-jili, isc-jlechtne, isc-kiyer, isc-pbarton and isc-tleavitt as code owners April 13, 2026 19:41

isc-dchui force-pushed the db-packaging branch from 6884e70 to ccc3b3e Compare April 13, 2026 19:42

isc-dchui added this to the v.0.11.0 milestone Apr 22, 2026

isc-jili requested changes Apr 27, 2026

View reviewed changes

isc-dchui force-pushed the db-packaging branch 2 times, most recently from bba26d8 to 216223e Compare June 1, 2026 19:58

isc-dchui added 17 commits July 6, 2026 15:26

Add tests first

672c80e

First pass implementation of package-database

b0451f8

First pass on installing database-packaged module

ca723ee

Bug fixing and test improvement

6553c68

Handle non-compiled files and test files with some bug fixes and tests

ea4b597

All tests pass now

42cdb46

Refactor

e6ccf85

Fix python requirements.txt gap, add some tests, and minor refactoring

acdd917

Cleanup after new tests

7b932b9

Fix gap with invoke timing

204b8ad

Refactor tests to carefully reuse namespace for performance gains

93c83fc

Minor refactor and remove test timing

fcfd6e5

Properly handle update steps during database packaging/installing

02d764d

Add publish-database command for publishing a database-packaged modul…

9cc5d3b

…e in one call

Add changelog and merge two tests into one

9f553a7

Readd comment

2612eff

Adjust GitHub CI phase steps timing

5eb3344

isc-dchui added 6 commits July 6, 2026 15:27

Combine python tests into one

224e839

Add source vs database-packaged module filtering in ORAS with OCI tag…

d2e0933

… and packaging property

Address review comments

80bf3b3

Reorganize methods

73ddc0f

Update to new version of IPM

95043b8

Add check for mirrored databases

079678d

isc-dchui force-pushed the db-packaging branch from 216223e to 079678d Compare July 6, 2026 19:28

isc-cborbonm reviewed Jul 13, 2026

View reviewed changes

Comment thread CHANGELOG.md

isc-eneil reviewed Jul 20, 2026

View reviewed changes

Uh oh!

Conversation

isc-dchui commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Overview

Key Details

Testing

Not Yet Handled

Checklist

Uh oh!

isc-jili left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

isc-dchui commented Apr 13, 2026 •

edited

Loading