fix: disable APT timers during Supabase upgrades

[cfunkhouser]

Disable APT systemd timers during the upgrade process. Generally, we wish for system state to be as static and predictable as possible during the upgrade process, and these timers trigger jobs which can mutate system state.

Also builds on #2356 to address some outstanding issues; this change also:

• Finishes the migration to the log utility for all remaining scripts
• Replaces global lock timeout with --skip-locked on vacuumdb
• One run_sql call per statement with inline timeout, instead of batch
• set -e suppression in execute_{,extension_upgrade_}patches
• conninfo string sanitization

INDATA-1220

[Crispy1975]

LGTM. I suppose the only thing I can think of here is that if there is a hard fail and the ERR trap isn't triggered we are left with no running Salt service (say from an OOM). Not a hard blocker.

[hunleyd]

4 sites (3 in initiate.sh, 1 in complete.sh) call bare exit 1 after timers are disabled. exit does not fire trap cleanup ERR; the restore never runs, full stop. The diff's own comment at initiate.sh:380 states the exact rule ("failing as a normal command, not exit 1, lets the ERR trap run cleanup") and follows it later in the same function just not at these 3 earlier sites.

enable_conflicting_timers restores state it never captured, and a failed stop can't converge under retry. It re-enables every timer in the list that's currently disabled, not just what this run disabled so a timer an operator deliberately turned off gets silently turned back on. Separately, a failed systemctl stop was swallowed with || true, so retry 3 could never detect or retry it.

[tomashley]

I understand the desire to stop salt running during processes such as upgrades and suspends.
I'm not keen on the approach of disabling the service. If for any reason the service is not enabled again, we lose contact with it.
It's not an easy task, very manual, to recover projects that have lost contact with salt.

I developed a feature in SAA whereby you can add an EC2 tag to an instance to stop it from running salt:
https://github.com/supabase/supabase-admin-agent/blob/main/pkg/salt/workflow.go#L21-L23

EC2 tag salt-enabled=false

By doing this, the timer can remain, but salt will not execute.
If at any point we 'lose' contact with the project, we can remove the EC2 tag and easily get back into salt runs.

I'm not going to block this PR and it's approach - adding the EC2 tag will need to come via the worker at upgrade time and is a bit more involved. But I do think it's a more elegant solution to this problem that could be assessed.

Suspend and Wake will be making use of this EC2 tag during their Hibernation states to stop salt from running and restarting postgres.

[cfunkhouser]

The tag-based approach in SAA sounds great, since most of the concerns in this PR are "what happens if we don't re-enable." Let me see how difficult it will actually be to implement that; since the upgrade process involves a number of pgBoss jobs anyway, perhaps we can just shim it in now. Thanks for pointing that out, @tomashley.

[cfunkhouser]

I've decided to remove the Salt timer from this PR, but leave everything else in place for handling the APT timers. I believe I've addressed all outstanding comments by switching the {enable,disable}_conflicting_timers functions to runtime-mask the units, instead of disabling them. Runtime masking in SystemD relies on symlinks in /run, which is a tmpfs and will be cleared on reboot, putting the timers back in their pre-twiddling state.