Skip to content

fix: track created disks in the state file so an interrupt cannot orphan them - #160

Merged
tas50 merged 1 commit into
mainfrom
fix/track-created-disks-in-state
Aug 24, 2026
Merged

fix: track created disks in the state file so an interrupt cannot orphan them#160
tas50 merged 1 commit into
mainfrom
fix/track-created-disks-in-state

Conversation

@tas50

@tas50 tas50 commented Aug 23, 2026

Copy link
Copy Markdown
Member

The bug

Pressing Ctrl-C during a kitchen create that uses extra disks orphans a billable disk with no record of it anywhere:

=== disks ===
tk-datadisk-ubuntu-2204-fa573c-extra-disk  20GB  (no users)

=== state file ===
--- {}

=== kitchen destroy — the user's only cleanup lever ===
       Finished destroying <datadisk-ubuntu-2204> (0m0.00s)

=== disks after destroy ===
tk-datadisk-ubuntu-2204-fa573c-extra-disk  20GB   <- still there, still billing

Why

created_disk_names was @created_disk_names ||= [] — in memory only. The cleanup that consumes it hangs off create's rescue => e, which catches StandardError. Interrupt is not a StandardError, so Ctrl-C skips the rescue, and the process exits with the only record of the disk in a variable that just died with it.

Standalone disks are created before insert_instance and bill from creation, so this window is wide open — it is the whole "Creating a N GB disk / Waiting for disk to be ready" phase.

The fix

Record the disks in the state file, which Test Kitchen writes whatever happens (ensure state_file.write(state)), and have destroy delete anything still recorded once the instance is gone. This is the treatment the instance itself already gets.

Two supporting changes make that reachable:

  1. Pin the zone before anything billable exists. In region mode the zone is chosen at random, so cleanup would otherwise have no idea where to look.
  2. Record the server name before the insert is issued, not after it returns. Live testing caught this: on one interrupt GCE created the instance but the request never returned, so state had the disk but not the instance holding it, and cleanup failed with resourceInUseByAnotherResource. Recording the name first closes that window, and fix: clear the state file when the instance is already gone #159 makes destroy tolerate a name that never became an instance.

Verification

# state file after Ctrl-C — now has everything cleanup needs
zone: us-central1-a
server_name: tk-datadisk-ubuntu-2204-0a8388
created_disks:
- tk-datadisk-ubuntu-2204-0a8388-extra-disk
kitchen destroy
       GCE instance <tk-datadisk-ubuntu-2204-0a8388> destroyed.
       Unable to locate disk tk-datadisk-...-extra-disk   <- already gone with the instance

Final state: 0 instances, 0 disks.

Base automatically changed from fix/destroy-clears-stale-state to main August 23, 2026 18:35
@tas50
tas50 force-pushed the fix/track-created-disks-in-state branch 2 times, most recently from c72bc4b to de43dfa Compare August 23, 2026 18:51
…han them

Standalone disks were recorded in an instance variable, and the cleanup
that deletes them hangs off `rescue => e`, which catches StandardError.
Interrupt is not a StandardError, so Ctrl-C during a create -- the most
common way a create ends early -- skipped cleanup entirely and left a
billable disk behind with no record of it anywhere:

    disks:  tk-datadisk-...-extra-disk  20GB  (no users)
    state:  --- {}
    kitchen destroy -> Finished destroying (0m0.00s)

Record the disks in the Test Kitchen state file instead, which is written
whatever happens, and have `destroy` delete anything still recorded once
the instance is gone. This mirrors what the instance itself already does.

Two supporting changes are needed to make that reachable:

- Pin the zone before any billable resource exists. In region mode it is
  chosen at random, so cleanup would not otherwise know where to look.
- Record the server name before the insert request is issued rather than
  after it returns. Live testing turned up a case where GCE created the
  instance but the interrupted request never returned, so the state file
  had the disk but not the instance holding it, and cleanup failed with
  resourceInUseByAnotherResource.

Confirmed against a real project: an interrupted create is now fully
reclaimed by `kitchen destroy`, leaving no instances and no disks.
@tas50
tas50 force-pushed the fix/track-created-disks-in-state branch from de43dfa to 4d37187 Compare August 23, 2026 19:44
@tas50
tas50 merged commit 913eb59 into main Aug 24, 2026
8 checks passed
@tas50
tas50 deleted the fix/track-created-disks-in-state branch August 24, 2026 00:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant