Unexpected disk usage running acquisitions

[SJShep]

We're seeing unexpected disk usage on the C drive when running acquisitions, even though data is being saved to another drive.

• The pymmcore-log file is a reasonable size, so it's likely not the main cause.
• Disk memory was freed after exiting the gui.
• This behavior wasn't present in earlier versions of pymmcore-gui. Perhaps it is related to NDViewer for live preview?
• pymmcore-gui version: 0.1.dev336+g80f5b4c
• pymmcore-plus version: 0.13.4

Could you help us identify where this disk usage is coming from and how to prevent it?

[tlambert03]

Perhaps it is related to NDViewer for live preview?

That would be a prime candidate for memory leaks... (and almost certainly does currently have some memory leaks) but I'm struggling to think of why it would have disk usage.

Can you tell me where on the C drive the space is being written to? you're just noticing "free space go down" but not sure where it's going? Can you run a disk inspection tool to see what exactly is taking up the space? like windirstat?

[SJShep]

Correct, we just notice the free space go down. We just had another clue, in the live preview mode this error has come up a couple times, and it repeats until the window stops responding. In this session we imaged a few sections and then while in the live preveiw had a crash. The C drive started at ~60Gb remaining and now has 12Gb:

C:\Users\qi2lab\miniforge3\envs\opm-pymmcore-gui\Lib\site-packages\ndv\views\_qt\_main_thread.py:27: UserWarning: Error fetching data: Unable to allocate 606. MiB for an array with shape (77, 2172, 1900) and data type uint16
  result = func(*args, **kwargs)

I will run the disk inspection tool next.

Thanks for the help!

[SJShep]

Looking at the windirstat results, I think the "pagefile.sys" grew by ~30Gb after running pymmcore-gui.

[gselzer]

In this session we imaged a few sections and then while in the live preveiw had a crash. The C drive started at ~60Gb remaining and now has 12Gb:

C:\Users\qi2lab\miniforge3\envs\opm-pymmcore-gui\Lib\site-packages\ndv\views\_qt\_main_thread.py:27: UserWarning: Error fetching data: Unable to allocate 606. MiB for an array with shape (77, 2172, 1900) and data type uint16

The NDV live preview retains the previous 99 (by default) images coming out of micro-manager. If all of those images are of size (77, 2172, 1900), I'm not surprised that you would run out of space after a number of snaps. @tlambert03 we probably need to make the size of this buffer configurable from the GUI, huh?

[tlambert03]

If all of those images are of size (77, 2172, 1900)

they shouldn't each be that size... they should each be a 2d image... and even at that size, 100 of those 2D images at uint16 should be less than a single GB. (~800MB), which is well below the pagefile problem of 30GB

I certainly do think it should be configurable... but I'm not convinced that alone would cause this yet.

My suspicion instead is that closing the image preview window isn't recovering the ram, causing memory leaks, and which is in turn causing the pagefile to grow as windows attempt to satisfy new requests.

I'll look into that, but @SJShep, in the meantime if I were you, I'd just pin your group's pyproject.toml to commit f0eb39a here rather than using @main as you currently do ...

[tlambert03]

also @SJShep, could you try one thing for me? I need you to modify your installation of pymmcore-gui. So, either directly edit the pymmcore-gui code inside of your site-packages... or fork this repo, clone it locally, create a branch for edits, and then pip install your local version into your environment. And then, change this line right here:

pymmcore-gui/src/pymmcore_gui/_main_window.py

Lines 506 to 508 in 80f5b4c

	# small hack ... we need to retain a pointer to the viewer
	# otherwise the viewer will be garbage collected
	dw._viewer = ndv_viewer # type: ignore

to

q_viewer._viewer = ndv_viewer

(i.e. bind it to the q_viewer rather than the dock widget)

[tlambert03]

for a little background here. the problem we're trying to avoid with that line 508 is one where the data is gone, but the viewer persists. You will know if you hit that problem if you create a second preview window, and then go back to a previous preview window and find that moving the slider does nothing at all (no new data is shown)

[SJShep]

also @SJShep, could you try one thing for me? I need you to modify your installation of pymmcore-gui. So, either directly edit the pymmcore-gui code inside of your site-packages... or fork this repo, clone it locally, create a branch for edits, and then pip install your local version into your environment. And then, change this line right here:

pymmcore-gui/src/pymmcore_gui/_main_window.py

Lines 506 to 508 in 80f5b4c

small hack ... we need to retain a pointer to the viewer

otherwise the viewer will be garbage collected

dw._viewer = ndv_viewer # type: ignore

to

q_viewer._viewer = ndv_viewer

(i.e. bind it to the q_viewer rather than the dock widget)

After applying this change, we do not see the drive filling as before!

[tlambert03]

very good to know! thanks!

I've made some progress on fixing this the "proper" way with tests to ensure we don't have future regressions. But came across some deeper issues in the process. It's a high priority to fix, so expect a fix in the near future (but do probably pin to f0eb39a until then