TableScan meets the issue pools_.size() != 0 (1 vs 0)

[boneanxs]

Backend

VL (Velox)

Bug description

emory Allocator[MALLOC capacity UNLIMITED allocated bytes 2097152 allocated pages 4 mapped pages 0]
ARBITRATOR[GLUTEN] CAPACITY 8388608.00TB numRequests 0 numRunning 0 numSucceded 0 numAborted 0 numFailures 0 numNonReclaimableAttempts 0 reclaimedFreeCapacity 0B reclaimedUsedCapacity 0B maxCapacity 0B freeCapacity 0B freeReservedCapacity 0B], Source: RUNTIME, ErrorCode: INVALID_STATE
terminate called after throwing an instance of 'facebook::velox::VeloxRuntimeError'
  what():  Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: pools_.size() != 0 (1 vs 0). There are unexpected alive memory pools allocated by user on memory manager destruction:
Memory Manager[capacity UNLIMITED alignment 64B usedBytes 16.00KB number of pools 4
List of root pools:
__sys_root__ usage 0B reserved 0B peak 0B
    __sys_shared_leaf__0 usage 0B reserved 0B peak 0B
    __sys_tracing__ usage 0B reserved 0B peak 0B
    __sys_caching__ usage 0B reserved 0B peak 0B
    __sys_spilling__ usage 0B reserved 0B peak 0B


Top 1 leaf memory pool usages:
    op.0.0.0.TableScan usage 16.00KB reserved 1.00MB peak 151.18MB

root usage 16.00KB reserved 1.00MB peak 154.00MB
    task.test_cursor_605 usage 16.00KB reserved 1.00MB peak 154.00MB
        node.0 usage 16.00KB reserved 1.00MB peak 152.00MB
            op.0.0.0.TableScan usage 16.00KB reserved 1.00MB peak 151.18MB

	refcount 2
Memory Allocator[MALLOC capacity UNLIMITED allocated bytes 2097152 allocated pages 4 mapped pages 0]
ARBITRATOR[GLUTEN] CAPACITY 8388608.00TB numRequests 0 numRunning 0 numSucceded 0 numAborted 0 numFailures 0 numNonReclaimableAttempts 0 reclaimedFreeCapacity 0B reclaimedUsedCapacity 0B maxCapacity 0B freeCapacity 0B freeReservedCapacity 0B]
Retriable: False
Function: ~MemoryManager
File: /workspaces/gluten/ep/build-velox/build/velox_ep/velox/common/memory/Memory.cpp
Line: 196
Stack trace:
# 0  _ZN8facebook5velox7process10StackTraceC1Ei
# 1  _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2  _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEvRKNS1_18VeloxCheckFailArgsET0_
# 3  0x000000000229e776
# 4  _ZN6gluten18VeloxMemoryManagerD1Ev
# 5  _ZN6gluten18VeloxMemoryManagerD0Ev
# 6  _ZN6gluten13MemoryManager7releaseEPS0_
# 7  Java_org_apache_gluten_memory_NativeMemoryManagerJniWrapper_release
# 8  0x00007f21842497ea
# 9  0x00007f217d9d9ca4

Looks this issue still persist even after the #9316 (comment)

cc @FelixYBW @rui-mo

Gluten version

main branch

Spark version

Spark-4.1.x

Spark configurations

No response

System information

No response

Relevant logs

[boneanxs]

Might relate to the velox facebookincubator/velox#12729

[FelixYBW]

12729 is already picked into gluten.

Is it reproducible?

[boneanxs]

12729 is already picked into gluten.

Is it reproducible?

Yes, it is reproducible, I mean facebookincubator/velox#12729 might cause this issue.

Our instrumentation shows that the async load submitted by DirectBufferedInput::makeLoads() can still be queued after Gluten's 30s runtime teardown timeout; it has not started running. The AsyncLoadHolder introduced by facebookincubator/velox#12729 intentionally retains a shared_ptr while that queued closure exists, so the task root pool remains alive (refcount 2) even though its usage is 0.

Gluten then proceeds to destroy NativeMemoryManager after the timeout, and Velox reports pools_.size() != 0.

[FelixYBW]

before Gluten's 30s teardown, all queued tasks are either cancelled or wait() to finish here https://github.com/apache/gluten/pull/11697/changes

the sequence is confirmed here:
#11452 (comment)

If we still see outstanding memory requests before ~VeloxMemoryManager(), there should be a similar leak like https://github.com/facebookincubator/velox/pull/16979/changes

[FelixYBW]

Oh, I see the issue. It's because load->cancel(); doesn't really release the memory pool's shared_ptr, even after a 30s timeout; the load is still alive.

@rui-mo Let's see how we can make sure the load does be destroyed

@boneanxs see if it works: https://github.com/apache/gluten/pull/12919/changes

[philo-he]

This issue hasn't been reported in a long time. Is there a reproducible test that can be included in Gluten?