[improvement](be): reduce duplicate key memtable memory

[zghong]

What problem does this PR solve?

Issue Number: close #67505

Problem Summary:

What does this PR optimize?

For DUP_KEYS memtables, each input row was previously represented by a shared_ptr<RowInBlock>. However, duplicate-key tables do not require the per-row aggregation state stored in RowInBlock; they only need the source row position for sorting.

This PR introduces a compact representation for DUP_KEYS:

• Store row positions in a contiguous uint32_t vector instead of allocating a shared_ptr<RowInBlock> for every row.
• Keep _row_in_blocks only for UNIQUE_KEYS and AGG_KEYS, which still need per-row aggregation state.
• Generalize the sorting logic so that it supports both compact row positions and RowInBlock objects.
• Store allocated LSNs in a sidecar vector and apply the same permutation as the sorted rows, preserving row-to-LSN correspondence.
• Preserve the existing ordering semantics for duplicate keys, where newer rows are ordered before older rows with the same key.
• Skip sorting entirely for zero-key duplicate tables and retain their physical insertion order.

Benchmark

The benchmark measures the row-index construction path directly affected by this change. It was run with a Release build and five repetitions:

./be/output/lib/benchmark_test \
    --benchmark_filter='BM_DuplicateKeyRowIndex.*' \
    --benchmark_repetitions=5 \
    --benchmark_report_aggregates_only=true

CPU mean results:

Rows	Previous shared_ptr<RowInBlock>	Compact uint32_t index	Speedup	Time reduction
4,096	132 us	0.370 us	356.76x	99.72%
65,536	3,186 us	9.50 us	335.37x	99.70%
1,048,576	49,139 us	305 us	161.11x	99.38%

Estimated row-metadata footprint:

Representation	Estimated bytes per row
Previous shared_ptr<RowInBlock> representation	56 bytes
Compact row position without LSN	4 bytes
Compact row position with LSN	12 bytes

This reduces the estimated row-metadata footprint by:

• 92.86% without LSNs (56 -> 4 bytes/row), approximately 14x more compact.
• 78.57% with LSNs (56 -> 12 bytes/row), approximately 4.67x more compact.

The previous estimate does not include the allocation/control-block overhead of make_shared, so the actual memory reduction may be larger.

Note that this is a focused microbenchmark for row-index construction and does not represent the end-to-end performance improvement of a complete load job.

Release note

Reduce duplicate-key load memory use and row-index construction overhead.

Check List (For Author)

• Test

  • Regression test
  • Unit Test
  • Manual test (add detailed scripts or steps below)
  • No need to test or manual test. Explain why:
    • This is a refactor/code format and no logic has been changed.
    • Previous test can cover this change.
    • No code files have been changed.
    • Other reason

• Behavior changed:

  • No.
  • Yes.

• Does this need documentation?

  • No.
  • Yes.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[zghong]

run buildall

[hello-stephen]

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	63.02% (29554/46894)
Line Coverage	48.07% (309927/644761)
Region Coverage	43.65% (250415/573696)
Branch Coverage	45.24% (116534/257598)

[hello-stephen]

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	76.25% (34635/45420)
Line Coverage	61.32% (390866/637398)
Region Coverage	57.44% (328296/571512)
Branch Coverage	58.30% (149758/256859)