Skip to content

[refactor](statistics) Organize statistics classes by responsibility - #67483

Draft
morrySnow wants to merge 2 commits into
apache:masterfrom
morrySnow:statistics-dir
Draft

[refactor](statistics) Organize statistics classes by responsibility#67483
morrySnow wants to merge 2 commits into
apache:masterfrom
morrySnow:statistics-dir

Conversation

@morrySnow

@morrySnow morrySnow commented Sep 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: N/A

Related PR: N/A

Problem Summary:

The org.apache.doris.statistics package mixed optimizer-facing statistics models with collection workflows, cache internals, and statistics-table persistence, making class ownership difficult to identify.

This PR reorganizes those classes by responsibility:

  • keeps only cross-cutting statistics definitions in the statistics root package;
  • moves statistics values, histograms, builders, ranges, and metric types to statistics.model;
  • moves collection jobs, tasks, scheduling, metadata, descriptors, and update events to statistics.analysis;
  • moves cache loaders, keys, invalidation, refresh, and synchronization targets to statistics.cache;
  • moves internal statistics-table access, persisted row models, and cleanup to statistics.repository;
  • mirrors the same package layout in unit tests and documents the boundaries in README.md and package-info.java.

No runtime logic or persisted field format changes.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Test result

  • Statistics-related FE unit tests: 202 passed, 0 failures, 0 errors.
  • FE main and test source compilation passed in the unit-test reactor; Checkstyle reported 0 violations.
  • bash build-support/tests/test-fe-connector-imports.sh: passed.
  • Full-tree scan found no references to the moved classes under their old package names.

@morrySnow
[refactor](fe) Organize statistics classes by responsibility
ec0f450 
### What problem does this PR solve?

Issue Number: N/A

Related PR: N/A

Problem Summary: The statistics package mixed optimizer-facing value models with collection workflows, cache internals, and statistics-table persistence, making ownership difficult to identify. Move those classes into analysis, cache, and repository packages, mirror the test layout, update all imports, and document the package boundaries without changing runtime logic.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - Statistics-related FE unit tests: 109 passed
    - FE build and Checkstyle passed with MVN_OPT=-Dmaven.test.skip=true ./build.sh --fe
- Behavior changed: No
- Does this need documentation: No
@hello-stephen

hello-stephen commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morrySnow morrySnow changed the title [refactor](fe) Organize statistics classes by responsibility [refactor](statistics) Organize statistics classes by responsibility Sep 3, 2026
@morrySnow
morrySnow marked this pull request as draft September 3, 2026 09:00
@morrySnow
[refactor](fe) Separate statistics value models
9aa7a7f 
### What problem does this PR solve?

Issue Number: N/A

Related PR: N/A

Problem Summary: Move shared statistics value objects, histograms, builders, ranges, and metric types into the statistics.model package, and move collection-specific descriptors into statistics.analysis. This completes the package responsibility split without changing runtime behavior.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - ./run-fe-ut.sh --run <24 statistics test classes> (202 tests)
    - bash build-support/tests/test-fe-connector-imports.sh
- Behavior changed: No
- Does this need documentation: No
@morrySnow

morrySnow commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

hello-stephen commented Sep 3, 2026

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 16794 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9aa7a7ffe2389aefecc47149bc41619156358154, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17566	3098	3094	3094
q2	2108	258	224	224
q3	10225	881	488	488
q4	4670	251	200	200
q5	7680	599	376	376
q6	135	111	92	92
q7	589	506	413	413
q8	9235	963	920	920
q9	3451	2379	2373	2373
q10	6514	852	707	707
q11	414	198	183	183
q12	624	257	196	196
q13	18134	1546	1135	1135
q14	159	152	136	136
q15	q16	431	388	366	366
q17	1405	889	755	755
q18	3080	2244	2261	2244
q19	1111	892	777	777
q20	381	278	197	197
q21	5059	1696	1850	1696
q22	327	275	222	222
Total cold run time: 93298 ms
Total hot run time: 16794 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3485	3396	3410	3396
q2	500	392	369	369
q3	2197	2473	2229	2229
q4	1194	1163	901	901
q5	2156	2102	2107	2102
q6	167	114	86	86
q7	1095	899	858	858
q8	1627	1431	1433	1431
q9	3136	3138	3093	3093
q10	1864	1802	1602	1602
q11	357	271	254	254
q12	452	426	351	351
q13	1473	1534	1146	1146
q14	170	168	163	163
q15	q16	398	396	355	355
q17	3578	3229	3192	3192
q18	4795	4412	4709	4412
q19	858	815	944	815
q20	1023	956	831	831
q21	3897	3178	3338	3178
q22	406	340	338	338
Total cold run time: 34828 ms
Total hot run time: 31102 ms

@hello-stephen

hello-stephen commented Sep 3, 2026

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 82230 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9aa7a7ffe2389aefecc47149bc41619156358154, data reload: false

query5	4250	402	348	348
query6	381	142	121	121
query7	4954	439	216	216
query8	282	121	130	121
query9	8719	2898	2915	2898
query10	392	209	190	190
query11	5391	1029	932	932
query12	112	71	76	71
query13	1201	463	321	321
query14	6047	2197	2086	2086
query14_1	2003	1995	1958	1958
query15	170	120	116	116
query16	948	356	348	348
query17	811	455	364	364
query18	2329	332	246	246
query19	165	140	112	112
query20	70	70	70	70
query21	200	102	87	87
query22	5453	5420	5357	5357
query23	6607	6198	6113	6113
query23_1	5980	6143	6030	6030
query24	7293	1109	775	775
query24_1	782	767	806	767
query25	433	308	262	262
query26	1240	241	131	131
query27	2773	404	247	247
query28	4721	1507	1522	1507
query29	942	451	354	354
query30	253	154	136	136
query31	834	414	356	356
query32	130	73	77	73
query33	466	228	198	198
query34	984	858	491	491
query35	418	411	342	342
query36	572	555	547	547
query37	131	85	77	77
query38	1023	846	846	846
query39	518	470	476	470
query39_1	475	436	477	436
query40	201	92	85	85
query41	63	57	61	57
query42	81	73	72	72
query43	244	244	214	214
query44	1027	542	567	542
query45	117	111	101	101
query46	793	852	534	534
query47	751	761	702	702
query48	302	307	224	224
query49	546	230	183	183
query50	761	260	203	203
query51	8338	8076	8023	8023
query52	69	67	57	57
query53	200	205	149	149
query54	214	159	145	145
query55	87	76	78	76
query56	231	162	165	162
query57	683	651	654	651
query58	193	176	157	157
query59	1252	1240	1120	1120
query60	236	194	170	170
query61	140	113	115	113
query62	387	196	181	181
query63	170	145	143	143
query64	2819	734	581	581
query65	1642	1576	1561	1561
query66	1795	260	198	198
query67	9517	9585	9610	9585
query68	2914	1254	720	720
query69	360	228	189	189
query70	665	617	634	617
query71	255	186	168	168
query72	2363	1699	1586	1586
query73	652	634	331	331
query74	1850	1236	1170	1170
query75	1189	1110	965	965
query76	2312	730	556	556
query77	245	262	209	209
query78	4219	3598	3232	3232
query79	2824	863	574	574
query80	1591	338	295	295
query81	525	154	131	131
query82	637	124	98	98
query83	283	213	192	192
query84	294	110	90	90
query85	857	362	292	292
query86	487	178	166	166
query87	1018	983	906	906
query88	3204	2150	2095	2095
query89	274	196	176	176
query90	2186	126	126	126
query91	133	120	102	102
query92	100	68	72	68
query93	2916	1134	701	701
query94	631	271	221	221
query95	505	313	223	223
query96	830	563	267	267
query97	1029	1038	976	976
query98	176	160	141	141
query99	434	352	308	308
Total cold run time: 180704 ms
Total hot run time: 82230 ms

@hello-stephen

hello-stephen commented Sep 3, 2026

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.67 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9aa7a7ffe2389aefecc47149bc41619156358154, data reload: false

query1	0.00	0.00	0.01
query2	0.08	0.04	0.04
query3	0.25	0.10	0.10
query4	1.60	0.10	0.10
query5	0.17	0.16	0.15
query6	1.25	0.69	0.68
query7	0.03	0.00	0.01
query8	0.05	0.04	0.03
query9	0.28	0.22	0.21
query10	0.36	0.34	0.35
query11	0.16	0.12	0.11
query12	0.14	0.12	0.11
query13	0.31	0.31	0.31
query14	0.43	0.44	0.45
query15	0.36	0.35	0.35
query16	0.23	0.22	0.23
query17	0.67	0.72	0.69
query18	0.18	0.17	0.18
query19	1.16	1.19	1.17
query20	0.01	0.01	0.01
query21	15.44	0.18	0.11
query22	5.05	0.04	0.04
query23	16.18	0.25	0.10
query24	3.05	0.31	0.27
query25	0.11	0.04	0.03
query26	0.81	0.17	0.11
query27	0.03	0.03	0.04
query28	3.69	0.57	0.30
query29	12.42	3.21	2.56
query30	0.26	0.12	0.13
query31	2.75	0.37	0.17
query32	3.54	0.33	0.22
query33	1.45	1.40	1.62
query34	15.38	2.24	1.77
query35	1.77	1.70	1.70
query36	0.48	0.28	0.29
query37	0.06	0.03	0.04
query38	0.04	0.03	0.03
query39	0.03	0.02	0.02
query40	0.12	0.08	0.08
query41	0.08	0.02	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 90.53 s
Total hot run time: 14.67 s

@hello-stephen

hello-stephen commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 1.64% (1/61) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou Awaiting requested review from gavinchou gavinchou is a code owner
@englefly englefly Awaiting requested review from englefly englefly is a code owner
@924060929 924060929 Awaiting requested review from 924060929 924060929 is a code owner
@mymeiyi mymeiyi Awaiting requested review from mymeiyi mymeiyi is a code owner
@starocean999 starocean999 Awaiting requested review from starocean999 starocean999 is a code owner
@CalvinKirs CalvinKirs Awaiting requested review from CalvinKirs CalvinKirs is a code owner
@luwei16 luwei16 Awaiting requested review from luwei16 luwei16 is a code owner
@dataroaring dataroaring Awaiting requested review from dataroaring dataroaring is a code owner
@morningman morningman Awaiting requested review from morningman morningman is a code owner
@seawinde seawinde Awaiting requested review from seawinde seawinde will be requested when the pull request is marked ready for review seawinde is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@morrySnow @hello-stephen