HIVE-29322: Avoid TopNKeyOperator When Map-Side LIMIT Pushdown Provides Better Pruning for ORDER BY LIMIT Queries #6202
Conversation
Indhumathi27
force-pushed
the
disable_topn
branch
from
3a7b050 to
cbb41cf
Compare
asf-ci-hive
added
tests pending
tests unstable
and removed
tests unstable
tests pending
labels
Indhumathi27
changed the title
Indhumathi27
force-pushed
the
disable_topn
branch
from
cbb41cf to
75f9861
Compare
|
@kasakrisz @deniskuzZ Can you help to review this PR. Thanks |
Indhumathi27
force-pushed
the
disable_topn
branch
4 times, most recently
from
db493f2 to
a003d03
Compare
Indhumathi27
changed the title
asf-ci-hive
added
tests pending
tests passed
and removed
tests unstable
tests pending
labels
Indhumathi27
force-pushed
the
disable_topn
branch
from
a003d03 to
bc90bb4
Compare
asf-ci-hive
added
tests pending
tests passed
and removed
tests passed
tests pending
labels
|
@ayushtkn @zabetak @kasakrisz @deniskuzZ Can you help to review the PR. thanks |
ql/src/java/org/apache/hadoop/hive/ql/optimizer/LimitPushdownOptimizer.java
Outdated
Show resolved
Hide resolved
| // Skip the current optimization when a simple global ORDER BY...LIMIT is present | ||
| // (topN > -1 and hasOnlyOrderByLimit()). | ||
| // This plan structure is handled more efficiently by the specialized 'TopN In Reducer' optimization. | ||
| if (reduceSinkDesc.getTopN() > -1 && reduceSinkDesc.hasOnlyOrderByLimit()) { |
There was a problem hiding this comment.
WDYT @zabetak, @kasakrisz , @thomasrebele, @okumin, @ngsg
There was a problem hiding this comment.
I'm also curious about the two questions in the JIRA ticket. I don't have an instant answer on whether it is optimal to reuse Top-N for the global sort. However, in my impression, it is at least suboptimal and might not be so bad.
Disclaimer: My next response might not be timely because I have to visit a hospital every day.
Indhumathi27
force-pushed
the
disable_topn
branch
from
bc90bb4 to
035e8bf
Compare
asf-ci-hive
added
tests pending
tests unstable
and removed
tests passed
tests pending
labels
…es Better Pruning
Indhumathi27
force-pushed
the
disable_topn
branch
from
035e8bf to
4e3f88b
Compare
|
|
I left some questions under the JIRA ticket to better understand the issue that we are trying to solve here. |
zabetak
left a comment
zabetak
left a comment
There was a problem hiding this comment.
I went over the plan changes and they look reasonable based on the goal of this PR. In terms of changes in the optimizer I left a few comments that could make the code a bit easier to follow.
| sort order: ++ | ||
| Statistics: Num rows: 1 Data size: 178 Basic stats: COMPLETE Column stats: COMPLETE | ||
| value expressions: _col1 (type: string) | ||
| TopN Hash Memory Usage: 0.1 |
There was a problem hiding this comment.
When the plan contains a Limit operator all subsequent TopN optimizations in the same mapper/reducer seem redundant. It's out of scope of the current PR but it may be worth logging a separate JIRA ticket as a potential improvement.
| private transient boolean hasOrderBy = false; | ||
|
|
||
| // used to decide whether topn key optimisation can be applied | ||
| private transient boolean hasOnlyOrderByLimit = false; |
There was a problem hiding this comment.
Do we really need this new indicator here? IT seems to be a hint about the structure of the plan rather than a property/quality of the ReduceSink operator.
| if (operator instanceof GroupByOperator || operator instanceof JoinOperator) { | ||
| hasOnlyOrderByLimit = false; |
There was a problem hiding this comment.
The decision to introduce or not a TopNKeyOperator is a responsibility of the the TopNKeyProcessor so if we need to make decisions based on the structure of the plan it makes more sense to do put the GBY/JOIN checks there.
| // Skip the current optimization when a simple global ORDER BY...LIMIT is present | ||
| // (topN > -1 and hasOnlyOrderByLimit()). | ||
| // This plan structure is handled more efficiently by the specialized 'TopN In Reducer' optimization. | ||
| if (reduceSinkDesc.getTopN() > -1 && reduceSinkDesc.hasOnlyOrderByLimit()) { |
There was a problem hiding this comment.
It seems that we don't want to introduce a TopNKeyOperator if the path between TS and RS does not contain a JOIN or GBY operator. We can add such checks at some point here or maybe better modify the respective RuleRegExp to only trigger when there is a GBY or JOIN in the path.
5 participants
What changes were proposed in this pull request?
This PR updates TopNKeyProcessor to skip creating TopNKeyOperator when ReduceSinkDesc.topN is already set by LIMIT pushdown for ORDER BY LIMIT case. This prevents TopNKey from overriding pushdown and ensures the map-side LIMIT optimization is applied correctly.
Why are the changes needed?
Currently, when a query includes ORDER BY + LIMIT, LIMIT pushdown is generated during planning but is effectively overridden by the subsequent TopNKey rewrite. As a result, TopNKey operator receives full input rather than a reduced data set, leading to worse performance (e.g., 16M rows forwarded to reducer instead of a few). In cases where global ordering uses a single reducer, LIMIT pushdown is sufficient and far more efficient. This fix prevents unnecessary TopNKey creation so that pushdown can reduce shuffle and significantly improve execution time.
Test reports:
For query: select * from table order by h limit 100;
Total num of rows: 67764224
with fix:
without fix:
Does this PR introduce any user-facing change?
No
How was this patch tested?
Manual testing + existing testcases