Avoid CN OOM by Pulling User/Roles to DN When Cache Misses. #16888
Conversation
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/auth/ClusterAuthorityFetcher.java
Show resolved
Hide resolved
wenyanshi-123
force-pushed
the
fixOOMForPipe
branch
from
eeb9e25 to
f768082
Compare
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/auth/ClusterAuthorityFetcher.java
Show resolved
Hide resolved
|
And, how is this PR related to PIPE? |
wenyanshi-123
force-pushed
the
fixOOMForPipe
branch
from
f768082 to
4028775
Compare
wenyanshi-123
changed the title
wenyanshi-123
force-pushed
the
fixOOMForPipe
branch
from
4028775 to
b602ee6
Compare
| Role cachedRole = iAuthorCache.getRoleCache(rolename); | ||
| if (cachedRole == null) { | ||
| checkRoleFromConfigNode(username, rolename); | ||
| cachedRole = iAuthorCache.getRoleCache(rolename); | ||
| } |
There was a problem hiding this comment.
There is still a chance that the role is evicted between line 183 and 184, therefore, cachedRold can still be null.
One possible solution is to make checkRoleFromConfigNode() return the Role it just fetched.
And may rename checkRoleFromConfigNode() to fetchRoleFromConfigNode() which is more precise.
There was a problem hiding this comment.
And what if the role is deleted concurrently? You can still not fetch it.
There was a problem hiding this comment.
I have resolved the possible NPE issue, but the concurrency problem requires a more systematic design.
|
Avoid using force-push, which will mix your new changes with the old ones. |
3 participants
When there is no User Cache on the DN side, the entire time series will be handed over to CN for authentication, which ultimately leads to CN OOM in some PIPE scenarios with a large number of time series. To solve this problem, it is necessary to prohibit passing all time series to CN during authentication, and instead pull the user and it's corresponding roles to dn when the user cache and role caches fail.