HDFS-17909. [ARR] AsyncRouterHandlerExecutors should use bounded queue by kokonguyen191 · Pull Request #8448 · apache/hadoop

kokonguyen191 · 2026-04-21T07:17:19Z

Description of PR

When a lot of requests arrive at a router and there is a backlog due to downstream namespaces not being able to fully handle the requests, routers can go OOM if there is no limit to the queue used by the async handlers.

This situation can be simulated using the unit test attached with this patch.

public static void main(String[] args) throws Exception {
    setUpCluster();
    RemoteMethod method = new RemoteMethod("listOpenFiles", new Class<?>[] {long.class, EnumSet.class, String.class}, 0, EnumSet.of(OpenFilesIterator.OpenFilesType.BLOCKING_DECOMMISSION), new RemoteParam());
    UserGroupInformation ugi = RouterRpcServer.getRemoteUser();
    Class<?> protocol = method.getProtocol();
    String bigPath = "/veryBigOperation";
    Object[] params = new Object[] {0, EnumSet.of(OpenFilesIterator.OpenFilesType.BLOCKING_DECOMMISSION), bigPath};
    List<? extends FederationNamenodeContext> namenodes = asyncRpcClient.getOrderedNamenodes(ns0, true);
    while (true) {
      asyncRpcClient.invokeMethod(ugi, namenodes, true, protocol, method.getMethod(), params);
      asyncReturn(Map.class);
    }
  }

Rising memory usage from unbounded growing queue

How was this patch tested?

This patch has been used in our prod cluster for over a year and everything's working properly.

hfutatzhanghb · 2026-04-21T08:10:44Z

@kokonguyen191 Thanks for pushing this. Could you please provide more details in the description about how this situation was triggered? use pseudocode or any other way to make it clearly. Thanks a lot.

kokonguyen191 · 2026-04-22T09:32:35Z

Hi @hfutatzhanghb, thanks for taking a look at this patch. I have updated the description as well as the unit test for easier issue repro.

hadoop-yetus · 2026-04-22T11:28:08Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 34s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	41m 35s		trunk passed
+1 💚	compile	1m 10s		trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚	compile	1m 33s		trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚	checkstyle	1m 4s		trunk passed
+1 💚	mvnsite	1m 39s		trunk passed
+1 💚	javadoc	1m 8s		trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚	javadoc	1m 1s		trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚	spotbugs	2m 24s		trunk passed
+1 💚	shadedclient	30m 9s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 0s		the patch passed
+1 💚	compile	0m 39s		the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚	javac	0m 39s		the patch passed
+1 💚	compile	0m 59s		the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚	javac	0m 59s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 27s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0)
+1 💚	mvnsite	1m 5s		the patch passed
+1 💚	javadoc	0m 34s		the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
-1 ❌	javadoc	0m 34s	/results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-17.0.18+8-Ubuntu-124.04.1.txt	hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-17.0.18+8-Ubuntu-124.04.1 with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 generated 1 new + 937 unchanged - 0 fixed = 938 total (was 937)
+1 💚	spotbugs	2m 5s		the patch passed
+1 💚	shadedclient	28m 47s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	43m 52s		hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	0m 37s		The patch does not generate ASF License warnings.
		164m 23s

Subsystem	Report/Notes
Docker	ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8448/2/artifact/out/Dockerfile
GITHUB PR	#8448
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux 2b8dcc5a7c7f 5.15.0-173-generic #183-Ubuntu SMP Fri Mar 6 13:29:34 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `2c3f471`
Default Java	Ubuntu-17.0.18+8-Ubuntu-124.04.1
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.10+7-Ubuntu-124.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.18+8-Ubuntu-124.04.1
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8448/2/testReport/
Max. process+thread count	4205 (vs. ulimit of 10000)
modules	C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8448/2/console
versions	git=2.43.0 maven=3.9.11 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2026-04-22T11:52:15Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 53s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	48m 46s		trunk passed
+1 💚	compile	1m 5s		trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚	compile	1m 33s		trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚	checkstyle	0m 57s		trunk passed
+1 💚	mvnsite	1m 39s		trunk passed
+1 💚	javadoc	1m 3s		trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚	javadoc	1m 0s		trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚	spotbugs	2m 34s		trunk passed
+1 💚	shadedclient	35m 36s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 7s		the patch passed
+1 💚	compile	0m 39s		the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚	javac	0m 39s		the patch passed
+1 💚	compile	1m 5s		the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚	javac	1m 5s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 27s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0)
+1 💚	mvnsite	1m 12s		the patch passed
+1 💚	javadoc	0m 32s		the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
-1 ❌	javadoc	0m 33s	/results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-17.0.18+8-Ubuntu-124.04.1.txt	hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-17.0.18+8-Ubuntu-124.04.1 with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1 generated 1 new + 937 unchanged - 0 fixed = 938 total (was 937)
+1 💚	spotbugs	2m 16s		the patch passed
+1 💚	shadedclient	34m 27s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	48m 56s		hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	0m 37s		The patch does not generate ASF License warnings.
		188m 15s

Subsystem	Report/Notes
Docker	ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8448/3/artifact/out/Dockerfile
GITHUB PR	#8448
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux 0f998e8f35ff 5.15.0-173-generic #183-Ubuntu SMP Fri Mar 6 13:29:34 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `2c3f471`
Default Java	Ubuntu-17.0.18+8-Ubuntu-124.04.1
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.10+7-Ubuntu-124.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.18+8-Ubuntu-124.04.1
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8448/3/testReport/
Max. process+thread count	4093 (vs. ulimit of 10000)
modules	C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8448/3/console
versions	git=2.43.0 maven=3.9.11 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

HDFS-17909. [ARR] AsyncRouterHandlerExecutors should use bounded queue

8c64217

github-actions Bot added HDFS trunk RBF labels Apr 21, 2026

ZanderXu requested a review from hfutatzhanghb April 21, 2026 07:43

kokonguyen191 added 2 commits April 22, 2026 16:42

HDFS-17909. Redo the unit test

a8e96da

HDFS-17909. Wrap long lines

2c3f471

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-17909. [ARR] AsyncRouterHandlerExecutors should use bounded queue#8448

HDFS-17909. [ARR] AsyncRouterHandlerExecutors should use bounded queue#8448
kokonguyen191 wants to merge 3 commits intoapache:trunkfrom
kokonguyen191:async-router-server-f

kokonguyen191 commented Apr 21, 2026 •

edited

Loading

Uh oh!

hfutatzhanghb commented Apr 21, 2026

Uh oh!

kokonguyen191 commented Apr 22, 2026

Uh oh!

hadoop-yetus commented Apr 22, 2026

Uh oh!

hadoop-yetus commented Apr 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

kokonguyen191 commented Apr 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

How was this patch tested?

Uh oh!

hfutatzhanghb commented Apr 21, 2026

Uh oh!

kokonguyen191 commented Apr 22, 2026

Uh oh!

hadoop-yetus commented Apr 22, 2026

Uh oh!

hadoop-yetus commented Apr 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kokonguyen191 commented Apr 21, 2026 •

edited

Loading