[feat] PIP-468: Add ScalableTopicController and broker infrastructure#25559
Open
merlimat wants to merge 3 commits intoapache:masterfrom
Open
[feat] PIP-468: Add ScalableTopicController and broker infrastructure#25559merlimat wants to merge 3 commits intoapache:masterfrom
merlimat wants to merge 3 commits intoapache:masterfrom
Conversation
Implement the broker-side scalable topic engine: ScalableTopicController handles per-topic leader election and consumer session management, SegmentLayout manages the hash-range DAG, SubscriptionCoordinator tracks consumer assignments across segments. Add ScalableTopicService for per-topic controller lifecycle, ScalableTopicResources for metadata persistence, and BrokerService integration. Includes unit tests for SegmentLayout and SubscriptionCoordinator. Split/merge operations, the admin REST layer, the client protocol wiring, and the end-to-end controller/service tests follow in subsequent PRs of the PIP-468 series.
merlimat
added
type/feature
merlimat
changed the title
- Remove unused imports from BrokerService, ScalableTopicController, and ScalableTopicService. - Move io.github.merlimat.slog.Logger import to the correct position in SubscriptionCoordinator. - Add package-info.java for the service.scalable package. - Drop unused assertNull import and extra import-group blank lines in the test classes.
This was referenced Apr 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First PR in the PIP-468 series introducing scalable topics. This PR lands
the core broker-side infrastructure:
ScalableTopicController— per-topic coordinator with leader election,consumer session management, and layout tracking.
SegmentLayout— pure data/math for the hash-range segment DAG.SubscriptionCoordinator— in-memory consumer-assignment trackingper subscription.
ScalableTopicService— per-broker service managing controllerlifecycle (get/release controllers, lifecycle hooks on
BrokerService).ScalableTopicResources,ScalableTopicMetadata,ConsumerRegistration,SubscriptionMetadata,SubscriptionType.SegmentLayoutandSubscriptionCoordinator.Tests
Only the self-contained pure-data components have unit tests in this PR
(
SegmentLayoutTest,SubscriptionCoordinatorTest).End-to-end tests of
ScalableTopicControllerandScalableTopicServiceare deferred to subsequent PRs because their setup depends on code that
lands later:
PulsarAdmin.scalableTopics(), a type added by theadmin-API PR.
splitSegment/mergeSegmentslive with the admin layer (they invokeadmin.scalableTopics().createSegmentAsync/terminateSegmentAsync)and their tests move with them.
Test plan
:pulsar-broker:compileJava/:pulsar-broker:compileTestJavasucceed on this commit in isolation.
./gradlew :pulsar-broker:test --tests "org.apache.pulsar.broker.service.scalable.*"passes (
SegmentLayoutTest,SubscriptionCoordinatorTest).