A Kafka cluster consists of one or more connected brokers, and receives messages published by producers and delivers them to consumers. Additionally, Kafka stores received messages on disk for a certain period of time to prepare for failures and allows consumers to read them again as needed.
Kafka brokers use disk devices, and when operating a Kafka cluster, disk usage may inevitably vary. Unbalanced disk usage means that the amount of messages sent by the producer is different in the first place, and the network usage is different.
Therefore, it can be concluded that moving some topic partitions to balance disk usage between brokers can ultimately balance the usage of network and other resources.
meshIQ Kafka Console provides Smart Rebalance, a function to balance disk usage based on collected information. To use this function, select the Smart Rebalance menu on the left side of the screen.
The Recommendation step shows the following fields:
- Max recursive limit. Maximum number of calculation iterations.
- New recommendation. Recalculate partition moves.
- Total Data Movement. Amount of data moving as a result of calculation.
Create a new rebalancing task
You can create a new smart relocation task by selecting Configure rebalance on the screen.
Setting up rebalancing tasks
Depending on the results calculated in the Recommendation step, when you advance to the Configuration step, you can display a list of the topic partitions to be moved and set the throttling value to be applied during rebalancing.
- TOPIC. Topic name to move.
- PARTITION NO. Partition number to move.
- SIZE. partition size.
- BROKER MOVEMENT. Planned move between brokers (from >> to).
An estimate of the amount of time required for rebalancing is displayed as Estimated Time:
Refer to the section below for throttling recommendations. When ready to begin rebalancing, click the Smart rebalance button:
Setting up throttling
Set up throttling to limit the network resources that are used for rebalancing operations.
Rebalancing is a process of moving topic partitions, so it uses network resources. If there is a large amount of data being moved, failure to limit bandwidth usage may have a negative impact on the original service. Therefore, it is recommended that you set up throttling. You can calculate the appropriate throttling value using the Would You Like a Throttling Recommendation? link. You must set a value between the Lower Bound and Upper Bound values.