Skip to content

Discourage rebalance, warn against stopping it #1298

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

Discourage rebalance, warn against stopping it #1298

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a9bfde7
draft: discourage rebalance, warn against stopping it
feorlen Aug 14, 2024
40606ce
additional discussion of mismatched pool sizes
feorlen Aug 22, 2024
File filter

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions source/operations/concepts.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,12 +149,14 @@ For more information on write preference calculation logic, see :ref:`Writing Fi
Rebalancing data across all pools after an expansion is an expensive operation that requires scanning the entire deployment and moving objects between pools.
This may take a long time to complete depending on the amount of data to move.

Starting with MinIO Client version RELEASE.2022-11-07T23-47-39Z, you can manually initiate a rebalancing operation across all server pools using :mc:`mc admin rebalance`.
MinIO does not recommend manual rebalancing.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we remove the recommendation text?
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kannappanr you mean we should say it's ok to manually rebalance?
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we instead use a cautionary statement of using only with consultation with MinIO Engineering?
The ask we had for this PR was specifically to discourage use of this feature.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct we have made a release already, and the fixes are also in the EOS binaries so to some extent we have addressed this already.

We should perhaps talk about a more broader tone that rebalance is not a real requirement if you size your pools properly.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically discourage budget setups
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't expand in this manner

first pool

  • 100 nodes which is now 90% used

you botched buying hardware now you just expand by 20 nodes

  • 20 nodes is second pool

This 20 nodes will take all the I/O hit causing significant slowness, the sizing must be appropriate to the load that 100 node was handling. if 20 can handle and its a new hardware no problem but if its not then it is going to cause outage etc.

A cautionary guidance on why rebalance don't solve the problem of high utilization the second pool. IT may look like that but it won't solve the problem.
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We mention the "2 years" guidance in the hardware checklist, although there isn't a specific and obvious section along the lines of "How big should I make my pool?" (Noticed that AJ's blog from January recommends minimum 3 years of capacity.)

I think we can reinforce this in the Storage section of the hardware checklist, maybe mention it elsewhere too. Like the concepts page. To make the point that tacking on a bit of new capacity here and there doesn't go well and is not a reliable plan.
If required, you can manually initiate a rebalancing operation across all server pools using :mc:`mc admin rebalance`.
MinIO recommends `SUBNET <https://min.io/pricing?jmp=docs>`__ users `log in <https://subnet.min.io/>`__ and create a new issue to discuss appropriate rebalancing strategies for deployments.

Rebalancing does not block ongoing operations and runs in parallel to all other I/O.
This can result in reduced performance of regular operations.
Consider scheduling rebalancing operations during non-peak periods to avoid impacting production workloads.
You can start and stop rebalancing at any time


How do I upload objects to MinIO?
---------------------------------
Expand Down
5 changes: 4 additions & 1 deletion source/operations/install-deploy-manage/expand-minio-deployment.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,10 @@ MinIO calculates the probability of a write operation to each of the pools as:
In addition to the free space calculation, if a write option (with parity) would bring a drive
usage above 99% or a known free inode count below 1000, MinIO does not write to the pool.

If desired, you can manually initiate a rebalance procedure with :mc:`mc admin rebalance`.
MinIO does not recommend manual rebalancing.
If required, you can manually initiate a rebalance procedure with :mc:`mc admin rebalance`.
MinIO recommends `SUBNET <https://min.io/pricing?jmp=docs>`__ users `log in <https://subnet.min.io/>`__ and create a new issue to discuss appropriate rebalancing strategies for deployments.

For more about how rebalancing works, see :ref:`managing objects across a deployment <minio-rebalance>`.

Likewise, MinIO does not write to pools in a decommissioning process.
Expand Down
12 changes: 8 additions & 4 deletions source/reference/minio-mc-admin/mc-admin-rebalance.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Description

.. start-mc-admin-rebalance-desc

The :mc-cmd:`mc admin rebalance` command allows starts, monitors, or stops a rebalancing operation on a MinIO deployment.
The :mc-cmd:`mc admin rebalance` command allows starts or monitors a rebalancing operation on a MinIO deployment.
Rebalancing redistributes objects across all pools in the deployment.

.. end-mc-admin-rebalance-desc
Expand All @@ -35,7 +35,7 @@ This is an expensive and time consuming operation.
Consider only running a rebalance procedure during light or no use of the deployment.
If write operations do occur during a rebalance operation, they process in parallel and write to a pool not actively in rebalancing.

You can stop a rebalance and start it again later as needed.
`MinIO SUBNET <https://min.io/pricing?jmp=docs>`__ users can `log in <https://subnet.min.io/>`__ and create a new issue to discuss appropriate rebalancing strategies for deployments.

Follow the progress of an ongoing rebalance operation using the following command:

Expand Down Expand Up @@ -67,8 +67,6 @@ The :mc-cmd:`mc admin rebalance` command has the following subcommands:
* - :mc-cmd:`mc admin rebalance status`
- Outputs the current status of an in-progress rebalance operation.

* - :mc-cmd:`mc admin rebalance stop`
- Stops an in-progress rebalance operation.

Syntax
------
Expand Down Expand Up @@ -135,6 +133,12 @@ Syntax

Ends an in-progress rebalance job on the specified deployment.

.. admonition:: Stop may cause data loss
:class: warning

At this time, MinIO does not recommend stopping an in-progress rebalance job.
Interrupting rebalance may result in data loss.

.. tab-set::

.. tab-item:: EXAMPLES
Expand Down