In this blog, we continue from where we left off in the previous post, InnoDB Cluster Setup: Building a 3-Node High Availability Architecture, where we demonstrated how to set up a MySQL InnoDB Cluster with three nodes to achieve high availability.
Here, we walk through the step-by-step process of performing a rolling upgrade of that cluster to MySQL version 8.4. This method ensures high availability and minimal disruption by upgrading one node at a time, starting with the secondary nodes and ending with the primary. Along the way, we address key considerations such as managing group replication behavior, monitoring cluster status, handling failovers, and upgrading cluster metadata.
Step 1: Verify cluster health and identify node roles before upgrade
Before initiating the upgrade process, it is essential to verify the current status of the InnoDB Cluster to ensure everything is functioning correctly. This involves checking the overall health of the cluster and identifying the role of each node specifically, which node is the primary and which ones are the secondary. As a best practice, upgrades should begin with the secondary nodes to minimize the impact on the cluster’s availability. The cluster.status() command provides a clear overview of node roles, replication status, and version details, helping you plan a smooth and controlled upgrade path.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 | MySQL Py > shell.connect('innoclusteradmin@ArunClusterND1:3306'); Creating a session to 'innoclusteradmin@ArunClusterND1:3306' Fetching schema names for auto-completion... Press ^C to stop. Your MySQL connection id is 3020 Server version: 8.0.42-0ubuntu0.24.04.1 (Ubuntu) No default schema selected; type use <schema> to set one. <ClassicSession:innoclusteradmin@ArunClusterND1:3306> MySQL ArunClusterND1:3306 ssl Py > MySQL ArunClusterND1:3306 ssl Py > cluster = dba.get_cluster() MySQL ArunClusterND1:3306 ssl Py > cluster.status() { "clusterName": "innodbtestcluster", "defaultReplicaSet": { "name": "default", "primary": "ArunClusterND1:3306", "ssl": "REQUIRED", "status": "OK", "statusText": "Cluster is ONLINE and can tolerate up to ONE failure.", "topology": { "ArunClusterND1:3306": { "address": "ArunClusterND1:3306", "memberRole": "PRIMARY", "mode": "R/W", "readReplicas": {}, "replicationLag": "applier_queue_applied", "role": "HA", "status": "ONLINE", "version": "8.0.42" }, "ArunClusterND2:3306": { "address": "ArunClusterND2:3306", "memberRole": "SECONDARY", "mode": "R/O", "readReplicas": {}, "replicationLag": "applier_queue_applied", "role": "HA", "status": "ONLINE", "version": "8.0.42" }, "ArunClusterND3:3306": { "address": "ArunClusterND3:3306", "memberRole": "SECONDARY", "mode": "R/O", "readReplicas": {}, "replicationLag": "applier_queue_applied", "role": "HA", "status": "ONLINE", "version": "8.0.42" } }, "topologyMode": "Single-Primary" }, "groupInformationSourceMember": "ArunClusterND1:3306" } MySQL ArunClusterND1:3306 ssl Py > |
Step 2: Ensure all users are using caching_sha2_password for secure and seamless upgrades
Upgrading all users in an InnoDB Cluster to use the caching_sha2_password plugin is recommended when moving from MySQL 8.0 to 8.4 because it is the default authentication method in newer versions, offering improved security, performance, and compatibility. It ensures seamless operation of MySQL Router, Group Replication, and other cluster components while also avoiding potential issues during upgrades. This plugin uses stronger SHA-256-based hashing and supports faster logins with caching, making it a more secure and efficient choice over the legacy mysql_native_password.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | MySQL ArunClusterND1:3306 ssl SQL > select Host,User,plugin from mysql.user; +-----------+---------------------------------+-----------------------+ | Host | User | plugin | +-----------+---------------------------------+-----------------------+ | % | appuser | caching_sha2_password | | % | innoclusteradmin | caching_sha2_password | | % | mysql_innodb_cluster_1814828393 | caching_sha2_password | | % | mysql_innodb_cluster_2728175459 | caching_sha2_password | | % | mysql_innodb_cluster_2850393796 | caching_sha2_password | | % | mysql_router1_bo7w2gz | caching_sha2_password | | % | mysql_router1_oc5ka4y | caching_sha2_password | | localhost | debian-sys-maint | caching_sha2_password | | localhost | mysql.infoschema | caching_sha2_password | | localhost | mysql.session | caching_sha2_password | | localhost | mysql.sys | caching_sha2_password | | localhost | root | caching_sha2_password | +-----------+---------------------------------+-----------------------+ 12 rows in set (0.0004 sec) |
Step 3: Prepare the secondary node (ArunClusterND3) for upgrade by configuring the MySQL APT repository
To begin the upgrade process on a secondary node, you must first configure the official MySQL APT repository. This involves downloading and installing the MySQL APT configuration package, which allows your system to access and install the latest MySQL 8.4 packages directly from Oracle’s trusted repository. According to the official MySQL documentation, this step ensures that all required components, such as MySQL Server, Shell, and Router, are properly registered and updated during the upgrade process.
The server is running Ubuntu 24.04.2 LTS, and the MySQL APT repository configuration has been successfully installed. This indicates that the system is ready to proceed with the upgrade to MySQL 8.4 packages.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | root@ArunClusterND3:/home/ubuntu# dpkg -s mysql-apt-config Package: mysql-apt-config Status: install ok installed Priority: optional Section: database Installed-Size: 35 Maintainer: MySQL Release Engineering <mysql-build@oss.oracle.com> Architecture: all Version: 0.8.34-1 Pre-Depends: debconf (>= 0.2.17), dpkg (>= 1.15), lsb-release, wget, bash (>= 4.0), gnupg Description: Auto configuration for MySQL APT Repo. MySQL is a fast, stable and true multi-user, multi-threaded SQL database server. SQL (Structured Query Language) is the most popular database query language in the world. The main goals of MySQL are speed, robustness and ease of use. Homepage: http://dev.mysql.com/ root@ArunClusterND3:/home/ubuntu# |
Step 4: Upgrading MySQL Shell and performing server upgrade checks on secondary node ArunClusterND3
As the first step in the upgrade process, we begin by upgrading the MySQL Shell on the secondary node, ArunClusterND3. This ensures we have the latest tools compatible with MySQL 8.4. After upgrading MySQL Shell, we run the util.check_for_server_upgrade() utility to verify the current server’s readiness for upgrade. This utility scans the server for compatibility issues, deprecated system variables, changes in default values, and privilege warnings that need attention before proceeding. Addressing these findings helps avoid potential problems during the upgrade and ensures a smoother transition from MySQL 8.0 to 8.4.
- Upgrading MySQL Shell to the 8.4 version
1 2 3 4 5 6 7 | root@ArunClusterND3:/home/ubuntu# sudo apt install mysql-shell Reading package lists... Done Building dependency tree... Done root@ArunClusterND3:/home/ubuntu# mysqlsh --version mysqlsh Ver 8.4.5 for Linux on x86_64 - for MySQL 8.4.5 (MySQL Community Server (GPL)) root@ArunClusterND3:/home/ubuntu# |
- Connecting to cluster node
Connect to the cluster node using the command shell.connect() to initiate a connection to the MySQL server as the innoclusteradmin user.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | root@ArunClusterND3:/home/ubuntu# mysqlsh MySQL Shell 8.4.5 Copyright (c) 2016, 2025, Oracle and/or its affiliates. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help' or '?' for help; 'quit' to exit. MySQL SQL > MySQL Py > shell.connect('innoclusteradmin@ArunClusterND3:3306'); Creating a session to 'innoclusteradmin@ArunClusterND3:3306' Please provide the password for 'innoclusteradmin@ArunClusterND3:3306': **** Save password for 'innoclusteradmin@ArunClusterND3:3306'? [Y]es/[N]o/Ne[v]er (default No): y Fetching schema names for auto-completion... Press ^C to stop. Your MySQL connection id is 41 Server version: 8.0.42-0ubuntu0.24.04.1 (Ubuntu) No default schema selected; type use <schema> to set one. <ClassicSession:innoclusteradmin@ArunClusterND3:3306> MySQL ArunClusterND3:3306 ssl Py > |
3. Running upgrade compatibility check
Execute util.check_for_server_upgrade() in MySQL Shell to analyze the current MySQL server (version 8.0.42) for potential compatibility issues before upgrading to MySQL 8.4.5.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 | MySQL ArunClusterND3:3306 ssl Py > util.check_for_server_upgrade() The MySQL server at ArunClusterND3:3306, version 8.0.42-0ubuntu0.24.04.1 - (Ubuntu), will now be checked for compatibility issues for upgrade to MySQL 8.4.5. To check for a different target server version, use the targetVersion option. 1) Removed system variables (removedSysVars) Error: Following system variables that were detected as being used will be removed. Please update your system to not rely on them before the upgrade. binlog_transaction_dependency_tracking - Error: The system variable 'binlog_transaction_dependency_tracking' is set to WRITESET (PERSISTED) and will be removed. More information: https://dev.mysql.com/doc/refman/8.0/en/added-deprecated-removed.html#optvars-removed .. 12) Checks for partitions by key using columns with prefix key indexes (partitionsWithPrefixKeys) No issues found Errors: 1 Warnings: 18 Notices: 3 ERROR: 1 errors were found. Please correct these issues before upgrading to avoid compatibility issues. MySQL ArunClusterND3:3306 ssl Py > |
Fixing all errors reported by the util.check_for_server_upgrade() utility is crucial because any unresolved issues can cause the upgrade to fail or lead to instability in the cluster. Addressing these errors beforehand ensures a successful and smooth upgrade process.
Step 5: Resolving deprecated variable error
As reported by the upgrade check tool, the MySQL system variable binlog_transaction_dependency_tracking is currently set to WRITESET and persisted; however, it is deprecated and will be removed in MySQL 8.4. If this variable remains set, the MySQL 8.4 server will fail to start. To resolve this issue, verify the current value and persistence of the variable, then reset it. Clearing this persisted setting ensures compatibility with MySQL 8.4 and allows the upgrade process to continue without errors.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | MySQL ArunClusterND3:3306 ssl Py > sql Switching to SQL mode... Commands end with ; Warning (code 1287): '@@binlog_transaction_dependency_tracking' is deprecated and will be removed in a future release. MySQL ArunClusterND3:3306 ssl SQL > SELECT * FROM performance_schema.persisted_variables WHERE variable_name = 'binlog_transaction_dependency_tracking'; +----------------------------------------+----------------+ | VARIABLE_NAME | VARIABLE_VALUE | +----------------------------------------+----------------+ | binlog_transaction_dependency_tracking | WRITESET | +----------------------------------------+----------------+ 1 row in set (0.0008 sec) MySQL ArunClusterND3:3306 ssl SQL > RESET PERSIST binlog_transaction_dependency_tracking; Query OK, 0 rows affected (0.0036 sec) MySQL ArunClusterND3:3306 ssl SQL > SELECT * FROM performance_schema.persisted_variables WHERE variable_name = 'binlog_transaction_dependency_tracking'; Empty set (0.0004 sec) MySQL ArunClusterND3:3306 ssl SQL > select @@binlog_transaction_dependency_tracking; +------------------------------------------+ | @@binlog_transaction_dependency_tracking | +------------------------------------------+ | WRITESET | +------------------------------------------+ 1 row in set, 1 warning (0.0003 sec) Warning (code 1287): '@@binlog_transaction_dependency_tracking' is deprecated and will be removed in a future release. MySQL ArunClusterND3:3306 ssl SQL > |
Step 6: Disable automatic group replication startup before upgrading
Before upgrading to MySQL 8.4, it’s a good precaution to set group_replication_start_on_boot to OFF. This prevents automatic startup of Group Replication after a server restart, allowing you to manually verify and control the cluster’s behavior post-upgrade.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 | MySQL ArunClusterND3:3306 ssl SQL > SELECT VARIABLE_NAME, VARIABLE_VALUE FROM performance_schema.global_variables WHERE VARIABLE_NAME = 'group_replication_start_on_boot'; +---------------------------------+----------------+ | VARIABLE_NAME | VARIABLE_VALUE | +---------------------------------+----------------+ | group_replication_start_on_boot | ON | +---------------------------------+----------------+ 1 row in set (0.0019 sec) MySQL ArunClusterND3:3306 ssl SQL > MySQL ArunClusterND3:3306 ssl SQL > SET PERSIST group_replication_start_on_boot = OFF; Query OK, 0 rows affected (0.0039 sec) MySQL ArunClusterND3:3306 ssl SQL > SELECT VARIABLE_NAME, VARIABLE_VALUE FROM performance_schema.global_variables WHERE VARIABLE_NAME = 'group_replication_start_on_boot'; +---------------------------------+----------------+ | VARIABLE_NAME | VARIABLE_VALUE | +---------------------------------+----------------+ | group_replication_start_on_boot | OFF | +---------------------------------+----------------+ 1 row in set (0.0013 sec) MySQL ArunClusterND3:3306 ssl SQL > |
Step 7: Upgrading to MySQL Server 8.4
- Checking available MySQL Server versions
The apt-cache madison mysql-server command lists all available versions of the MySQL server package from configured repositories, showing the source and version numbers for upgrade planning.
1 2 3 4 5 6 7 | root@ArunClusterND3:/home/ubuntu# apt-cache madison mysql-server mysql-server | 8.4.5-1ubuntu24.04 | http://repo.mysql.com/apt/ubuntu noble/mysql-8.4-lts amd64 Packages mysql-server | 8.0.42-0ubuntu0.24.04.1 | http://us-east-1.ec2.archive.ubuntu.com/ubuntu noble-updates/main amd64 Packages mysql-server | 8.0.42-0ubuntu0.24.04.1 | http://security.ubuntu.com/ubuntu noble-security/main amd64 Packages mysql-server | 8.0.36-2ubuntu3 | http://us-east-1.ec2.archive.ubuntu.com/ubuntu noble/main amd64 Packages mysql-community | 8.4.5-1ubuntu24.04 | http://repo.mysql.com/apt/ubuntu noble/mysql-8.4-lts Sources root@ArunClusterND3:/home/ubuntu# |
- Stopping the MySQL service
systemctl stop mysql is used to safely stop the MySQL server service before performing the upgrade task.
1 2 | root@ArunClusterND3:/home/ubuntu# systemctl stop mysql root@ArunClusterND3:/home/ubuntu# |
- Checking MySQL InnoDB cluster status
The cluster.status() command in MySQL Shell shows the current health and topology of the InnoDB cluster, including the roles of nodes, replication status, and any connection issues with cluster members. Since we have stopped the ArunClusterND3 node for an upgrade, its status is reflected as “status”: “(MISSING)”.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 | MySQL ArunClusterND1:3306 ssl Py > cluster.status() { "clusterName": "innodbtestcluster", "defaultReplicaSet": { "name": "default", "primary": "ArunClusterND1:3306", "ssl": "REQUIRED", "status": "OK_NO_TOLERANCE_PARTIAL", "statusText": "Cluster is NOT tolerant to any failures. 1 member is not active.", "topology": { "ArunClusterND1:3306": { "address": "ArunClusterND1:3306", "memberRole": "PRIMARY", "mode": "R/W", "readReplicas": {}, "replicationLag": "applier_queue_applied", "role": "HA", "status": "ONLINE", "version": "8.0.42" }, "ArunClusterND2:3306": { "address": "ArunClusterND2:3306", "memberRole": "SECONDARY", "mode": "R/O", "readReplicas": {}, "replicationLag": "applier_queue_applied", "role": "HA", "status": "ONLINE", "version": "8.0.42" }, "ArunClusterND3:3306": { "address": "ArunClusterND3:3306", "memberRole": "SECONDARY", "mode": "n/a", "readReplicas": {}, "role": "HA", "shellConnectError": "MySQL Error 2003: Could not open connection to 'ArunClusterND3:3306': Can't connect to MySQL server on 'ArunClusterND3:3306' (111)", "status": "(MISSING)" } }, "topologyMode": "Single-Primary" }, "groupInformationSourceMember": "ArunClusterND1:3306" } MySQL ArunClusterND1:3306 ssl Py > |
- Upgrading MySQL Server
To upgrade the MySQL Server, execute the following command:
1 2 3 4 | root@ArunClusterND3:/home/ubuntu# apt-get install mysql-server Reading package lists... Done Building dependency tree... Done Reading state information... Done |
This command refreshes the package list and upgrades the installed MySQL server package to the latest version available from the configured MySQL APT Repository, ensuring your node runs the most recent MySQL release.
Here, the node ArunClusterND3 has successfully upgraded to version 8.4.5.
- Verify the cluster status after upgrade with group replication disabled
The output of cluster.status() shows ArunClusterND3 in the cluster with “version”: “8.4.5”, but its “memberState” is “OFFLINE” and “status” is “(MISSING)”. This is expected because we previously set group_replication_start_on_boot to OFF for safety, preventing the automatic startup of group replication and allowing manual control after the upgrade. We can confirm that the node ArunClusterND3 has been successfully upgraded to version 8.4.5.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 | MySQL ArunClusterND1:3306 ssl Py > cluster.status() { "clusterName": "innodbtestcluster", "defaultReplicaSet": { "name": "default", "primary": "ArunClusterND1:3306", "ssl": "REQUIRED", "status": "OK_NO_TOLERANCE_PARTIAL", "statusText": "Cluster is NOT tolerant to any failures. 1 member is not active.", "topology": { "ArunClusterND1:3306": { "address": "ArunClusterND1:3306", "memberRole": "PRIMARY", "mode": "R/W", "readReplicas": {}, "replicationLag": "applier_queue_applied", "role": "HA", "status": "ONLINE", "version": "8.0.42" }, "ArunClusterND2:3306": { "address": "ArunClusterND2:3306", "memberRole": "SECONDARY", "mode": "R/O", "readReplicas": {}, "replicationLag": "applier_queue_applied", "role": "HA", "status": "ONLINE", "version": "8.0.42" }, "ArunClusterND3:3306": { "address": "ArunClusterND3:3306", "instanceErrors": [ "NOTE: group_replication is stopped." ], "memberRole": "SECONDARY", "memberState": "OFFLINE", "mode": "n/a", "readReplicas": {}, "role": "HA", "status": "(MISSING)", "version": "8.4.5" } }, "topologyMode": "Single-Primary" }, "groupInformationSourceMember": "ArunClusterND1:3306" } MySQL ArunClusterND1:3306 ssl Py > |
- Re-enabling group replication startup
After verifying the upgrade, we re-enable automatic Group Replication startup to ensure the node automatically rejoins the cluster after reboots.
1 2 3 4 5 6 7 8 9 10 11 12 | MySQL ArunClusterND3:3306 ssl SQL > SET PERSIST group_replication_start_on_boot = ON; Query OK, 0 rows affected (0.0050 sec) MySQL ArunClusterND3:3306 ssl SQL > SELECT VARIABLE_NAME, VARIABLE_VALUE FROM performance_schema.global_variables WHERE VARIABLE_NAME = 'group_replication_start_on_boot'; +---------------------------------+----------------+ | VARIABLE_NAME | VARIABLE_VALUE | +---------------------------------+----------------+ | group_replication_start_on_boot | ON | +---------------------------------+----------------+ 1 row in set (0.0015 sec) MySQL ArunClusterND3:3306 ssl SQL > |
7. Restart the MySQL service
Restart the MySQL service, applying the persisted configuration change and allowing the upgraded node to rejoin the cluster.
1 2 | root@ArunClusterND3:/etc/mysql/mysql.conf.d# service mysql restart root@ArunClusterND3:/etc/mysql/mysql.conf.d# |
8. Verifying full cluster health post-upgrade
Running cluster.status() now shows that ArunClusterND3 is “ONLINE” and fully integrated into the cluster with version 8.4.5. The cluster status is “OK”, confirming it is healthy and tolerant to one member failure — a successful upgrade and reintegration.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 | MySQL ArunClusterND1:3306 ssl Py > cluster.status() { "clusterName": "innodbtestcluster", "defaultReplicaSet": { "name": "default", "primary": "ArunClusterND1:3306", "ssl": "REQUIRED", "status": "OK", "statusText": "Cluster is ONLINE and can tolerate up to ONE failure.", "topology": { "ArunClusterND1:3306": { "address": "ArunClusterND1:3306", "memberRole": "PRIMARY", "mode": "R/W", "readReplicas": {}, "replicationLag": "applier_queue_applied", "role": "HA", "status": "ONLINE", "version": "8.0.42" }, "ArunClusterND2:3306": { "address": "ArunClusterND2:3306", "memberRole": "SECONDARY", "mode": "R/O", "readReplicas": {}, "replicationLag": "applier_queue_applied", "role": "HA", "status": "ONLINE", "version": "8.0.42" }, "ArunClusterND3:3306": { "address": "ArunClusterND3:3306", "memberRole": "SECONDARY", "mode": "R/O", "readReplicas": {}, "replicationLag": "applier_queue_applied", "role": "HA", "status": "ONLINE", "version": "8.4.5" } }, "topologyMode": "Single-Primary" }, "groupInformationSourceMember": "ArunClusterND1:3306" } MySQL ArunClusterND1:3306 ssl Py > |
To continue the rolling upgrade process, follow the same sequence of steps for the second secondary node in the cluster, ArunClusterND2. This includes stopping the MySQL service, upgrading the MySQL server to version 8.4.5, verifying and clearing any deprecated persisted variables (if any), ensuring group_replication_start_on_boot is set to OFFbefore upgrade, and then setting it back to ON after a successful restart. Once the node is restarted, it should automatically rejoin the cluster. Finally, use cluster.status() to confirm that ArunClusterND2 is back ONLINE, now running version 8.4.5, and the cluster remains healthy and tolerant.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 | root@ArunClusterND2:/etc/mysql/mysql.conf.d# service mysql restart root@ArunClusterND2:/etc/mysql/mysql.conf.d# MySQL ArunClusterND2:3306 ssl Py > cluster.status() { "clusterName": "innodbtestcluster", "defaultReplicaSet": { "name": "default", "primary": "ArunClusterND1:3306", "ssl": "REQUIRED", "status": "OK", "statusText": "Cluster is ONLINE and can tolerate up to ONE failure.", "topology": { "ArunClusterND1:3306": { "address": "ArunClusterND1:3306", "memberRole": "PRIMARY", "mode": "R/W", "readReplicas": {}, "replicationLag": "applier_queue_applied", "role": "HA", "status": "ONLINE", "version": "8.0.42" }, "ArunClusterND2:3306": { "address": "ArunClusterND2:3306", "memberRole": "SECONDARY", "mode": "R/O", "readReplicas": {}, "replicationLag": "applier_queue_applied", "role": "HA", "status": "ONLINE", "version": "8.4.5" }, "ArunClusterND3:3306": { "address": "ArunClusterND3:3306", "memberRole": "SECONDARY", "mode": "R/O", "readReplicas": {}, "replicationLag": "applier_queue_applied", "role": "HA", "status": "ONLINE", "version": "8.4.5" } }, "topologyMode": "Single-Primary" }, "groupInformationSourceMember": "ArunClusterND1:3306" } MySQL ArunClusterND2:3306 ssl Py > |
Step 8: Upgrading the primary node (ArunClusterND1) to MySQL 8.4.5
Now that both secondary nodes have been successfully upgraded to MySQL 8.4.5 and rejoined the cluster, it’s time to upgrade the primary node, ArunClusterND1. This step follows the same upgrade procedure: stop the MySQL service, perform the package upgrade, check and clear any deprecated persisted variables if necessary, and ensure that group_replication_start_on_boot is set to OFF before restarting. After the upgrade, set the variable back to ON and restart the service. Once the node comes back online, verify with cluster.status() to ensure that ArunClusterND1 is now running version 8.4.5 and that the cluster is fully ONLINE and tolerant to failures.
Step 9: Observing automatic primary re-election during upgrade
When the primary node ArunClusterND1 was gracefully shut down for upgrade, the MySQL InnoDB Cluster automatically detected the unavailability and marked its status as “(MISSING)”. As part of the built-in high availability mechanism, the cluster elected a new primary node, and ArunClusterND3, which was already upgraded to MySQL 8.4.5, was promoted to “memberRole”: “PRIMARY”. This seamless promotion ensures continued write availability and cluster functionality during the upgrade process. It demonstrates the cluster’s fault-tolerant design and the importance of upgrading secondary nodes first to support a smooth primary failover.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 | root@ArunClusterND1:/home/ubuntu# service mysql stop root@ArunClusterND1:/home/ubuntu# MySQL ArunClusterND2:3306 ssl Py > cluster = dba.get_cluster() NOTE: The installed metadata version 2.1.0 is lower than the version required by Shell which is version 2.2.0. It is recommended to upgrade the metadata. See ? dba.upgrade_metadata for additional details. MySQL ArunClusterND2:3306 ssl Py > cluster.status() { "clusterName": "innodbtestcluster", "defaultReplicaSet": { "name": "default", "primary": "ArunClusterND3:3306", "ssl": "REQUIRED", "status": "OK_NO_TOLERANCE", "statusText": "Cluster is NOT tolerant to any failures.", "topology": { "ArunClusterND1:3306": { "address": "ArunClusterND1:3306", "memberRole": "SECONDARY", "mode": "n/a", "readReplicas": {}, "role": "HA", "shellConnectError": "MySQL Error 2003: Could not open connection to 'ArunClusterND1:3306': Can't connect to MySQL server on 'ArunClusterND1:3306' (111)", "status": "(MISSING)" }, "ArunClusterND2:3306": { "address": "ArunClusterND2:3306", "memberRole": "SECONDARY", "mode": "R/O", "readReplicas": {}, "replicationLag": "applier_queue_applied", "role": "HA", "status": "ONLINE", "version": "8.4.5" }, "ArunClusterND3:3306": { "address": "ArunClusterND3:3306", "memberRole": "PRIMARY", "mode": "R/W", "readReplicas": {}, "replicationLag": "applier_queue_applied", "role": "HA", "status": "ONLINE", "version": "8.4.5" } }, "topologyMode": "Single-Primary" }, "groupInformationSourceMember": "ArunClusterND3:3306" } MySQL ArunClusterND2:3306 ssl Py > |
Step 10: Final node upgrade and cluster validation
After successfully upgrading the ArunClusterND1 node to MySQL 8.4.5, we have now completed the upgrade process for all nodes in the InnoDB Cluster. A quick check using MySQL Shell confirms that each node ArunClusterND1, ArunClusterND2, and ArunClusterND3 is ONLINE, with ArunClusterND3 currently acting as the PRIMARY. The cluster status shows “OK” with fault tolerance for up to one failure, indicating a healthy and fully operational cluster running on the latest version. This concludes the rolling upgrade to MySQL 8.4.5 with no downtime or service disruption.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 | root@ArunClusterND1:/etc/mysql/mysql.conf.d# mysqlsh MySQL Shell 8.4.5 Copyright (c) 2016, 2025, Oracle and/or its affiliates. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help' or '?' for help; 'quit' to exit. MySQL SQL > py Switching to Python mode... MySQL Py > shell.connect('innoclusteradmin@ArunClusterND1:3306'); Creating a session to 'innoclusteradmin@ArunClusterND1:3306' Fetching schema names for auto-completion... Press ^C to stop. Your MySQL connection id is 37 Server version: 8.4.5 MySQL Community Server - GPL No default schema selected; type use <schema> to set one. <ClassicSession:innoclusteradmin@ArunClusterND1:3306> MySQL ArunClusterND1:3306 ssl Py > cluster = dba.get_cluster() NOTE: The installed metadata version 2.1.0 is lower than the version required by Shell which is version 2.2.0. It is recommended to upgrade the metadata. See ? dba.upgrade_metadata for additional details. MySQL ArunClusterND1:3306 ssl Py > cluster.status() { "clusterName": "innodbtestcluster", "defaultReplicaSet": { "name": "default", "primary": "ArunClusterND3:3306", "ssl": "REQUIRED", "status": "OK", "statusText": "Cluster is ONLINE and can tolerate up to ONE failure.", "topology": { "ArunClusterND1:3306": { "address": "ArunClusterND1:3306", "memberRole": "SECONDARY", "mode": "R/O", "readReplicas": {}, "replicationLag": "applier_queue_applied", "role": "HA", "status": "ONLINE", "version": "8.4.5" }, "ArunClusterND2:3306": { "address": "ArunClusterND2:3306", "memberRole": "SECONDARY", "mode": "R/O", "readReplicas": {}, "replicationLag": "applier_queue_applied", "role": "HA", "status": "ONLINE", "version": "8.4.5" }, "ArunClusterND3:3306": { "address": "ArunClusterND3:3306", "memberRole": "PRIMARY", "mode": "R/W", "readReplicas": {}, "replicationLag": "applier_queue_applied", "role": "HA", "status": "ONLINE", "version": "8.4.5" } }, "topologyMode": "Single-Primary" }, "groupInformationSourceMember": "ArunClusterND3:3306" } MySQL ArunClusterND1:3306 ssl Py > |
Step 11: Notice: Metadata version mismatch warning
While reconnecting to the cluster after the upgrade, you might encounter the following warning in MySQL Shell:
1 2 | MySQL ArunClusterND1:3306 ssl Py > cluster = dba.get_cluster() NOTE: The installed metadata version 2.1.0 is lower than the version required by Shell which is version 2.2.0. It is recommended to upgrade the metadata. See ? dba.upgrade_metadata for additional details. |
Solution: Upgrading cluster metadata
This message indicates that the cluster metadata schema is outdated and needs to be upgraded to match the requirements of the MySQL Shell version 8.4.5.
To resolve the warning and restore full AdminAPI functionality, upgrade the metadata schema from version 2.1.0 to 2.2.0. This process also automatically updates the grants for any MySQL Router accounts created during bootstrapping and then cleans up a backup of the metadata to ensure a safe upgrade.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | MySQL ArunClusterND1:3306 ssl Py > js Switching to JavaScript mode... MySQL ArunClusterND1:3306 ssl JS > dba.upgradeMetadata() Metadata Schema Upgrade The topology you are connected to is using an outdated metadata schema version 2.1.0 and needs to be upgraded to 2.2.0. Without doing this upgrade, no AdminAPI calls except read only operations will be allowed. The grants for the MySQL Router accounts that were created automatically when bootstrapping need to be updated to match the new metadata version's requirements. Updating Router accounts... NOTE: 2 Router accounts have been updated. WARNING: If MySQL Routers have been bootstrapped using custom accounts, their grants can not be updated during the metadata upgrade, they have to be updated using the setupRouterAccount function. For additional information use: ? setupRouterAccount Upgrading metadata at 'ArunClusterND3:3306' from version 2.1.0 to version 2.2.0. Creating backup of the metadata schema... Step 1 of 1: upgrading from 2.1.0 to 2.2.0... Removing metadata backup... Upgrade process successfully finished, metadata schema is now on version 2.2.0 MySQL ArunClusterND1:3306 ssl JS > |
Conclusion: Upgrading a MySQL InnoDB cluster to version 8.4
Upgrading a MySQL InnoDB cluster to version 8.4 requires careful planning and execution to ensure minimal downtime and maintain cluster stability. You can achieve a smooth transition by following the step-by-step process—safely upgrading secondary nodes first, controlling group replication startup, and finally upgrading the primary node. Additionally, addressing deprecations, adjusting configuration variables, and upgrading cluster metadata are crucial to fully leverage the new version’s features and maintain cluster health. With these best practices, your MySQL cluster will be robust, up-to-date, and ready to support your applications reliably. To dive deeper into MySQL upgrade testing, especially using tools like pt-upgrade and checkForServerUpgrade, check out this detailed post.
About the Author
Arunjith AravindanArunjith consults with Percona's customers on building and maintaining reliable and high-performance MySQL infrastructures. Arunjith joined Percona in Jul 2014.
