Why can't I use a secondary IP address to connect to my Amazon EC2 Linux instance?

Short description

Before you use a secondary IP address to connect to your EC2 instance, make sure that the instance adheres to the configuration requirements. Then, use SSH with verbose messaging activated to identify and resolve the error.

If you still encounter issues, then use one of the following methods to connect to your instance:

• Connect with the primary IP address
• Use Amazon EC2 Serial console
• Use a rescue instance

Then, check the network configuration file and logs for errors.

Resolution

Make sure that the instance adheres to the configuration requirements

Make sure that the operating system (OS) on your instance recognizes the secondary private IP address. For Amazon Linux, see How do I configure a secondary private IPv4 address on my EC2 instance? For Ubuntu, see How do I make my secondary network interface work in my Ubuntu Amazon EC2 instance? For other Linux distributions, see the network configuration documentation for your distribution.

Note: It's a best practice to keep backups of your instances and data. Before you troubleshoot, create an Amazon Machine Image (AMI) of your instance. Or, create a snapshot of your instance's Amazon Elastic Block Store (Amazon EBS) volumes.

Use SSH with verbose messaging activated on to identify the error

Prerequisites: Make sure that you adhere to the SSH connection prerequisites.

For instructions, see Use SSH with verbose messaging turned on on How do I resolve SSH connection issues to my Amazon EC2 Linux instance?

Note: If you launched your instance in a private subnet, then use the secondary private IP address to connect through SSH. Make sure to choose the secondary IPv4 address from the IPv4 CIDR block of the subnet for the elastic network interface. If you launched your instances in a public subnet, then allocate an Elastic IP address to your instance's secondary private IP address.

Use the primary IP address to connect to your instance

Complete the following steps:

1. Use the primary IP address to connect to your instance with SSH.

2. To verify that the instance's secondary network interface is running, run one of the following commands:

   ifconfig -a

   -or-

   ip a show eth1 up

   Note: Replace eth1 with the name of your secondary network interface.

3. To view the secondary network interface configuration file details, run the following command based on your distribution.
   Amazon Linux, Red Hat Enterprise Linux (RHEL), or CentOS:

   sudo cat /etc/sysconfig/network-scripts/ifcfg-eth1

   Ubuntu:

   sudo cat /etc/network/interfaces.d/51-eth1.cfg  

   -or-

   sudo cat /etc/netplan/51-eth1.yaml

   Example configuration file:

   DEVICE=eth1
   
   BOOTPROTO=dhcp
   
   ONBOOT=yes
   
   TYPE=Ethernet
   
   USERCTL=yes
   
   PEERDNS=yes
   
   DHCPV6C=yes
   
   DHCPV6C_OPTIONS=-nw
   
   PERSISTENT_DHCLIENT=yes
   
   RES_OPTIONS="timeout:2 attempts:5"

4. To check your network interface name and MAC addresses, run the following command:

   ip link

5. Make sure that the secondary network interface configuration file parameters are correct. For example, check the MAC addresses, secondary IP addresses, and secondary interface names. If you find incorrect parameter values, then update the file.

6. Run the following command based on your Linux distribution to restart the network services.
   Amazon Linux (AL2), RHEL versions earlier than 9, or CentOS:

   sudo service network restart

   Amazon Linux 2023 (AL2023) or RHEL 9 or later:

   sudo systemctl restart systemd-networkd.service

   Ubuntu:

   sudo netplan apply

If you still encounter issues, then check the system and authentication logs to identify issues at the time when you tried to access the instance.

Use the EC2 Serial Console to access your instance

Use the EC2 Serial console to connect to instances that are built on the AWS Nitro System and supported bare metal instances. Then, troubleshoot your instance.

Note: If you're using the serial console for the first time, then review the prerequisites and configure access before you connect to the instance.

Use a rescue instance to access the network configuration file and logs

Configure your instance for a stop and start

Note: When you stop and start an instance, the instance's public IP address changes. It's a best practice to use an Elastic IP address to route external traffic to your instance instead of a public IP address. If you use Amazon Route 53, then you might need to update the Route 53 DNS records when the public IP address changes. A stop and start is different from an instance reboot. For more information, see How EC2 instance stop and start works.

Before you stop and start your instance, take the following actions:

• If your instance uses an instance store, then save the volume data on the instance store to persistent storage. For example, you can save the data to Amazon EBS volumes or Amazon Simple Storage Service (Amazon S3) buckets.
  Important: Amazon EC2 deletes instance store data when you stop the instance.
• Create a snapshot of your EBS volume. If you experience issues, then you can use the snapshot to restore your instance.
• Temporarily remove the instance from its Amazon EC2 Auto Scaling group so that you don't accidentally terminate the instance when you stop it.
  Note: EC2 Auto Scaling might terminate stopped instances in an Auto Scaling group based on your scale-in protection settings. Instances that you launch with Amazon EMR, AWS CloudFormation, or AWS Elastic Beanstalk might be in an Auto Scaling group.
• Set the instance shutdown behavior to Stop to make sure that the instances don't terminate when you stop them.

Launch the rescue instance

Complete the following steps:

1. Launch a new rescue instance in your virtual private cloud (VPC). Use the same AMI and the same Availability Zone as the original instance.
   Note: It's a best practice to use an AL2023 instance as the rescue instance. This configuration means that the rescue instance doesn't boot up from the attached volume because the UUID and volume name are the same.

2. Stop the original instance.

3. Detach the EBS root volume, such as /dev/xvda or /dev/sda1, from the original instance. Note the device name of your root volume.

4. Attach the volume as a /dev/sdf secondary device to the rescue instance.
   Note: If your instance has AWS Marketplace codes and isn't an Amazon Linux instance, then stop the rescue instance before you attach the root volume. For example, an instance that you launched from an official RHEL or CentOS AMI might have AWS Marketplace codes.

5. Start the rescue instance.

6. Use SSH to connect to your rescue instance.

7. Run the following command to verify that the EBS volume successfully attached to the rescue instance:

   lsblk

   Example output:

   NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
   xvda    202:0    0   20G  0 disk 
   └─xvda1 202:1    0   20G  0 part /
   xvdf    202:80   0  100G  0 disk

8. To change to the root user, run the following command:

   sudo su

9. To create a mount point directory for the volume that you attached to the rescue instance, run the following command:

   sudo mkdir /rescue

   Note: Replace /rescue with your mount point directory name.

10. To mount the volume to the new directory, run the following command:

    sudo mount /dev/xvdf1 /rescue

    Note: Replace dev/xvdf1 with your root volume's device name and /rescue with your mount point directory name. If you receive an error when you run the preceding command, then see Why can't I mount my Amazon EBS volume?

11. To view the system logs, run the following commands based on your distribution:
    AL2, RHEL, CentOS:

    sudo cat /rescue/var/log/messages

    Ubuntu and Debian:

    sudo cat /rescue/var/log/syslog

    Note: To view AL2023 system logs, you must use journalctl.
    To view the authentication logs, run the following commands based on your distribution:
    Amazon Linux, RHEL, CentOS:

    sudo cat /rescue/var/log/secure

    Ubuntu and Debian:

    sudo cat /rescue/var/log/auth.log

    In the output, check for errors at the time when you tried to access the instance.

12. To view the secondary network interface configuration file details, run the following command based on your distribution:
    Amazon Linux, RHEL, or CentOS:

    sudo cat /etc/sysconfig/network-scripts/ifcfg-eth1

    Ubuntu:

    sudo cat /etc/network/interfaces.d/51-eth1.cfg  

    -or-

    sudo cat /etc/netplan/51-eth1.yaml

13. To check your network interface name and MAC addresses, run the following command:

    ip link

14. Make sure that the secondary network interface configuration file parameters are correct. For example, check the MAC addresses, secondary IP addresses, and secondary interface names. If you find incorrect parameter values, then update the file.

15. To unmount the secondary device from your rescue instance, run the following command:

    sudo umount /rescue

    Note: Replace /rescue with your mount point directory name.

16. Detach the volume from the rescue instance.

17. Attach the volume to the original instance as the /dev/xvda or /dev/sda1 root volume.

18. Start the instance.