GNU/Linux, Open Source, Cloud Computing, DevOps and more...

Script to automatically change all gp2 volumes to gp3 with aws-cli

18 comments

Last December Amazon announced its new EBS gp3 volumes, which offer better performance and a cost saving of 20% compared to those that have been used until now (gp2). Well, after successfully testing these new volumes with multiple clients, I can do nothing but recommend their use, because they are all advantages and in these 2 and a half months that have passed since the announcement I have not noticed any problems or side effects.

Making the change is extremely simple, just select a volume from the AWS console and change its type like this:

However, doing it this way when you have lots of clients that may have hundreds or thousands of these volumes, makes this a tedious, time-consuming and error-prone task, as we may inadvertently select a volume that was not gp2, or change it to another type that is not the expected gp3. And once we have made a mistake, the volume remains in the “modifying” state for quite some time and then in the “optimizing” state for even longer, which will require us to move on to something else and then remember to review all the volumes and correct those where we made a mistake.

To avoid these problems when we have a large number of EBS volumes, I wrote a very simple bash script that completely automates the task, saving us a lot of time and tedium and protecting us from making mistakes:

#! /bin/bash

region='us-east-1'

# Find all gp2 volumes within the given region
volume_ids=$(/usr/bin/aws ec2 describe-volumes --region "${region}" --filters Name=volume-type,Values=gp2 | jq -r '.Volumes[].VolumeId')

# Iterate all gp2 volumes and change its type to gp3
for volume_id in ${volume_ids};do
    result=$(/usr/bin/aws ec2 modify-volume --region "${region}" --volume-type=gp3 --volume-id "${volume_id}" | jq '.VolumeModification.ModificationState' | sed 's/"//g')
    if [ $? -eq 0 ] && [ "${result}" == "modifying" ];then
        echo "OK: volume ${volume_id} changed to state 'modifying'"
    else
        echo "ERROR: couldn't change volume ${volume_id} type to gp3!"
    fi
done

The only requirement is to have the jq tool to better handle the JSON returned by the aws command. It can be installed by simply running apt install jq or yum install jq, as it is included in the repositories of all Linux distributions.

Finally, it should be noted that this script will cause the default values for the IOPS (3000) and Throughput (125 MB/s) parameters to be set, but as these far exceed the performance offered by the gp2 volumes, this should not be a problem.

Needless to say, this same script can be used to massively change EBS volumes of other types, such as io1, io2, sc1, st1, etc.

Hope it helps you save time and headaches!

 

About the author

Daniel López Azaña
Freelance AWS Cloud Solution Architect & Linux Sysadmin

Entrepreneur, a generator of ideas and restless mind. Passionate about new technologies, especially Linux systems and Open Source Software. I also like to write about Technology News, Cloud Computing, AWS, DevOps, DevSecOps, System Security, Web Development and Programming, SEO, Science, Innovation, Entrepreneurship, etc.

DanielScript to automatically change all gp2 volumes to gp3 with aws-cli

Related Posts

18 comments

Join the conversation
  • Zachary Armstrong - 27/03/2021 reply

    You should remove the ! from the error output message, as it will crap out in bash.

    Daniel - 27/03/2021 reply

    The ! is not interpreted by bash as it’s part of a string between quotes. I don’t see any problem by using it, but feel free to remove it if it’s causing any problem to you!

    Zachary Armstrong - 31/03/2021 reply

    That’s not true. If it’s enclosed in single quotes, you can use the !, but if you use double quotes, as you did in the scrip, you have to slash escape it \!. If it’s working for you with double quotes, that means you have history expansion disabled, as ! is the default history expansion character. History expansion is on by default, so you (or someone else) must have purposely disabled it, or you’re using some odd flavor of bash/csh that hs it disabled by default. More info here: https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html#History-Interaction

    Also, the background of the reply window I’m typing in is the dar gray of the site and the font color is a dark gray, too, making it near impossible to write this out. I ended up writing this response in a text editor then pasting it. I’ll email you a screenshot.

    Daniel - 05/04/2021 reply

    Sorry, but history expansion is enabled by default only for interactive shells. Scripts don’t run using interactive shells, so history expansion is disabled and you can perfectly use the ! between double quotes. I never disabled history expansion explicitly and I never had the problem you mentioned after writing hundreds of scripts in many different Linux systems. But of course I’m not saying that there can’t be some linux distro or some specific configuration on some systems that brings history expansion enabled for non-interactive shells. If so, just escape the character as you said and you’re done.

    Thanks for your warning about the font color, now it’s fixed!

    Zachary - 05/04/2021

    Still having the font color issue on this reply form 🙁

    I think there is where the disconnect is — you’re running it in non-interactive, while I’m running in interactive. To me, this seems like a run once script to migrate older machines, so I wouldn’t be running it non-interactively. Curious as to your use case that would make non-interactive ideal.

    Daniel - 13/04/2021

    With non-interactive I mean that it’s run from a script even if it’s launched interactively from the command line. Take a look at this simple example and you will see what I mean:

    $ cat hello-world.txt
    #! /bin/bash

    echo “Hello world!”

    $ ./hello-world.txt
    Hello world!

    $ echo “Hello world!”
    -bash: !”: event not found

    It’s exactly the same command, but first I run it from a script and it works as expected, and then I run it directly from command line and it fails as you said. But as my article is about a script, the first approach is perfectly valid.

    Regards!

    jack - 07/09/2022 reply

    How do we run this script ? can you please provide steps for that ?

    Daniel - 11/09/2022 reply

    You save it to a file, ie change-gp2-to-gp3.sh, then run the command chmod u+x change-gp2-to-gp3.sh to give it execution permission, and then run ./change-gp2-to-gp3.sh from the same directory the script is stored to execute it.

  • Dennis Ogunfiditimi - 06/07/2021 reply

    Hi Daniel – How can I add your script to a Lambda function and Run a monthly event to check all the volume in my account and auto modify volume types from a gp2 – gp3 on a Monthly basis? Thank you

    Rook Bravo - 31/10/2021 reply

    Create a trigger in cloudwatch events or eventbridge and set a schedule or a cron expression to trigger this lambda function.

  • Victor Chavez - 08/07/2021 reply

    no need for jq. use:

    –query ‘Volumes[].VolumeId’

  • Rook Bravo - 27/10/2021 reply

    I took the liberty of enhancing this script a little further to Migrate gp2 to gp3 volumes of specific ec2 instances by tags

    region=”us-east-1″
    for ec2_instance_id in $(aws ec2 describe-instances –region $region –filter “Name=tag:env, Values=dev” | jq -r ‘.Reservations[].Instances[].InstanceId’)
    do
    for volume_id in $(aws ec2 describe-instances –region $region –instance-id $ec2_instance_id | jq -r ‘.Reservations[].Instances[].BlockDeviceMappings[].Ebs.VolumeId’)
    do
    volume_type=$(aws ec2 describe-volumes –region $region –volume-id $volume_id | jq -r ‘.Volumes[].VolumeType’)
    if [ $volume_type == ‘gp2’ ]; then
    echo “Volume id: $volume_id is of type $volume_type and belongs to Instance id: $ec2_instance_id”
    result=$(aws ec2 modify-volume –region ${region} –volume-type=gp3 –volume-id ${volume_id} | jq ‘.VolumeModification.ModificationState’ | sed ‘s/”//g’)
    if [ $? -eq 0 ] && [ $result == “modifying” ]; then
    echo “OK: volume ${volume_id} changed to state modifying”
    else
    echo “ERROR: could not change volume ${volume_id} type to gp3”
    fi
    fi
    done
    done

    Daniel - 01/11/2021 reply

    Thanks!

    Lokesh - 25/03/2022 reply

    getting error for thisjq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end (Unix shell quoting issues?) at , line 1:
    ‘.Reservations[].Instances[].InstanceId’
    jq: 1 compile error

    ‘ascii’ codec can’t decode byte 0xe2 in position 0: ordinal not in range(128)
    script .

  • Ratul Dutta - 02/02/2022 reply

    Hi Daniel.. your script helped a lot today . need to convert more than 200 disks from gp2 to gp3.. done easily using your script… Thanks a lot.. 🙂

    Daniel - 02/02/2022 reply

    Great! Good to know 🙂

  • Rocket - 03/06/2022 reply

    Hi Daniel, this script LGTM OMG!
    your script saved me from a lot of gp2 volumes hell. this script works good without any problem.

  • Chandler - 11/08/2022 reply

    Hi Daniel!
    Thanks for the awesome script, works like a charm 🙂

Leave a Reply

Your email address will not be published.