AOTP Dev

In my last post on Infrastructure as Actual Code, I reviewed all of the options for writing code that talks to the three major hyperscalers: AWS, Azure and GCP. While there are SDKs for many different languages and different levels of support, the main languages supported across all three are Python, Go and JavaScript. I wanted to take a closer look at all three and build the infrastructure I need to run Portworx:

• a VPC
• subnet in the VPC
• routing to subnet from the interwebs
• security rules to allow traffic
• VMs and VM config
• Kubernetes install
• Portworx install

So what’s it like to write code in Python that builds cloud infrastructure? Well, I’m gonna skip the part where we go through the dance of managing Python versions, tools to create/manage virtual environments, dependencies, and IDEs. I’ll tell you what I’m using here, but you do you.

For managing python versions, I am using pyenv. This lets me define the version of python I want to use for a particular projects. I create my virtual environments with virtualenv, I’m using straight up pip to manage installing modules, and for this project I’m using PyCharm as my IDE.

Python and AWS

I picked Python on AWS to write about first because it’s the most well understood and documented at this point.

You can find the AWS SDK for Python here. Most folks know this as Boto3. Boto was originally a customer contributed library but eventually moved to become the official Python SDK for AWS. If you’re interested in why the project is named boto, there is an issue filed on that.

I mentioned this in the 1st part of the article – these SDKs are built for more than just infrastructure. If you look at the services that Boto3 works with, it’s just one of many, many services. How do we get started with Boto3? For me, it started like this:

mkdir snake-mf-aws
cd snake-mf-aws
pyenv local 3.10.0
python -m virtualenv venv
source venv/bin/activate.fish
pip install boto3

Now I open the directory in PyCharm. I’m not building an advanced library, just some tactical bit of code (for now) so I create smfa.py and start with importing boto3 and os. I’m going to go with the tried and true method of pulling credentials from environment variables, to avoid getting myself into trouble and somehow hard coding. I’d like to do something to get these credentials into Vault or some very basic secret management, but this will do for now.

I only needed to import two modules to get started:

import boto3
import os

First I built a function that will take credentials and return a boto session client. For a much more detailed look at the differences between using the boto’s sessions or using module level functions, check out Ben Kehoe’s excellent medium post. For my code, I was thinking that if I use python going forward, I would want a more abstract approach to connection. Ideally I’d want to create an abstraction for connection that could work across multiple clouds. Building the session client in a function may or may not be necessary but it did make for some clean looking code in the end. I take in an access key, secret key and default region, and return the session client.

def create_aws_client(access_key: str,
                      secret_key: str,
                      region: str) -> boto3.Session.client:

    try:
        aws_session = boto3.Session(
            aws_access_key_id=access_key,
            aws_secret_access_key=secret_key,
            region_name=region
        )
        print("Starting AWS Session...")
        aws_client = aws_session.client("ec2")
        return aws_client
    except Exception as e:
        print(e)

As a simple test, I build an EC2 instance. It turns out this is straightforward once you see a few examples, although for me the documentation wasn’t super helpful. Maybe that’s just because I’m a part time dev, and I don’t work with other elements of the AWS SDK. Actually, it probably has a lot to do with the fact that I’m an infrastructure nerd first, and thinking about a VM as an object like all the other objects that can be addressed is counter-intuitive for me. Here’s my function for creating EC2 instances:

def create_ec2_instance(aws_client: boto3.Session.client,
                        image_id: str,
                        instance_count: int,
                        ssh_key_name: str):

    try:
        instance_return = aws_client.run_instances(
            ImageId=image_id,
            MinCount=instance_count,
            MaxCount=instance_count,
            InstanceType="t3.large",
            BlockDeviceMappings=[
                {
                    'DeviceName': '/dev/sdd',
                    'Ebs': {
                        'DeleteOnTermination': True,
                        'VolumeSize': 80,
                        'VolumeType': 'standard'
                    }
                }
            ],
            TagSpecifications=[
                {
                    'ResourceType': 'instance',
                    'Tags': [
                        {
                            'Key': 'Owner',
                            'Value': 'chris'
                        },
                    ]
                },
            ],
            EbsOptimized=True,
            KeyName=ssh_key_name,
            UserData=""
        )
    except Exception as e:
        print(e)

My function takes in a boto.session.client object, the AMI id (Image Id), the number of instances I want to deploy, and the name of the already created SSH Key Pair to be used. I could expand the parameters to include more things, but I was kind of getting impatient at this point, so I left the rest as hard coded. Of note here – the instance type, EBS disk, and a Tag. We have an app running against our AWS regions that will clean up anything that’s not tagged, so it’s important when I deploy to tag everything.

This code will work, but it will go off and deploy the instance or instances and not give you much feedback about what’s going on. So I added a little bit of extra code before the exception handling. Originally, the code was just running the run_instances function, but I quickly realized I needed to get the output from that call and parse details so I could do additional work like below.

        print("Creating Instances...")
        for i in range(instance_count):
            print(f"{instance_return['Instances'][i]['InstanceId']} has been created...")
        instance_id_waiting = instance_return['Instances'][instance_count - 1]['InstanceId']
        # Trying the instance_status_ok waiter for now
        print(f"Waiting for instance {instance_id_waiting} to pass Status check...")
        aws_client.get_waiter('instance_status_ok').wait(
            InstanceIds=[instance_id_waiting]
        )
        print("Instance is up and running!")

I wanted to print out the InstanceId for each instance, just so I knew how to get instance information out of what comes from the run_instances call that happens before this. Then I was thinking – how would I know when I’m ready to move to the next stage? If after this deployment, I was going to layer on Kubernetes and Portworx, how would I know when the instances are ready? Boto provides “waiter” functions that allow you to wait for specific parameters. In this case, I use the session client passed into the function, and use a waiter for ‘instance_status_ok’ and pass it the last InstanceId I got from my call to run_instances.

That’s the code that I have so far. It works great to deploy EC2 instances. My main() function looks like this to call:

if __name__ == "__main__":
    aws_access_key = os.environ['AWS_ACCESS_KEY_ID']
    aws_secret_key = os.environ['AWS_SECRET_ACCESS_KEY']
    aws_region_name = os.environ['AWS_DEFAULT_REGION']
    aws_image_id = "ami-uuid"
    instances = 1
    ssh_key = "ssh-key-pair"
    ec2_client = create_aws_client(aws_access_key, aws_secret_key, aws_region_name)
    create_ec2_instance(ec2_client, aws_image_id, instances, ssh_key)

This requires the environment variables to be defined at runtime, and also requires you to know the UUID of the ami image you want to use, and to have an SSH Key Pair already defined that you have access to. Note that the code as is will deploy a t3.large instance to your default VPC, default subnet. This will incur charges on your account if you use it! If you want to test micro images and keep costs down, change the InstanceType and make sure you supply an AMI that supports the instance type.

If this was something I was going to use longer term, I would definitely look to make the exception handling and logging more robust. As my comparison continues it will likely make sense to circle back on this point.

That’s a good start to seeing how Python / Boto3 works with AWS. Next, I’m going to move on to testing the same basic create connection and run EC2 instances with Go.