Security Baselining AWS Accounts

January 17, 2018 Liz Duke

As someone who works with cloud solutions for a security company, I am very aware of the stories in various media about security breaches in cloud accounts. Usually these are along the lines of “Company X exposes millions of customer data records by storing them in a bucket in the cloud,” or “Company Y charged thousands for instances created with credentials found in GitHub.” Sometimes the story comes with an inference that the cloud is insecure.

So, is the cloud insecure? My experience is based on Amazon Web Services (AWS), so I’ll answer that based on my experience.

“Are AWS solutions insecure?”

My answer, “No, unless people make them insecure.”

Why do I say this?

In my experience,  AWS is secure by default. Not only that, but they write lots of documentation explaining their security and where the borderline exists between what is their responsibility and what is the customers responsibility. They also stage hundreds of free sessions in AWS summits around the world on security and how to secure your cloud installations.

AWS does give us the ability to control our solutions which can lead to unintended consequences. No one intentionally exposes their data (well, I hope not). If you have a small installation or a single account and follow the guidelines and documentation it should be fairly straightforward to manage your security.

If, on the other hand, you grow rapidly and have lots of different accounts and levels of knowledge then you may find yourself facing security issues.  It is important to define a security baseline, a set of objectives for your security that must be met, and then ensure that they are being followed.

How does AWS help you secure your accounts?

Amazon provides some great tools which help you flag security issues in your accounts and audit your accounts. This lets you understand your basic security. With the right tools in place you can also track the events up to, and around, a security issue.

Automating security baseline tools is something that can help keep track of and alert you to changes in your security posture. With this in mind I had a look at some of the tools and asked myself how would I configure these, and could I roll out the configuration across both accounts that were in use and new accounts.  The templates I created are provided as a GitHub link below.

What Tools to Use for Automating Security Baselining

The tools I thought best fit for this are the following:

  • Trusted Advisor
  • CloudTrail
  • CloudWatch Alarms and Dashboard
  • Config – this is not templated as yet as integrating it with accounts that have config already enabled is a little more challenging and to enable in all regions requires stack-sets.

These tools work together to build up a picture of, and audit trail for, your account security. They won’t track application vulnerabilities (there are other tools to do that) but they will let you know if there are holes in your perimeter security and when those holes were created.

Trusted Advisor

Trusted Advisor is available in 2 flavours – a limited set of checks for a standard account and a more detailed, expanded set of checks for accounts with Business or Enterprise level support.

There are a lot of rules you can enable for Trusted Advisor. I wanted to know if the ones that most concerned me from a security standpoint were being breached. Things like:

  • Is someone creating rules allowing access from anywhere to an instance, load-balancer or RDS database?
  • Are people opening non-standard ports to the internet.
  • Are people rotating their IAM Access keys? We use Identity Federation with MFA for most accounts but there are legacy users or accounts that are being used as Sandboxes where this is not the current situation.
  • Are people storing IAM Access keys in public repositories?
  • Are people opening up S3 bucket permissions for write access.
Template

The template to do this is fairly small, it sets up a CloudWatch Events rule to log the above security information from the source of Trusted Advisor to an SNS topic which is also created. There is a parameter to add an email address as a subscriber to the SNS topic.

Before you enable Trusted Advisor checks and sign up to the SNS topic included in the template in an existing account,  I would recommend doing an audit to remediate the status of the account and any checks in error status. I set the check level at “ERROR” status,  as I wanted this to be about immediate threats and not to get cluttered with warnings.

The template also configures an SNS topic for alerts to be sent to and an email address which is provided when creating the stack to subscribe to the SNS topic.

Warning – this can create a lot of emails if you do not audit the account first. You may use a separate email for this so that you get an idea of your security progression and when you are happy, add your monitoring team to the notifications.

Setup – Which checks are included in the template?

Security Groups – Specific Ports Unrestricted” – Alerts if access to port 20, 21, 1433, 1434, 3306, 3389, 4333, 5432, or 5500 is unrestricted.
Security Groups – Unrestricted Access” – Alerts if a Security Group rule has a source IP address with a /0 suffix for ports other than 25, 80, or 443
ELB Listener Security” – Alerts if a load balancer listener uses an insecure cipher or protocol.
ELB Security Groups” – Alerts if a load balancer has a Security Group assigned which does not exist.
Amazon RDS Security Group Access Risk” – Alerts when a Security Group assigned to an RDS database grants global access.
IAM Access Key Rotation” – Alerts when the access key is active and has not been rotated in the last 2 years.
IAM Use” – Alerts if there are no IAM users in use.
Amazon S3 Bucket Permissions” – Alerts if the bucket ACL allows Upload/Delete access for “Everyone” or “Any Authenticated AWS User”.
Exposed Access Keys” – Alerts if AWS suspects IAM credentials are exposed from scanning of public repositories or irregular use of the credentials.

You can read more about the checks in the AWS console for Trusted Advisor, which has an explanation of each check in a dropdown menu under the check itself. As there are two levels of use for Trusted Advisor the only checks that are available at the free level in the above list are “Security Groups – Specific Ports Unrestricted” and “IAM Use”.

CloudTrail

CloudTrail is an auditing and security tool. Most AWS services are integrated to CloudTrail and as it is a much shorter list to show what is not integrated with CloudTrail currently, that is the link I have shared here: CloudTrail Unsupported AWS Services

Template

This template sets up and configures CloudTrail. The CloudTrail events are then sent to CloudWatch logs which are then monitored by metrics setup in the CloudWatch template.

Trail Setup

The trail is applied to all regions. One trail per region is free for each account. The reason to do this is to ensure that if someone gains access to an account and creates resources in a region not used normally within that account the activity is still picked up.

By default, trails are configured to log management events. All events that are not data events are management events. Example management events include the EC2 RunInstancesDescribeInstances, and TerminateInstances API operations. Management events can also include non-API events that occur in your account. For example, when a user logs in to your account, CloudTrail logs the ConsoleLogin event. For more information, see the AWS documentation at Non-API Events Captured by CloudTrail.

The trails are setup to log Management Events in Write only mode. Write-only events include API operations that modify (or might modify) your resources. For example, the Amazon EC2 RunInstances and TerminateInstances API operations modify your instances.

The trails are also setup to log all Data Events in Read only mode. This logs all “GET” requests to your S3 buckets. This is useful if you have sensitive information in your bucket(s) that you want to ensure is not accessed by anyone that should not have access.

A CloudTrail bucket is created by the template and a Bucket Policy created and attached to the bucket. The policy allows the CloudTrail service to push objects to the bucket. An SNS topic is also setup to provide a stream of CloudTrail events that can be subscribed to. The CloudTrail logs pushed to the bucket are validated and an MD5 checksum of each log is created so that logs cannot be tampered with.

CloudWatch Logs are also written to by CloudTrail – there is an IAM policy created that allows this action and the logs are  rotated after a set period ( this is currently at 7 days but will be increased before use). This allows CloudTrail events to be monitored through the logs – as the events are monitored then we can alert  on specific conditions.

CloudWatch

This template is adapted from the one provided by AWS to create CloudWatch Logs for events that could be interesting from a security perspective.

The template can be applied to any AWS account and will then start gathering logs when any of the monitored activities happen. Each alert has metrics applied to it which decides what goes into the CloudWatch Log and then the Alert will trigger when the threshold is passed.

Template
Metrics Monitored
Security Group Changes Metric Filter

If a Security Group is modified in anyway, including adding to an existing Security Group, then an entry is added to the log. The reason behind this is that Security Groups should not be changing without some request and therefore we need to monitor the updates to any Security Groups.

Network ACL Changes Metric Filter

If an ACL is modified in any way, created, deleted, updated etc. an entry is added to the Cloudwatch Log. Again, for reasons of security, these entries should not be changing without people being aware.

Gateway Changes Metric Filter

If a customer gateway is created or deleted, an internet gateway is created, deleted, attached or detached then an entry is added to the log. This prevents connections to on-premise or to the internet from being modified without being logged.

VPC Changes Metric Filter

If a VPC is created, deleted, has its attributes changed or a peering connection is modified in anyway, then an entry is added to the log. This adds to the gateway metric to alert if there are any changes to the VPC itself, including what it is peered to.

EC2 Changes Metric Filter

If an instance is created, terminated, stopped, started or rebooted an entry is sent to the log. This will flag if instances are being created when they should not be or if an instance has an issue. This is important for security (why are instances being created if there is no schedule for that) and for costs. If you have constant scaling actions in your account you may want to change or remove this metric.

EC2 Large Instance Changes Metric Filter

This is specifically targeted at 4x and 8x instances being created as they are very costly. The actions logged are the same as the ones in the standard EC2 metric.

CloudTrail Changes Metric Filter

As CloudTrail is a security tool we need to know if a trail is being changed in any way (creation, deletion, modification) including the stopping and starting of logging to that trail. All such actions are logged with this filter.

Console Signin Failures Metric Filter

If someone tries to login to the AWS console and fails authentication, this is logged. This helps to track is someone is trying to gain access to one of our accounts.

Authorization Failures Metric Filter

If an unauthorized call is made to an api, the call is logged.  This would help track if access was being attempted to more resources than the user was granted.

IAM Policy Changes Metric Filter

Any changes to IAM policies are logged.

In addition to the filters there are alarms created for each filter which have individual thresholds. These alarms are also sent to an SNS topic which is available for subscription (or could be fed into a 3rd party logging system).

Outputs

For each alarm an output is created with the “ARN” (Amazon Resource Name)of the alarm, this value is exported. The reason behind doing this was to provide the required inputs for a dashboard to be created with the alarms in which is the final template. The threshold for the Alarms generated by these metrics should be tuned dependent on your account requirements. If you have development environments that regularly get created/destroyed you will definitely need to tune the thresholds.

Bug Alert

Currently there is a bug with Cloudtrail being redirected from the global endpoint to the destination S3 bucket. This flags a permissions error in the logs which means this metric is spiky, The bug is a known issue in AWS and is planned to be fixed in the near future.

CloudWatch Dashboard

The reason I included a CloudWatch Dashboard template was that although we use Nagios and Splunk for a lot of our monitoring coverage, that tends to apply to more customer facing accounts/environments. I wanted to provide a dashboard that could be used in all types of environments so that it is a visual reminder of when security changes are made.

Template

All of the alarms in the CloudWatch Alert Template have a widget created which is added to a dashboard.

A brief explanation is required here, as this was painful to implement.

The Dashboard widget creation requires a string. This string is very long and doesn’t appear to support using pseudo parameters such as region or account variables. To work around this the Alert Template exports an ARN for each Alarm, then the Fn:: Join is used to connect the outputs from an import of those parameters back into the string.

Example with explanation

DashboardName": "MyDashboard",
        "DashboardBody": {"{\"widgets\":[{\"type\":\"metric\",\"x\":0,\"y\":0,\"width\":6,\"height\":6,\"properties\":{\"title\":\"CloudTrailNetworkAclChanges\",\"annotations\":{\"alarms\":[\"arn:aws:cloudwatch:us-east-1:404063407815:alarm:CloudTrailNetworkAclChanges\"]},\"view\":\"timeSeries\",\"stacked\":false}},{\"type\":\"metric\",\"x\":6,\"y\":0,\"width\":6,\"height\":6,\"properties\":{\"title\":\"CloudTrailIAMPolicyChanges\",\"annotations\":{\"alarms\":[\"arn:aws:cloudwatch:us-east-1:404063407815:alarm:CloudTrailIAMPolicyChanges\"]},\"view\":\"timeSeries\",\"stacked\":false}},{\"type\":\"metric\",\"x\":12,\"y\":0,\"width\":6,\"height\":6,\"properties\":{\"title\":\"CloudTrailAuthorizationFailures\",\"annotations\":{\"alarms\":[\"arn:aws:cloudwatch:us-east-1:404063407815:alarm:CloudTrailAuthorizationFailures\"]},\"view\":\"timeSeries\",\"stacked\":false}}]}"
                          }

The above Dashboard has 2 alarm widgets created, the dimensions after the metric

“x, y, width, height”,  refer to the location along the x and y axes of a 24 by 24 grid.

x=0, y=0 refer to the top left hand corner of the screen and the width and height control the size of the widget.

The property “annotations” is a particular type of widget. If you are referencing an Alarm (we are) you need an annotations array. There is only one value for an Alarm annotation, that of the ARN of the alarm.

The property “view” with the value “timeseries” displays this metric as a graph.

The property “stacked” means this is displayed as separate lines.

Obviously the ARN is specific to each Alarm and therefore must be substituted to make the Template re-useable for multiple accounts.

If you look at the api for CloudWatch you can use a much more friendly json view but after some digging and looking at the examples, it appears that the only way to do this at present in CloudFormation is with a very long string.

Dashboard Widget with ARN substitution example:

"DashboardBody": {
                        "Fn::Join": [
                          "",
                        [
                        "{\"widgets\":[{\"type\":\"metric\",\"x\":0,\"y\":0,\"width\":6,\"height\":6,\"properties\":{\"title\":\"CloudTrailSecurityGroupChanges\",\"annotations\":{\"alarms\":[\"",
                          {
                            "Fn::ImportValue" : "SecurityGroupChangesAlarmArn"
                          },
                            "\"]},\"view\":\"timeSeries\",\"stacked\":false}},",
                        "{\"type\":\"metric\",\"x\":6,\"y\":0,\"width\":6,\"height\":6,\"properties\":{\"title\":\"NetworkACLChangesAlarm\",\"annotations\":{\"alarms\":[\"",
                        {
                          "Fn::ImportValue" : "NetworkAclChangesAlarmArn"
                        },
						"\"]},\"view\":\"timeSeries\",\"stacked\":false}}]}"
                        ]
                      ]
                }

I’ve shown 2 widgets, as you can see, the first part of the string includes the “widgets” keyword and then you divide the strings into a portion before and after the ARN value which is substituted using an intrinsic function. The “Join” function surrounds the string to knit it all back together.

The templates described in this post are available at  Cloud Formations Examples for Security Baselining on GitHub

Conclusion

As the agility of the cloud frees us to make faster decision and deployments we still have to maintain our security. Using the tools provided by AWS and automating them so that the setup is consistent across accounts helps maintain a security baseline posture. This is important in production and non-production accounts.

Previous Article
Software Protection is Like a SPIDER Web…
Software Protection is Like a SPIDER Web…

Software Protection: Integrity, Diversity, Entanglement and Renewability (SPIDER) We’ve spent a lot of time...

Next Article
The Spectre of Un-Patchable Hardware Haunts Us All — Don’t Meltdown!
The Spectre of Un-Patchable Hardware Haunts Us All — Don’t Meltdown!

Ofttimes it has been difficult to explain the role of software protection in hardware-protected secure syst...