How exactly to perform automated incident reaction in a multi-account environment

How quickly you react to security incidents is paramount to minimizing their impacts. Automating incident reaction can help you scale your features, decrease the scope of compromised sources rapidly, and reduce repetitive function by security teams. However when you make use of automation, you need to manage exceptions to standard response procedures also.

In this article, I provide a design and ready-produced templates for a scalable multi-account set up of an automated incident reaction process with minimal program code base, using native AWS tools. I furthermore explain how to create exception dealing with for approved deviations predicated on resource tags.

Because security reaction is really a broad topic, A synopsis is provided by me personally of some alternative methods to consider. Incident reaction is one section of an overarching governance, danger, and compliance (GRC) system that each organization should implement. To find out more, see Scaling a governance, risk, and compliance program for the cloud.

Important: Be careful when introducing automation. Test each one of the automatic responses in a non-production environment carefully, as you ought never to use untested automated incident reaction for business-critical applications.

< h2>Solution deliverables and benefits

In the perfect solution is below described, you utilize AWS Systems Manager automation to execute the majority of the incident response steps. Generally, you can write your personal or use pre-written runbooks either. AWS maintains ready-produced operational runbooks (automation documents) which means you don’t have to sustain your own code bottom for them. The AWS runbooks include many predefined use instances, such as for example enabling Amazon Simple Storage Service (S3) bucket encryption, starting a Jira ticket, or terminating an Amazon Elastic Compute Cloud (EC2) instance. Every execution for these will be properly documented and repeatable in AWS Systems Manager. For a couple cases where you can find no ready-made automation paperwork, I provide three extra AWS Lambda functions with the mandatory response actions within the templates. The Lambda features require minimal code, without exterior dependencies outside AWS indigenous tools.

You utilize a central safety account to execute the incident response actions in the automation runbooks. You don’t should do anything within the ongoing support accounts where you keep track of and react to incidents. In this article, you find out about and receive styles for:

  • Responding to a good incident predicated on Amazon GuardDuty alerts or AWS Config findings.
  • Deploying templates for the multi-account setup with the main security account and several service accounts. All account resources should be in exactly the same AWS component and Region of exactly the same AWS organization.
  • AWS Systems Supervisor automation to execute many current AWS managed automation runbooks (you will need usage of the AWS Management Console to see just about all documents as of this link). Systems Supervisor automation comes in these AWS Regions.
  • Prewritten Lambda functions to:
    • Confine permissive (open) security groupings to the more-constrained CIDR (classless interdomain routing), like the VPC (virtual personal cloud) variety for that security team. This prevents knowable system configuration mistakes, such as for example too-open security organizations.
    • Isolate the compromised EC2 example by attaching it to an individual potentially, empty security team and removing its current security groups.
    • Block an AWS Identity and Access Management (IAM) principal (an IAM user or even part) by attaching a deny all policy compared to that principal.
    • Send notifications to the Amazon Simple Notification Service (Amazon SNS) for alerting along with (and in collaboration with) automated response activities.
  • Exception handling when activities ought never to be executed. I would recommend decentralized exception handling predicated on AWS source tags.
  • Integrating AWS Security Hub with customized actions for GuardDuty findings. This is often used for guide triggering of remediations which should not receive a computerized response.
  • Customized AWS Config principle for detecting permissive (open up) security groupings on any interface.
  • Mapping the protection finding to the reaction action defined in the particular CloudWatch Events rule design is extendable. I offer an example of how exactly to extend to new responses and findings.

Out-of-scope because of this solution

This post only demonstrates how to begin with with security response automation for accounts which are section of the same AWS organization and Region. To learn more on creating a comprehensive plan for incident reaction, see How to get ready for & react to security incidents in your AWS environment.

Understanding the difference between this solution and AWS Config remediation

The AWS Config remediation feature provides remediation of non-compliant assets making use of AWS Systems Manager inside a single AWS accounts. The solution you find out in this post contains both GuardDuty AWS and alerts Config, a multi-account technique managed from the central security account, guidelines exception handling predicated on resource tags, and extra response activities using AWS Lambda features.

Choosing the proper approach for incident reaction

There are several technical solutions and patterns for giving an answer to security incidents. When considering the correct one for your firm, you must consider just how much versatility for response activities you will need, what resources come in scope, and the level of skill of one’s security engineers. Think about dependencies on external program code and applications also, software licensing costs, as well as your organization’s encounter level with AWS.

If you want the utmost extensibility and versatility beyond AWS Systems Manager found in this post, I would recommend AWS Step Functions as described within the session How to get ready for & react to security incidents in your AWS atmosphere and in DIY guide to runbooks, incident reports, and incident response. It is possible to create workflows for the runbooks and also have full control of most possible actions.



Determine 1: Architecture diagramDetermine 1: Architecture diagram

The architecture works the following:

In the services account (the surroundings for which you want to keep track of and react to incidents):

  1. GuardDuty findings are usually forwarded to CloudWatch Activities.
  2. Changes within the AWS Config compliance position are usually forwarded to CloudWatch Occasions.
  3. CloudWatch Events for AWS and GuardDuty Config are usually forwarded to the main security account, with a CloudWatch Events bus.

In the central safety account:

    1. Each event from the service account is definitely mapped to one or even more response actions using CloudWatch Events guidelines. Every rule can be an incident response activity that’s executed on one or even more security findings, as described in the case pattern. When there is an exception guideline, the response actions aren’t executed.
      The set of possible actions that may be taken include:

      1. Trigger a Techniques Manager automation record, invoked by the Lambda perform StratSsmAutomation within the security accounts.
      2. Isolate an EC2 example by attaching a clear security team to it and eliminating any prior security organizations, invoked by the Lambda functionality IsolateEc2. This assumes the incident reaction role in the prospective service account.
      3. Prevent the IAM principal simply by attaching a deny all plan, invoked simply by the Lambda perform BlockPrincipal, simply by assuming the incident reaction role in the mark service account.
      4. Confine security team to secure CIDR, invoked by the Lambda function ConfineSecurityGroup, by assuming the incident response role within the prospective service account.
      5. Send the obtaining to an SNS subject for digesting outside this solution; for instance, simply by guide evaluation or for details simply.
    2. Invoke AWS Techniques Manager within the protection account, with a target of the ongoing service account and exactly the same AWS Region.
    3. The Systems Supervisor automation document can be executed from the safety account contrary to the resources in the continuing service account; for instance, EC2, S3, or IAM resources.
    4. The response actions are usually executed from the protection accounts to the service accounts directly. This is completed by assuming an IAM function, for example, to isolate a compromised EC2 example potentially.

trigger safety responses using Security Hub customized actions

  1. Manually. This can be ideal for guide handling of complex results that require investigation before motion.

Response actions choice

Your organization really wants to articulate information security plan decisions and create a listing of corresponding automated security responses then. Factors to consider are the classification of the sources and the specialized complexity of the automation necessary. Focus on common, less complex situations to obtain immediate gains, and enhance complexity as you get experience. Several stakeholders in your company, such as for example business, IT operations, info security, and compliance and risk, should be involved with deciding which incident reaction actions ought to be automated. You will need executive assistance for the political will to execute guidelines.

Creating exceptions to automatic responses

There could be cases where it’s unwise to consider an automated response. For instance, you will possibly not want an automatic response to incidents concerning a primary production database server that’s critical to business functions. Instead, you’d desire to use individual judgment phone calls before responding. Or you understand you can find alarms that you don&rsquo perhaps;t dependence on certain resources, like alerting for an open up security group by using it as a open public web server intentionally. To handle these exceptions, there exists a carve-out. If the AWS resource includes a tag with the real name SecurityException, a response is n’t executed. The tag title is defined during set up.

Table 1 has an exemplory case of the responses applied within this solution template. You might choose different actions and various priorities for the organization. A current set of GuardDuty findings are available at Active Finding Types and for AWS Config from AWS Config Managed Rules.


See Backdoor Finding Types and

Trojan Finding Types

Isolate EC2 with empty security team.

Archive the GuardDuty getting.

Send SNS notification.

UnauthorizedAccess:IAMUser/MaliciousIPCallerSee Unauthorized Finding Types

Prevent IAM principal by attaching deny all policy.

Archive the GuardDuty locating.

Send SNS notification.

3AWS ConfigS3_BUCKET_SERVER_Aspect_ENCRYPTION_ENABLEDNotice the documentation for s3-bucket-server-side-encryption-allowed

Enable server-aspect encryption with Amazon S3-Managed keys (SSE-S3) with SSM record (AWS-EnableS3BucketEncryption).

Send SNS notification.

4AWS ConfigS3_BUCKET_PUBLIC_Study_PROHIBITEDDiscover the documentation for s3-bucket-public-read-prohibited

Disable S3 PublicRead and PublicWrite with SSM record (AWS-DisableS3BucketPublicReadWrite).

Send SNS notification.

5AWS ConfigS3_BUCKET_Open public_WRITE_PROHIBITEDFind the documentation for s3-bucket-public-write-prohibited

Disable S3 PublicRead and PublicWrite with SSM record (AWS-DisableS3BucketPublicReadWrite).

Send SNS notification.

6AWS ConfigProtection_GROUP_OPEN_PROHIBITEDSee template, customized configuration.

Confine security group to secure CIDR 172.31. 0.0/16

Send SNS notification.

7AWS ConfigENCRYPTED_VOLUMESObserve the documentation for encrypted-volumesSend SNS notification.8AWS ConfigRDS_STORAGE_ENCRYPTEDNotice the documentation for rds-storage-encryptedSend SNS notification.


  1. Within the security account, start the template by selecting Launch Stack.Select this image to open a web link that starts building the CloudFormation stack
    Additionally, you will discover the most recent code on
    GitHub, where one can donate to the sample code furthermore.
  2. Provide the next parameters for the protection account (see Figure 2):
    • S3 bucket with sources: This bucket contains all sources, like the Lambda templates and function. If you’re not customizing the resources, it is possible to leave the default textual content.
    • Prefix for S3 bucket with resources: Prefix for several items. If you’re not customizing the resources, you can depart the default.
    • Security IR-Role brands: This is actually the part assumed for the reaction activities by the Lambda features in the security accounts. The role is established by the stack released in the continuous service account.
    • Safety exception tag: This defines the tag title for security exceptions. Sources marked with this particular tag aren’t changed in reaction to a security acquiring automatically. For example, you can include an exception tag for a legitimate open security team for a public site.
    • Organization ID: That is your AWS company ID used to authorize forwarding of CloudWatch Events to the safety account. Your accounts should be members of exactly the same AWS organization.
    • Allowed network range IPv4: This CIDRv4 range can be used to confine all open up security groups that aren’t tagged for exception.
    • Allowed network range IPv6: This CIDRv6 range can be used to confine all open up security groups that aren’t tagged for exception.
    • Isolate EC2 findings: It is a set of all GuardDuty findings which should result in an EC2 example being isolated. Comma delimited.
    • Prevent principal finding: It is a set of all GuardDuty results that should result in blocking this role or even user by attaching the deny all plan. Comma delimited.
    Determine 2: Stack launch within security accountFigure 2: Stack release in security account
  3. In each assistance account, start the template by selecting Launch Stack.Select this image to open a web link that starts building the CloudFormation stack
    Additionally, you can get the most recent code on GitHub, where one can also donate to the sample code.
  4. Provide the next parameters for every service account:
    • S3 bucket with sources: This bucket contains all sources, such as for example Lambda templates and functions. If you’re not customizing the resources, it is possible to leave the default textual content.
    • Prefix for S3 bucket with resources: Prefix for several items. If you’re not customizing the resources, it is possible to leave the default textual content.
    • IR-Security role: This is actually the role that’s created and utilized by the security accounts to execute response activities.
    • Security accounts ID: The CloudWatch Events are usually forwarded to the central security accounts.
    • Enable AWS Config: Define whether you need this stack make it possible for AWS Config. In case you have enabled AWS Config currently, leave this worth false then.
    • Create SNS subject: Supply the name of a good SNS topic only when you enable AWS Config and desire to stream the configuration alter to SNS (optional). Or else, leave this industry blank.
    • SNS topic title: This is actually the title of the SNS subject to end up being created only when enabling AWS Config. The default text Config is.
    • Create S3 bucket for AWS Config: In the event that you allow AWS Config, the template creates an S3 bucket for AWS Config.
    • Bucket title for AWS Config: The title of the S3 bucket designed for AWS Config. The default textual content is config-bucket-AccountId.
    • Enable GuardDuty: For those who have not already enabled GuardDuty within the service account, then you can certainly here do it.


Once you have deployed each stacks, you can attempt your environment by following these example steps in another of your service accounts. Before you check, you can sign up to the SNS subject with prefix Protection_Alerts_[Your_Stack] to become notified of a protection event.

  1. Create and open up security group without creating an exception tag. After several minutes, the security group will be confined to the safe CIDR that you defined in your stack.
  2. Create an S3 bucket without allowing encryption. After several minutes, the default encryption AES-256 will be set on the bucket.
  3. For GuardDuty blocking of IAM principal, it is possible to define a listing of malicious IPs beneath the Threat Checklist in the GuardDuty panel. Develop a test user or even role. Once you execute an API contact out of this IP with the developed check user or role, a GuardDuty finding is generated that creates blocking of the IAM consumer or role.
  4. You can deploy Amazon GuardDuty tester and generate results such as Trojan:EC2/DNSDataExfiltration or even CryptoCurrency:EC2/BitcoinTool.B!DN. The GuardDuty findings result in isolation of an EC2 instance by detatching all current security groupings and attaching a clear one. This new empty group may then be later configured for forensic access.

Exceptions for reaction actions

If the tag is had by the reference name SecurityException, a response action isn’t executed. The tag title is really a parameter of the CloudFormation stack in the safety account and will be customized at set up. The worthiness of the tag isn’t validated, nonetheless it is good exercise for the worthiness to make reference to an approval record like a Jira concern. In this way, it is possible to create an auditing chain of the acceptance. For example:

Tag name:  SecurityException
Tag value: Jira-1234

Be sure that the security tag can only just be modified or established by a proper role. You can certainly do this in two methods. The first way would be to attach a deny declaration to all plans that do not need privileges to assign this tag. A good example policy declaration to deny setting, getting rid of, and editing of the tag for IAM, EC2, and S3 providers is demonstrated below. This policy will not prohibit dealing with the resources, such as for example stopping or beginning an EC2 instance with this particular tag. See Controlling access predicated on tag keys to learn more.

            "Sid": "S3-Sec-Tag",
            "Effect": "Deny",
            "A     ction": [
            "Resource": "*",
                    "aws:TagKeys": "TAG-NAME-SecurityException"
            "Sid": "EC2-VPC-Sec-Tag",
            "Effect": "Deny",
            "Action": "*",
            "Resource": "*",
                    "aws:TagKeys": "TAG-NAME-SecurityException"

In the aforementioned policy, you must change the TAG-NAME-SecurityException to fit your own tag name.

The second solution to restrict the attachment of the tag is by using Tag policies to manage tags across multiple AWS accounts.

Centralized versus decentralized exception administration

Attaching protection tags is really a decentralized approach where you don’t require a centralized data source to report remediation exceptions. A centralized exception data source requires you know every individual resource ID to create exceptions, and this isn’t possible always. Amazon EC2 Auto Scaling is an excellent example where you will possibly not understand the EC2 instance ID beforehand. On an up-scaling occasion, the example ID can’t end up being known beforehand and preapproved. Moreover, a central database should be held in sync with the lifecycle of the assets, like with an example down-scaling event. Therefore, using tags on sources is a decentralized strategy. If needed, the automated scaling launch construction can propagate the safety exception tag; see Tag your auto scaled EC2 instances.

It is possible to manage the IAM roles and policies for tagging either centrally or in a AWS CodePipeline. In this way, it is possible to put into action a centralized enforcement stage for protection exceptions but decentralize storing of the useful resource tags to the assets themselves.

Using AWS resource groups, it is possible to always find all sources which have the Security tag for auditing and inventory proposes. To find out more, see Build queries and groups in AWS resource groups.

Monitoring for safety alerts

You can sign up to the SNS topic with prefix Security_Alerts_[Your_Stack] to be notified of a protection event or even an execution which has failed.

Every execution is triggered from CloudWatch Events, and each one of these events generates CloudWatch metrics. Under CloudWatch guidelines, you can view the rules for occasion forwarding from program accounts, or for triggering the responses in the main security account. For every CloudWatch rule, you will see the metrics by checking the container next to the principle, as shown in Physique 3, and choose Display metrics for the guideline then.

Determine 3: Metrics for Forwarding GuardDuty Events within CloudWatchNumber 3: Metrics for Forwarding GuardDuty Events within CloudWatch

Creating new customization

plus rules

The mapping of findings to responses is described in the CloudWatch Events rules in the security account. If you define brand new responses, you merely update the central safety account. You don’t adjust the configurations of service accounts since they just events to the protection account forward.

Here’s a good example: Your group decides a new SSM actions – Terminate EC2 example – ought to be triggered on the GuardDuty selecting Backdoor:EC2/C&CActivity.B!DNS. In the security accounts, head to CloudWatch in the AWS Administration Console. Develop a new principle as referred to in the following methods (and shown in Shape 4):

Figure 4: CloudWatch guideline creationFigure 4: CloudWatch guideline creation
  1. Move to CloudWatch Activities and develop a new principle. In the Occasion Pattern, under Build custom event design, define the next JSON:
      "detail-type": [
        "GuardDuty Finding"
      "source": [
        "type": [
  2. Set the mark for the rule because the Lambda functionality for execution of SSM, StratSsmAutomation. Use Input transformer to customize the parameters within invocation of the Lambda perform.
    For the Input paths map:


    The expression extracts the instanceId from the function as resourceId, and the Account ID as account.

    For the industry Input template, enter the next:


You pass the real title of the SSM automation record you want to execute being an input parameter. In this full case, the record is AWS-TerminateEC2Example and the record input parameters certainly are a JSON structure named AutomationParameters.

You have created a fresh response action that’s prepared to test now.


The automation execution that occurs in the security account is documented in AWS Techniques Manager under Automation Execution. You can even make reference to the AWS Systems Manager automation documentation.

If Lambda functionality execution fails, a CloudWatch alarm is triggered after that, and notification is delivered to an SNS subject with prefix Safety_Alerts_[Your_Stack]. Lambda perform execution in the safety account is certainly documented in the CloudWatch log team for the Lambda functionality. You may use the logs to comprehend whether the Lambda perform was executed successfully.

Protection Hub integration

AWS Security Hub aggregates alarms from GuardDuty along with other providers. To get Safety Hub to aggregate alarms, you need to first activate Protection Hub before deploying this option. Then, invite your atmosphere accounts to the protection account as master. To learn more, see Master and member accounts in AWS Security Hub.

The integration with Safety Hub is for guide triggering and isn’t area of the automated response itself. This could be useful for those who have findings that want safety investigation before triggering the activity.

From the Security Hub console, choose the finding and utilize the Actions drop down menu in top of the right corner to trigger a reply, as shown in Figure 5. A CloudWatch occasion invokes the automated reaction, like a Lambda functionality to isolate an EC2 example. The Lambda perform fetches the GuardDuty obtaining from the provider account and executes among the following: Isolate EC2 Instance, Block Principal, or Send SNS Notification.

Only one resource could be selected for execution. In the event that you select multiple assets, the response isn’t executed, and the Lambda log information is proven in CloudWatch Logs.

Determine 5: Security Hub integrationBody 5: Security Hub integration

Price of the alternative

The cost of the answer depends upon the events generated in your AWS account and chosen AWS Area. Costs may be less than several U.S. monthly and per account bucks. It is possible to model your expenses by understanding GuardDuty pricing, automation pricing in AWS system manager, and AWS Config prices for 6 AWS Config rules. The expenses for AWS Lambda and Amazon CloudWatch are anticipated to become minimal and may be contained in your totally free tier. You may use Security Hub optionally; see AWS Security Hub pricing.


In this article, you learned how exactly to deploy an automated incident reaction framework using AWS native functions. It is possible to extend this framework to meet up your own future needs easily. In the event that you would further prefer to extend it, contact AWS professional services or an AWS partner. When you have technical queries, please utilize the Amazon GuardDuty or AWS Config forums. Keep in mind, this solution is an launch to automated security reaction and isn’t a comprehensive solution.

Should you have feedback concerning this post, submit remarks in the Comments section below.

Want a lot more AWS Security how-to articles, news, and show announcements? Stick to us on Twitter.


Vesselin Tzvetkov

Vesselin is senior protection consultant at AWS Expert Providers and is passionate regarding safety engineering and architecture innovative options. Outside of technologies, he likes classical songs, philosophy, and sports. A Ph is held by him.D. in protection from TU-Darmstadt and a M.S. in electric engineering from Bochum University in Germany.

%d bloggers like this: