██FR█████ █INTELL███████████

Terraform Fundamentals: CloudWatch Observability Access Manager


This content originally appeared on DEV Community and was authored by DevOps Fundamental

Managing Observability Access with Terraform: A Deep Dive into CloudWatch Observability Access Manager

The relentless growth of microservices and distributed systems demands robust observability. However, granting broad access to observability data – logs, metrics, traces – creates significant security and compliance risks. Traditionally, managing access to CloudWatch (or similar services on other clouds) involved complex IAM policies, often over-permissive and difficult to audit. This leads to a constant tension between enabling engineering velocity and maintaining a secure, compliant environment. CloudWatch Observability Access Manager (OAM) addresses this directly, and integrating it into your Terraform workflows is crucial for modern infrastructure automation. This isn’t a “nice-to-have”; it’s becoming a necessity for organizations operating at scale. OAM fits squarely within a platform engineering stack, providing a self-service layer for observability access, orchestrated through IaC pipelines.

What is CloudWatch Observability Access Manager in Terraform Context?

CloudWatch Observability Access Manager (OAM) allows you to define granular access control to CloudWatch observability data. Instead of relying solely on IAM, OAM introduces access grants which define who can access what data, for how long, and under what conditions. Terraform manages these grants through the aws_cloudwatch_log_subscription_filter resource, coupled with IAM policies that leverage OAM’s service-linked roles.

Currently, there isn’t a dedicated Terraform provider specifically for OAM. Instead, it’s managed through existing AWS resources, primarily focusing on log subscription filters and IAM. This means understanding how OAM interacts with these resources is key. A significant caveat is that OAM relies heavily on IAM permissions. Incorrectly configured IAM policies can render OAM ineffective. Lifecycle management is also important; grants have expiration dates, requiring automation to renew or update them.

Use Cases and When to Use

  1. Developer Self-Service: Empower developers to access logs for their specific services without requiring manual intervention from SREs. This accelerates debugging and reduces bottlenecks.
  2. Auditing and Compliance: Enforce strict access controls to meet regulatory requirements (e.g., PCI DSS, HIPAA). OAM provides a clear audit trail of who accessed what data.
  3. Incident Response: Grant temporary, elevated access to observability data for incident responders during critical events. This allows for faster root cause analysis.
  4. Third-Party Vendor Access: Provide limited, time-bound access to observability data for vendors supporting your applications.
  5. Multi-Tenant Environments: Isolate observability data between different teams or customers in a multi-tenant environment. This is critical for SaaS providers.

Key Terraform Resources

  1. aws_iam_role: Defines the IAM role that will receive the access grant. 
   resource "aws_iam_role" "oam_role" {
     name = "oam-developer-role"
     assume_role_policy = jsonencode({
       Version = "2012-10-17",
       Statement = [
         {
           Action = "sts:AssumeRole",
           Principal = {
             Service = "oam.amazonaws.com"
           },
           Effect = "Allow",
           Sid = ""
         }
       ]
     })
   }
  1. aws_iam_policy: Grants permissions to the role, including access to CloudWatch Logs. 
   resource "aws_iam_policy" "oam_policy" {
     name        = "oam-developer-policy"
     description = "Policy for OAM developer role"
     policy      = jsonencode({
       Version = "2012-10-17",
       Statement = [
         {
           Action = [
             "logs:DescribeLogGroups",
             "logs:GetLogEvents",
             "logs:FilterLogEvents"
           ],
           Effect   = "Allow",
           Resource = "*"
         }
       ]
     })
   }
  1. aws_iam_role_policy_attachment: Attaches the policy to the role. 
   resource "aws_iam_role_policy_attachment" "oam_attachment" {
     role       = aws_iam_role.oam_role.name
     policy_arn = aws_iam_policy.oam_policy.arn
   }
  1. aws_cloudwatch_log_group: The log group to which access will be granted. 
   resource "aws_cloudwatch_log_group" "example" {
     name              = "/aws/lambda/my-function"
     retention_in_days = 7
   }
  1. aws_cloudwatch_log_subscription_filter: The core resource for creating the OAM grant. 
   resource "aws_cloudwatch_log_subscription_filter" "oam_filter" {
     name            = "oam-filter"
     log_group_name  = aws_cloudwatch_log_group.example.name
     filter_pattern  = "" # Empty filter pattern grants access to all logs

     destination_arn = "arn:aws:logs:us-east-1:123456789012:log-group:oam-destination-group:*"
     role_arn        = aws_iam_role.oam_role.arn
   }
  1. aws_iam_service_linked_role: Ensures the necessary service-linked role for OAM exists. 
   resource "aws_iam_service_linked_role" "oam_role" {
     name = "aws-service-role/cloudwatch.amazonaws.com/ObservabilityAccessManager"
   }
  1. data.aws_iam_policy_document: Dynamically generate IAM policies. 
   data "aws_iam_policy_document" "oam_policy_doc" {
     statement {
       sid = "AllowOAMAccess"
       effect = "Allow"
       actions = ["logs:DescribeLogGroups", "logs:GetLogEvents", "logs:FilterLogEvents"]
       resources = ["*"]
     }
   }
  1. aws_iam_user_policy_attachment: Attach policies to IAM users. 
   resource "aws_iam_user_policy_attachment" "example" {
     user       = "example-user"
     policy_arn = aws_iam_policy.oam_policy.arn
   }

Common Patterns & Modules

Using for_each with aws_cloudwatch_log_subscription_filter allows you to create multiple grants for different log groups or roles. Dynamic blocks within aws_iam_policy can be used to customize permissions based on input variables. A monorepo structure is recommended for managing infrastructure code, allowing for clear separation of concerns and reusable modules. Layered modules (e.g., a core OAM module and environment-specific modules) promote consistency and reduce duplication. Public modules for OAM are currently limited, so building your own is often necessary.

Hands-On Tutorial

This example grants a developer role access to logs from a Lambda function.

Provider Setup: (Assume AWS provider is already configured)

Resource Configuration: (See code snippets above for aws_iam_role, aws_iam_policy, aws_cloudwatch_log_group, and aws_cloudwatch_log_subscription_filter)

Apply & Destroy:

terraform init
terraform plan
terraform apply
# ... (Confirm apply) ...

terraform destroy

terraform plan Output (Snippet):

# aws_cloudwatch_log_subscription_filter.oam_filter will create...
# aws_iam_role.oam_role will create...
# aws_iam_policy.oam_policy will create...
# ...

This example assumes the infrastructure is deployed as part of a CI/CD pipeline (e.g., GitHub Actions) triggered by a pull request.

Enterprise Considerations

Large organizations leverage Terraform Cloud/Enterprise for state locking, remote operations, and collaboration. Sentinel or Open Policy Agent (OPA) can enforce policy-as-code, ensuring OAM grants adhere to security standards. IAM design should follow the principle of least privilege, with granular roles and policies. State locking is critical to prevent concurrent modifications. Multi-region deployments require careful consideration of IAM roles and policies, ensuring consistency across regions. Costs are primarily driven by CloudWatch log ingestion and storage, but OAM itself has minimal direct cost.

Security and Compliance

Enforce least privilege by granting only the necessary permissions to each role. Use aws_iam_policy to define granular access controls. Implement drift detection to identify unauthorized changes to OAM grants. Tag resources consistently for auditing and cost allocation. Regularly review OAM grants to ensure they remain valid and compliant.

Integration with Other Services

  1. AWS Lambda: OAM grants access to Lambda function logs. 
   graph LR
       A[Lambda Function] --> B(CloudWatch Logs)
       B --> C{OAM Access Grants}
       C --> D[IAM Role]
  1. Amazon ECS: OAM grants access to container logs. 
   graph LR
       A[ECS Task] --> B(CloudWatch Logs)
       B --> C{OAM Access Grants}
       C --> D[IAM Role]
  1. Amazon EKS: OAM grants access to Kubernetes pod logs. 
   graph LR
       A[EKS Pod] --> B(CloudWatch Logs)
       B --> C{OAM Access Grants}
       C --> D[IAM Role]
  1. AWS Config: Monitor OAM grant configurations for compliance. 
   graph LR
       A[OAM Grants] --> B(AWS Config)
       B --> C{Compliance Rules}
  1. Amazon EventBridge: Trigger automated actions based on OAM grant events. 
   graph LR
       A[OAM Grant Changes] --> B(EventBridge)
       B --> C[Automated Actions]

Module Design Best Practices

Abstract OAM configuration into reusable modules with well-defined input variables (e.g., log group name, role ARN, grant duration). Use output variables to expose relevant information (e.g., grant ARN). Leverage locals to simplify complex expressions. Thoroughly document the module with examples and usage instructions. Use a remote backend (e.g., S3) for state storage.

CI/CD Automation 

# .github/workflows/oam.yml

name: OAM Deployment

on:
  push:
    branches:
      - main

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: hashicorp/setup-terraform@v2
      - run: terraform fmt
      - run: terraform validate
      - run: terraform plan -out=tfplan
      - run: terraform apply tfplan

Pitfalls & Troubleshooting

  1. IAM Permissions: Incorrect IAM policies prevent OAM from functioning. Solution: Double-check IAM permissions and ensure the service-linked role exists.
  2. Grant Expiration: Grants expire, requiring renewal. Solution: Automate grant renewal using Terraform or a scheduled task.
  3. Filter Pattern Issues: Incorrect filter patterns result in unintended access. Solution: Carefully test filter patterns before deploying.
  4. Destination ARN Errors: Invalid destination ARNs cause deployment failures. Solution: Verify the destination ARN is correct and accessible.
  5. State Corruption: Corrupted Terraform state leads to inconsistencies. Solution: Implement state locking and regular backups.
  6. Service Linked Role Not Created: OAM requires the service linked role to exist. Solution: Ensure the aws_iam_service_linked_role resource is applied before other resources.

Pros and Cons

Pros:

  • Granular access control to observability data.
  • Improved security and compliance.
  • Developer self-service capabilities.
  • Auditable access logs.

Cons:

  • Requires careful IAM configuration.
  • No dedicated Terraform provider.
  • Grant expiration requires automation.
  • Increased complexity compared to basic IAM policies.

Conclusion

CloudWatch Observability Access Manager, when integrated with Terraform, provides a powerful mechanism for managing access to observability data in a secure and scalable manner. It’s no longer sufficient to simply grant broad access to CloudWatch Logs; organizations must adopt a more granular, policy-driven approach. Start by implementing OAM in a proof-of-concept environment, evaluate existing Terraform modules, and establish a CI/CD pipeline for automated deployment and management. This investment will pay dividends in terms of reduced risk, improved compliance, and increased engineering velocity.


This content originally appeared on DEV Community and was authored by DevOps Fundamental