Manual Remediation Implementation Plan

Created: 2026-02-07 Relates to: Failed Controls Audit Status: Pending manual execution

These three items cannot be fully automated via CloudFormation and require manual CLI or console steps with human verification.

---

1. CloudFront Origin Failover (Origin Groups)

Control: CloudFront.10 - CloudFront distributions should use origin failover Risk: If the primary S3 origin becomes unavailable, CloudFront returns 5xx errors to users instead of failing over to a backup.

Affected Distributions

Distribution	Template	Primary Origin
docs.vell.ai	cloudfront-oac-docs.yaml	DocsS3Origin → vell-docs bucket
docs+templates	cloudfront-oac-docs-and-templates.yaml	DocsOrigin → docs bucket, TemplatesOrigin → templates bucket

Why This Is Manual

CloudFormation AWS::CloudFront::Distribution supports OriginGroups natively, but implementing it requires:

1. A replica S3 bucket in a second region with cross-region replication (CRR) enabled
2. A second OAC and bucket policy for the failover origin
3. Careful testing to avoid breaking existing routing (URL rewrite functions, /templates/* path pattern)

This should be done through a planned change window, not a blind IaC deploy.

Implementation Steps

Step 1: Create Replica Buckets

# Choose failover region (primary is likely us-east-1 or us-east-2)
FAILOVER_REGION="us-west-2"
PRIMARY_REGION="us-east-1"

# Create replica bucket for docs
aws s3api create-bucket \
  --bucket vell-docs-failover \
  --region $FAILOVER_REGION \
  --create-bucket-configuration LocationConstraint=$FAILOVER_REGION

# Block public access
aws s3api put-public-access-block \
  --bucket vell-docs-failover \
  --public-access-block-configuration \
    BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

# Enable versioning on BOTH buckets (required for CRR)
aws s3api put-bucket-versioning \
  --bucket vell-docs \
  --versioning-configuration Status=Enabled

aws s3api put-bucket-versioning \
  --bucket vell-docs-failover \
  --versioning-configuration Status=Enabled

Step 2: Create Replication IAM Role

# Create the role trust policy
cat > /tmp/replication-trust.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "Service": "s3.amazonaws.com" },
    "Action": "sts:AssumeRole"
  }]
}
EOF

aws iam create-role \
  --role-name s3-docs-replication-role \
  --assume-role-policy-document file:///tmp/replication-trust.json

# Attach replication permissions
cat > /tmp/replication-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetReplicationConfiguration",
        "s3:ListBucket"
      ],
      "Resource": "arn:aws:s3:::vell-docs"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObjectVersionForReplication",
        "s3:GetObjectVersionAcl",
        "s3:GetObjectVersionTagging"
      ],
      "Resource": "arn:aws:s3:::vell-docs/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:ReplicateObject",
        "s3:ReplicateDelete",
        "s3:ReplicateTags"
      ],
      "Resource": "arn:aws:s3:::vell-docs-failover/*"
    }
  ]
}
EOF

aws iam put-role-policy \
  --role-name s3-docs-replication-role \
  --policy-name S3DocsReplicationPolicy \
  --policy-document file:///tmp/replication-policy.json

Step 3: Enable Cross-Region Replication

ROLE_ARN=$(aws iam get-role --role-name s3-docs-replication-role --query 'Role.Arn' --output text)

cat > /tmp/replication-config.json << EOF
{
  "Role": "$ROLE_ARN",
  "Rules": [{
    "ID": "docs-failover-replication",
    "Status": "Enabled",
    "Priority": 1,
    "Filter": {},
    "Destination": {
      "Bucket": "arn:aws:s3:::vell-docs-failover",
      "StorageClass": "STANDARD"
    },
    "DeleteMarkerReplication": { "Status": "Enabled" }
  }]
}
EOF

aws s3api put-bucket-replication \
  --bucket vell-docs \
  --replication-configuration file:///tmp/replication-config.json

Step 4: Initial Sync (Existing Objects)

CRR only replicates new objects. Sync existing content:

aws s3 sync s3://vell-docs s3://vell-docs-failover --region $FAILOVER_REGION

Step 5: Create OAC for Failover Origin

aws cloudfront create-origin-access-control \
  --origin-access-control-config \
    Name=OAC-vell-docs-failover,\
    Description="OAC for docs failover bucket",\
    OriginAccessControlOriginType=s3,\
    SigningBehavior=always,\
    SigningProtocol=sigv4

Save the returned Id for the next step.

Step 6: Add Bucket Policy to Failover Bucket

DISTRIBUTION_ARN="arn:aws:cloudfront::ACCOUNT_ID:distribution/DISTRIBUTION_ID"

cat > /tmp/failover-policy.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowCloudFrontOAC",
      "Effect": "Allow",
      "Principal": { "Service": "cloudfront.amazonaws.com" },
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::vell-docs-failover/*",
      "Condition": {
        "StringEquals": {
          "AWS:SourceArn": "$DISTRIBUTION_ARN"
        }
      }
    },
    {
      "Sid": "DenyInsecureTransport",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::vell-docs-failover",
        "arn:aws:s3:::vell-docs-failover/*"
      ],
      "Condition": {
        "Bool": { "aws:SecureTransport": "false" }
      }
    }
  ]
}
EOF

aws s3api put-bucket-policy \
  --bucket vell-docs-failover \
  --policy file:///tmp/failover-policy.json

Step 7: Update Distribution with Origin Group

# Get current distribution config
DIST_ID="YOUR_DISTRIBUTION_ID"
aws cloudfront get-distribution-config --id $DIST_ID > /tmp/dist-config.json

# Extract the ETag (required for updates)
ETAG=$(jq -r '.ETag' /tmp/dist-config.json)

# Edit the DistributionConfig to add:
# 1. A second origin (FailoverDocsS3Origin) pointing to vell-docs-failover
# 2. An OriginGroup that wraps both origins
# 3. Update DefaultCacheBehavior to target the OriginGroup ID

# The key additions to DistributionConfig:

Add these to the distribution config JSON:

{
  "Origins": {
    "Items": [
      {
        "Id": "DocsS3Origin",
        "DomainName": "vell-docs.s3.us-east-1.amazonaws.com",
        "S3OriginConfig": { "OriginAccessIdentity": "" },
        "OriginAccessControlId": "PRIMARY_OAC_ID"
      },
      {
        "Id": "DocsS3FailoverOrigin",
        "DomainName": "vell-docs-failover.s3.us-west-2.amazonaws.com",
        "S3OriginConfig": { "OriginAccessIdentity": "" },
        "OriginAccessControlId": "FAILOVER_OAC_ID"
      }
    ]
  },
  "OriginGroups": {
    "Quantity": 1,
    "Items": [
      {
        "Id": "DocsOriginGroup",
        "FailoverCriteria": {
          "StatusCodes": {
            "Quantity": 4,
            "Items": [500, 502, 503, 504]
          }
        },
        "Members": {
          "Quantity": 2,
          "Items": [
            { "OriginId": "DocsS3Origin" },
            { "OriginId": "DocsS3FailoverOrigin" }
          ]
        }
      }
    ]
  }
}

Then update DefaultCacheBehavior.TargetOriginId to "DocsOriginGroup".

# Apply the updated config
aws cloudfront update-distribution \
  --id $DIST_ID \
  --if-match $ETAG \
  --distribution-config file:///tmp/updated-dist-config.json

Step 8: Verify

# Wait for distribution to deploy
aws cloudfront wait distribution-deployed --id $DIST_ID

# Test - should work normally
curl -I https://docs.vell.ai/

# Verify origin group is configured
aws cloudfront get-distribution --id $DIST_ID \
  --query 'Distribution.DistributionConfig.OriginGroups'

Rollback

If issues occur, revert DefaultCacheBehavior.TargetOriginId back to DocsS3Origin and remove the origin group. The failover origin and replication can remain in place without impact.

Post-Implementation

After manual verification, update the CloudFormation templates to include origin groups so future deployments maintain the configuration. Add the OriginGroups block to both cloudfront-oac-docs.yaml and cloudfront-oac-docs-and-templates.yaml.

---

2. S3 MFA Delete

Control: S3.6 - S3 general purpose bucket policies should restrict access to other AWS accounts Related: S3 bucket versioning with MFA delete protection Risk: Without MFA delete, a compromised IAM credential can permanently delete versioned objects or disable versioning.

Why This Is Manual

AWS requires MFA delete to be enabled using the root account credentials with a valid MFA token. It cannot be done via: - CloudFormation - IAM user credentials (even with admin) - Assumed roles

Scope

Apply to critical buckets only (MFA delete adds operational overhead for legitimate deletions):

Bucket	Purpose	Priority
CloudTrail logs bucket	Audit trail integrity	Critical
S3 access logs bucket	Security logging	Critical
Prod assets bucket	Production content	High
Backup/DR buckets	Data protection	High

Do NOT enable on frequently-modified buckets (build artifacts, temp storage, CDK staging).

Implementation Steps

Step 1: Identify Target Buckets

# List all buckets with versioning status
for bucket in $(aws s3api list-buckets --query 'Buckets[].Name' --output text); do
  versioning=$(aws s3api get-bucket-versioning --bucket $bucket --query 'Status' --output text 2>/dev/null)
  mfa_delete=$(aws s3api get-bucket-versioning --bucket $bucket --query 'MFADelete' --output text 2>/dev/null)
  echo "$bucket | Versioning: $versioning | MFA Delete: $mfa_delete"
done

Step 2: Ensure Versioning Is Enabled

MFA delete requires versioning. Enable it first if not already active:

aws s3api put-bucket-versioning \
  --bucket BUCKET_NAME \
  --versioning-configuration Status=Enabled

Step 3: Enable MFA Delete (Root Credentials Required)

This must be run as the root user with MFA:

# Sign in as root user and configure CLI with root access keys (temporary)
# Get a TOTP code from the root MFA device

ROOT_MFA_SERIAL="arn:aws:iam::ACCOUNT_ID:mfa/root-account-mfa-device"
MFA_CODE="123456"  # Current 6-digit code from your MFA device

aws s3api put-bucket-versioning \
  --bucket BUCKET_NAME \
  --versioning-configuration Status=Enabled,MFADelete=Enabled \
  --mfa "$ROOT_MFA_SERIAL $MFA_CODE"

Step 4: Repeat for Each Target Bucket

# Critical buckets - replace BUCKET names with actual values
BUCKETS=(
  "vellocity-cloudtrail-logs"
  "vellocity-s3-access-logs"
  "vellocity-prod-assets"
)

for bucket in "${BUCKETS[@]}"; do
  echo "Enabling MFA Delete on: $bucket"
  MFA_CODE="ENTER_CURRENT_CODE"  # Must be fresh for each call
  aws s3api put-bucket-versioning \
    --bucket $bucket \
    --versioning-configuration Status=Enabled,MFADelete=Enabled \
    --mfa "$ROOT_MFA_SERIAL $MFA_CODE"
  echo "Done: $bucket"
  sleep 2  # Wait for new TOTP rotation if needed
done

Step 5: Verify

for bucket in "${BUCKETS[@]}"; do
  echo -n "$bucket: "
  aws s3api get-bucket-versioning --bucket $bucket
done

Expected output per bucket:

{
  "Status": "Enabled",
  "MFADelete": "Enabled"
}

Step 6: Remove Root Access Keys

Immediately after completing MFA delete setup:

# List root access keys
aws iam list-access-keys

# Delete the temporary root access keys
aws iam delete-access-key --access-key-id AKIAXXXXXXXXXXXXXXXX

Verify in IAM console that root has no active access keys.

Operational Impact

With MFA delete enabled: - Deleting object versions requires MFA authentication - Disabling versioning requires MFA authentication - Normal PutObject, GetObject, DeleteObject (delete markers) work normally - CI/CD pipelines are not affected for normal operations - Only destructive versioning operations require the additional MFA step

Rollback

To disable MFA delete (also requires root + MFA):

aws s3api put-bucket-versioning \
  --bucket BUCKET_NAME \
  --versioning-configuration Status=Enabled,MFADelete=Disabled \
  --mfa "$ROOT_MFA_SERIAL $MFA_CODE"

---

3. IAM User Credential Review

Controls: - IAM.3 - Access keys should be rotated every 90 days - IAM.8 - Unused IAM user credentials should be removed - IAM.22 - IAM user credentials unused for 45 days should be removed

Risk: Stale credentials are a top attack vector. Unused access keys or passwords that haven't been rotated are targets for credential stuffing and lateral movement.

Implementation Steps

Step 1: Generate Credential Report

# Request the report
aws iam generate-credential-report

# Wait for it to complete (poll until "COMPLETE")
aws iam generate-credential-report --query 'State' --output text

# Download the report
aws iam get-credential-report \
  --query 'Content' \
  --output text | base64 -d > /tmp/iam-credential-report.csv

Step 2: Analyze the Report

# View the report
column -t -s',' /tmp/iam-credential-report.csv | less -S

# Key columns to review:
# - password_last_used
# - access_key_1_active / access_key_1_last_used_date / access_key_1_last_rotated
# - access_key_2_active / access_key_2_last_used_date / access_key_2_last_rotated
# - mfa_active

Quick analysis commands:

# Users with access keys older than 90 days
awk -F',' 'NR>1 {
  if ($9 == "true") {
    split($10, d, "T");
    print $1, "key1_rotated:", d[1]
  }
  if ($14 == "true") {
    split($15, d, "T");
    print $1, "key2_rotated:", d[1]
  }
}' /tmp/iam-credential-report.csv

# Users with passwords but no recent login (>45 days)
awk -F',' 'NR>1 && $4 == "true" {
  print $1, "password_last_used:", $5
}' /tmp/iam-credential-report.csv

# Users without MFA enabled
awk -F',' 'NR>1 && $4 == "true" && $8 == "false" {
  print $1, "HAS PASSWORD BUT NO MFA"
}' /tmp/iam-credential-report.csv

Step 3: Categorize Users

Sort findings into action categories:

Category	Criteria	Action
Deactivate immediately	Access key unused >90 days AND user inactive >45 days	Disable key, disable console access
Rotate keys	Access key active but >90 days old	Create new key, update applications, delete old key
Enforce MFA	Console password active, MFA disabled	Contact user, set deadline for MFA enrollment
Review purpose	Service accounts with console passwords	Remove password, keep only programmatic access
No action	Keys <90 days, MFA enabled, recently active	Document as compliant

Step 4: Deactivate Stale Keys

# Deactivate an access key (non-destructive, can be re-enabled)
aws iam update-access-key \
  --user-name USERNAME \
  --access-key-id AKIAXXXXXXXX \
  --status Inactive

# Wait 7 days, then delete if no complaints
aws iam delete-access-key \
  --user-name USERNAME \
  --access-key-id AKIAXXXXXXXX

Step 5: Rotate Active Keys

For each user with keys >90 days old that are still in use:

# 1. Create new key
aws iam create-access-key --user-name USERNAME

# 2. Communicate new credentials to the user/service owner securely
#    (never send keys over email or chat)

# 3. Set a deadline (e.g., 14 days) for migration

# 4. After confirmation, deactivate old key
aws iam update-access-key \
  --user-name USERNAME \
  --access-key-id OLD_KEY_ID \
  --status Inactive

# 5. After 7 more days with no issues, delete old key
aws iam delete-access-key \
  --user-name USERNAME \
  --access-key-id OLD_KEY_ID

Step 6: Remove Unused Console Passwords

# For service accounts that should only have programmatic access
aws iam delete-login-profile --user-name SERVICE_ACCOUNT_NAME

Step 7: Enforce MFA

For users with console access but no MFA:

# Option A: Attach MFA enforcement policy
# This policy denies all actions except MFA self-management until MFA is configured
cat > /tmp/enforce-mfa-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowSelfMFAManagement",
      "Effect": "Allow",
      "Action": [
        "iam:CreateVirtualMFADevice",
        "iam:EnableMFADevice",
        "iam:ResyncMFADevice",
        "iam:ListMFADevices",
        "iam:GetUser",
        "iam:ChangePassword"
      ],
      "Resource": [
        "arn:aws:iam::*:mfa/${aws:username}",
        "arn:aws:iam::*:user/${aws:username}"
      ]
    },
    {
      "Sid": "DenyAllExceptMFASetupWithoutMFA",
      "Effect": "Deny",
      "NotAction": [
        "iam:CreateVirtualMFADevice",
        "iam:EnableMFADevice",
        "iam:ListMFADevices",
        "iam:GetUser",
        "iam:ChangePassword",
        "iam:ResyncMFADevice",
        "sts:GetSessionToken"
      ],
      "Resource": "*",
      "Condition": {
        "BoolIfExists": { "aws:MultiFactorAuthPresent": "false" }
      }
    }
  ]
}
EOF

aws iam put-user-policy \
  --user-name USERNAME \
  --policy-name EnforceMFA \
  --policy-document file:///tmp/enforce-mfa-policy.json

Step 8: Document Results

After completing the review, record:

# Re-generate credential report to confirm compliance
aws iam generate-credential-report
sleep 10
aws iam get-credential-report \
  --query 'Content' \
  --output text | base64 -d > /tmp/iam-credential-report-AFTER.csv

# Compare before/after
diff /tmp/iam-credential-report.csv /tmp/iam-credential-report-AFTER.csv

Ongoing Process

Set up automated monitoring so this doesn't require manual review again:

1. AWS Config Rule access-keys-rotated (maxAccessKeyAge: 90) - already available as a managed rule
2. AWS Config Rule iam-user-unused-credentials-check (maxCredentialUsageAge: 45)
3. SNS Notifications - Alert on non-compliant findings from Config
4. Quarterly review - Calendar reminder to pull and review credential reports

These Config rules are already deployable via the existing baseline-security.yml template if the AWS Config recorder is active.

---

Execution Schedule

Task	Requires	Estimated Window	Dependencies
IAM Credential Review	IAM admin access	Off-hours (no downtime)	None
S3 MFA Delete	Root account + MFA device	Off-hours (no downtime)	Identify target buckets
CloudFront Origin Failover	IAM admin + change window	Maintenance window (Sun 4AM UTC)	Replica buckets synced

Recommended Order

1. IAM Credential Review - No infrastructure changes, reduces attack surface immediately
2. S3 MFA Delete - Protects audit logs and critical data, no service impact
3. CloudFront Origin Failover - Requires most setup (CRR, OAC, testing), schedule during maintenance window

---

Verification Checklist

After completing all three items, verify in Security Hub:

• CloudFront.10 - Origin failover finding resolved
• S3.6 - MFA delete findings resolved (if applicable)
• IAM.3 - Access key rotation findings resolved
• IAM.8 / IAM.22 - Unused credential findings resolved
• No new findings introduced by changes

Findings refresh within 24-48 hours of remediation.