· 13 min read

What Would You Cut If Your AWS CI Role Was Compromised?

awssecurityiamincident-responsegithub-actions
Mariusz Gebala AWS SA, Azure Admin, Palo Alto PCNSA - cloud-audit author

TL;DR: Every scanner on the market tells you the attack path exists. Almost none of them tell you what to do in the ninety seconds after it gets used. This post is about the gap between “here is how someone becomes admin” and “here is the one thing to cut, right now, that stops it without taking down production.” That second thing is called containment, and it is strangely missing from our tooling.


It is 2:14 in the morning. GuardDuty just paged you. Your GitHub Actions deployment role, the one that pushes to production, the one with the comfortable pile of permissions nobody has audited since 2023, is making API calls from an IP address in a country your company does not operate in.

You are now the incident commander. Congratulations.

Here is the part nobody warns you about. You already know you have a problem. What you do not know, sitting there in the dark, is which single command to type to make the bleeding stop without also taking down the checkout service and waking up the entire on-call rotation standing behind you.

That question, what do I cut, turns out to be one of the least tooled problems in cloud security. We have spent a decade building gorgeous machinery to tell you the door is unlocked. We built almost nothing to help you decide which door to slam when someone finally walks through it.

Your CI role is the identity you trust most and audit least

Let us be honest about what a CI/CD role actually is. It is the single most powerful non-human identity in most AWS accounts, and it is also the one people think about the least. It has to be powerful, because it deploys everything: Lambda functions, ECS services, IAM roles, infrastructure. And it is trusted implicitly, because it is “just the pipeline.”

I wrote recently about how one poisoned npm package led to full AWS admin in about 72 hours through a GitHub Actions OIDC role. The uncomfortable takeaway from that story was not the npm package. It was that the CI role sat there, over-permissioned and quietly trusted, as the perfect pivot point. Steal the pipeline’s identity and you inherit its blast radius, and its blast radius is usually “most of the account.”

So when I say your CI role gets compromised, I am not describing an exotic edge case. I am describing the most common serious cloud incident of the last two years, dressed in slightly different clothes each time.

Detection tells you the door is open. It does not tell you which door to slam.

Here is where our industry has a blind spot the size of a data center.

We are very good at the first half of the problem. Scanners find the misconfiguration. Attack-path tools, cloud-audit included, chain individual findings into “here is how an attacker walks from this role to your database.” CNAPPs draw you a lovely graph. All of this is prevention and detection, and all of it is genuinely useful before anything goes wrong.

But detection and containment are different jobs. Detection answers how could someone get in. Containment answers a question that only exists after the alarm goes off: which one change cuts the path right now, and what does it break when I make it.

Notice that the second question has a constraint the first one does not: you cannot break production. During an incident you are not optimizing for a clean security posture, you are optimizing for stopping the attacker while keeping the lights on for everyone who is not attacking you. That constraint changes everything, and almost no tool models it.

Let us make it concrete: from CI role to your database password

Abstract “attack paths” are easy to nod along to and hard to act on, so here is a specific one that shows up constantly.

Your CI role can pass roles (iam:PassRole) because it deploys Lambdas, and it can create and invoke functions (lambda:CreateFunction, lambda:InvokeFunction). Somewhere in the account there is a Lambda execution role that can read the production database secret (secretsmanager:GetSecretValue, plus kms:Decrypt if that secret is encrypted with a customer managed key).

An attacker holding the CI role does not need to escalate cleverly. They write a five-line function, pass it that execution role, invoke it, and the function reads the secret and mails it home. This is one of the classic privilege escalation methods catalogued by Rhino Security Labs: a principal with iam:PassRole, lambda:CreateFunction, and lambda:InvokeFunction inherits the permissions of any Lambda role in the account, which “could range from no privilege escalation to full administrator access.”

One honest nuance, because it matters for what you cut later: creating the function is not the same as running it. PassRole plus CreateFunction gives you a loaded gun; you still need InvokeFunction, or an existing function you can overwrite and something that already triggers it, to pull the trigger. People who skip that detail end up cutting the wrong thing. Hold that thought.

The 90-second question: which brake do you pull?

You have confirmed the path. The CI role can reach the production DB secret through a Lambda role. Now you have to actually stop it. You have at least three candidate brakes, and here is the cruel part: each one is a different bet about your own environment.

Brake A: deny iam:PassRole for that execution role. This cuts the lateral hop. It is beautifully surgical, and here is the counterintuitive bit: it probably does not touch production, because production workloads do not call PassRole at steady-state runtime. PassRole happens at deploy time. So this brake mostly bites your deployment pipeline, which is already compromised and which you probably want frozen anyway.

Brake B: deny lambda:CreateFunction and lambda:UpdateFunctionCode. This cuts the code-execution primitive itself. It is also, unfortunately, precisely what your deployment pipeline does for a living. Pull this brake and you have frozen deploys. During an active incident that is usually an acceptable price. On a Tuesday it would get you yelled at.

Brake C: deny secretsmanager:GetSecretValue on that specific secret. This shields the crown jewel directly and it feels like the strongest move. It might also be the one that takes down production. Plenty of legitimate services read their database credentials from Secrets Manager at runtime, often on cold start. Deny that call account-wide and you have handed yourself a second outage on top of the incident. The strongest-looking brake is frequently the most dangerous.

Three brakes, three completely different blast radii on your own legitimate operations. And here is the question that should be keeping you up at night even when GuardDuty is quiet: which one is safe to pull?

You cannot answer that from the policy.

Why you cannot just call a deny “safe”

A policy document tells you what is possible. It says nothing about what is used. Reading the IAM policy, you can see that fourteen principals are allowed to call GetSecretValue on that secret. You have no idea whether one of them, or none of them, actually does.

The only place that truth lives is the log. Replay the last 30, 60, or 90 days of CloudTrail and ask a blunt question for each brake: did any legitimate principal successfully make the call I am about to deny? If, across 90 days, not one legitimate workload ever read that secret, then denying it has no observed impact.

But watch the language, because this is where honesty separates a real tool from a dangerous one. The verdict is NO_OBSERVED_IMPACT, and NO_OBSERVED_IMPACT is not the same as SAFE. Absence of evidence is not evidence of absence. Maybe the batch job that reads the secret runs on the first of the month and you replayed the wrong window. Maybe your trail has a gap. Maybe the call you care about is a data-plane operation that CloudTrail does not even log by default, which is a whole trap of its own: management events are logged out of the box, data events are not.

So the honest output of a containment check is never a green checkmark. It is a spectrum: NO_OBSERVED_IMPACT, KNOWN_IMPACT (someone legitimate used this last week, do not pull it lightly), or INSUFFICIENT_TELEMETRY (I genuinely cannot see enough to tell you, proceed with your eyes open). A tool that prints “SAFE” over an incident decision is lying to you with confidence, which is the worst way to be lied to.

This is also why you should prefer control-plane brakes over data-plane guesses when you can. iam:PassRole is a management event; it is always in CloudTrail, so you can reason about it with real evidence. Denying a data-plane read you cannot fully see is a bet, not a plan.

The gotcha that ruins your night: the deny might not do anything yet

Say you get it perfectly right. You pick the brake, you check the trail, you apply the deny. The attacker keeps working.

This is the detail that separates people who have read about incident response from people who have done it at 2am, and I have done it at 2am, and I did not enjoy it. Changing or removing a role’s permissions only affects future authorization decisions. The temporary credentials the attacker already holds keep working until they expire. AWS says this plainly in its own documentation on revoking role sessions: when credentials are exposed to an unauthorized third party, “that party has access for the duration of the session.”

To actually cut a live session you have to revoke it, not just re-permission it. AWS does this by attaching an AWSRevokeOlderSessions inline policy, or one with an aws:TokenIssueTime condition, that denies everything issued before a chosen moment. It works. It is also a sledgehammer, because, in AWS’s own words, it “applies to all users of the specified role, not just” the attacker. Every legitimate automation running under that role goes down with the intruder. And because IAM is eventually consistent, even the revoke has a propagation window; AWS builds in roughly 30 seconds of future buffer specifically to deal with that lag.

So the two things you most want during an incident, evict the attacker immediately and do not break my legitimate automation, are in direct tension, and you are being asked to choose between them in the dark, quickly, correctly. Fun.

The nuclear option exists. It is just nuclear.

To be fair, there is tooling for the “burn it down” end of this spectrum, and it is good. AWS Kill Switch, built by Jeffrey Lyon, formerly of Robinhood, is an open-source Lambda you deploy in a security account so an engineer can lock things down fast during an incident. Its verbs are exactly what you would want when things are truly on fire: apply an SCP to freeze an entire account, detach and delete a role’s policies, revoke all of a role’s sessions, or delete the role outright.

The tool is honest about what it is. Its README warns that the actions are “one-way operations,” that the default SCP “may break your application,” and that whatever you freeze or delete stays that way until a human manually reverses it. This is the fire-suppression system that floods the whole server room. Sometimes flooding the server room is exactly right. But it is a sledgehammer, and the space between “one scanner finding on a dashboard” and “flood the entire building” is almost completely empty.

That empty space is the interesting part. The precomputed, minimal, reversible, evidence-backed brake for this specific crown jewel basically does not exist as tooling. Everyone sells you the smoke detector. A couple of vendors sell you the flood system. Nobody hands you the labeled valve.

What I am building toward, and where you come in

Let me be precise about what is real and what is not, because I would rather lose your interest than your trust.

Today, cloud-audit maps attack chains, detects IAM privilege escalation paths, and computes blast radius. It is genuinely good at showing you that the door is open and how far in someone can walk. That part ships and works.

The piece I keep wishing existed while doing this on real client accounts is the containment layer. For each crown jewel, precompute the two or three candidate brakes. Replay CloudTrail to show what each one would have blocked over the last 90 days, using the honest NO_OBSERVED_IMPACT / KNOWN_IMPACT / INSUFFICIENT_TELEMETRY language instead of a comforting green check. Hand you the deny statement and the rollback in the same breath. Run entirely on your machine, so nothing about your crown jewels leaves your laptop.

That layer is not in the tool yet. This article is me thinking out loud in public and checking whether the thing in my head is as useful as I believe it is, or whether I have quietly talked myself into an elegant answer to a question that nobody actually asks at 2am.

So I am asking. I am looking for a small handful of teams, three to start, running GitHub Actions into AWS, who would let me walk one real scenario with them. This is not a sales call and it needs no account access to begin. It is one honest conversation: if your CI role got popped tomorrow, what would you cut, and are you sure it would not break something? If your answer is a confident shrug, you are exactly the person I want to talk to. If your answer is a specific runbook, I want to talk to you even more, because I want to know what I am missing.

You can reach me at kontakt@haitmg.pl or through the cloud-audit support page. Tell me your worst crown jewel and your least favorite brake.

The one line to remember

Detection is a smoke alarm. Everybody bought one, they beep reliably, and they have genuinely saved lives. But almost nobody has walked their own building and decided, in advance and in writing, which valve to turn before the room fills with smoke.

That walk is boring. Nobody gets promoted for it. It is also, on the night it matters, the entire difference between a bad evening and a bad quarter.


Sources: AWS - Revoke IAM role temporary security credentials, AWS - IAM policy evaluation logic, Rhino Security Labs - AWS IAM Privilege Escalation, AWS Kill Switch (secengjeff), Help Net Security - AWS Kill Switch.


Related reading: The GitHub Actions OIDC mistake that backdoors your AWS, how the major AWS security scanners compare, and 64 IAM privilege escalation paths, benchmarked across five tools.

Book 20-min AWS triage