· 12 min read

AWS Network Firewall blocked 0.59% of exploits in independent testing - what this means for your cloud

awssecuritypalo-altocloud-architecture
HAIT Cloud & DevOps Consulting

In the spring of 2025, the results of a test comparing cloud firewalls were published on the CyberRatings.org laboratory website. Ten providers were included in the test. The AWS firewall blocked 0.59% of exploits.

When additional bypass tests were applied, the effectiveness dropped to 0%.

In my DevOps career, I have implemented both native AWS firewalls and those from Palo Alto (VM-Series and CNGFW). To this day, some customers still use the AWS firewall. This article is not a criticism; it is a realistic and objective (at least I hope so) look at what these numbers actually mean, what you should do with them, and what to keep in mind if you use the AWS Network Firewall.

Three rounds of testing, same result

First and foremost, it’s worth noting: this wasn’t a one-time test. CyberRatings tested the AWS firewall three times:

April 2024 - 11 vendors were tested for 984 exploits and 1,645 bypasses. AWS scored 5.39% security effectiveness - the lowest among all tested vendors. The rating was “Caution” (6 vendors received “Recommended,” 1 “Neutral,” and 4 “Caution”).

November 2024 - Minitest. Only native AWS, Azure, and GCP firewalls were considered in the test. AWS achieved a result of 0.38% of blocked exploits (2 out of 522). Their counterparts - Azure 24.14%, GCP 50.57%. Keysight CyPerf v5.0 was used as the test platform.

April 2025 - Firewall Comparison Report for Q1 2025. Ten vendors were tested for 2,028 exploits and 2,500 attacks using 27 techniques. AWS (horror of horrors) 0.59%. After security bypass tests, 0%. For comparison - the largest vendors (Check Point, Fortinet, Juniper, Palo Alto Networks, Versa) from 99.61% to 100%. The differences are dramatic.

Wait - 0% doesn’t mean “does nothing”

And here’s a moment to pause - 0% doesn’t mean the firewall is doing nothing. Before you give up on your AWS firewall, let me explain what these results really mean.

CyberRatings tests exploits and resilience against signature-based vulnerabilities (CVEs) from the last 10 years, which use various techniques to bypass security at layers 3, 4, and 7 of the OSI model. The key is that AWS Firewall wasn’t designed as an IPS/IDS system in the traditional sense. It’s a resource managed by Suricata with a specific set of features (domain filtering, IP/port rules, and POSSIBLY threat signature management). Its primary purpose is network segmentation and traffic filtering - NOT catching exploits based on the CVE database.

The problem is that AWS sells its firewalls with advertising phrases like “intrusion prevention” and “threat detection.” Well, since you’re paying for IPS rule groups and just over half a percent of exploits are detected, it’s probably a problem, regardless of the design assumptions.

Why such low results? Here’s a technical explanation.

In my career, I’ve worked with both Suricata firewalls and those supporting App-ID. The difference stems from the underlying architecture.

Suricata in AWS NFW

AWS runs Suricata in the background. Suricata is an open-source IPS system, but AWS NFW does not support all of its functionality:

  • Lua scripts - most advanced Suricata rules use them for complex detection logic
  • File extraction - no downloading for analysis
  • iprep - no IP scoring
  • Datasets/datarep - no custom data matching
  • IKEv2 and IP-in-IP protocol detection
  • pcre is limited to working only with content, tls.sni, http.host, and dns.query

This shouldn’t be underestimated. Lua scripts alone significantly enhance Suricata’s detection. Without them, you’re only using a fraction of its capabilities.

Stateful and Stateless Switching Problem

This is the technical cause of the problem identified by CyberRatings.

AWS NFW processes traffic in two stages. First, through the stateless engine (5-tuple matching), and then optionally through the stateful engine (Suricata). Unfortunately, stateless rules have a higher priority and typically interfere with stateful inspection.

Nevertheless, CyberRatings documented that they followed AWS documentation. Furthermore, they hired a certified AWS consultant to configure the firewall and worked directly with AWS engineers. Despite this, they found erroneous switching between engines. AWS Best Practice currently recommends setting the default stateless action to “Forward to Stateful Rule Groups” and completely avoiding configuring stateless rules. Simply put, it’s killing half the engine because it’s not cooperating with the other half.

Evasion is the real killer

The exploit’s 0.59% score is bad. The 0% evasion score is even worse.

CyberRatings tested 2,500 attacks using 27 bypass techniques at Layers 3, 4, and 7. When a firewall fails to handle evasion at lower layers, the score drops dramatically - even if it detects several exploits at first glance. AWS NFW failed bypass tests so badly that it nullified several detected exploits.

AWS has not publicly commented on or disputed the CyberRatings test results.

For context: bypass techniques include things like IP fragmentation, TCP segmentation, and protocol-level obfuscation. These are standard techniques used daily by penetration testers and real attackers. A production-grade firewall must be able to handle them.

It’s not just AWS - all three hyperscalers failed

Cloud ProviderExploit Block RateOverall (after evasions)
AWS Network Firewall0.59%0%
Azure Firewall55.28%0%
GCP Cloud Firewall96.60%0%
Third-party average (5 vendors)99.61-100%99.61-100%

GCP detected most of the exploits. But what good is that if bypass tests also yielded 0%? Azure is even worse than GCP.

So what’s the conclusion? AWS isn’t terrible. The problem is that cloud-native firewalls aren’t designed as next-generation firewalls (NGFWs). As SDxCentral put it, “first-class cybersecurity firewall services aren’t the highest priority for hyperscale cloud providers, whose first orders of business are to store and distribute data and not lose it.”

Let’s be honest about this. AWS provides the infrastructure, and they sell security features as a bonus. Companies like Palo Alto Networks, Fortinet, and Check Point are strictly security-focused. The priorities are different, and therefore, the results are different.

What are CyberRatings anyway? Is it worth paying attention to them?

This is, contrary to appearances, a very crucial question. Or maybe they’re deliberately trying to make AWS look bad? Let’s examine their credibility.

CyberRatings.org is a non-profit organization founded in November 2020 by Vikram Phatak. Phatak founded NSS Labs in 2007 and managed it for over a decade - NSS Labs was the industry standard for independent firewall testing before its closure. CyberRatings is currently working with the revived NSS Labs as an official testing partner.

A few controversies: NSS Labs was involved in a legal dispute with CrowdStrike from 2017 to 2018, in which it admitted to “inaccurate” testing of the CrowdStrike Falcon endpoint product and subsequently filed an antitrust lawsuit against CrowdStrike, AMTSO, and several other vendors. The CrowdStrike portion was settled confidentially in 2019; the broader antitrust claims were dismissed by the court. This is a significant story and worth covering.

CyberRatings’ AWS NFW testing was self-funded, with no vendor involvement. It used the industry-standard methodology (Keysight CyPerf), and the results were consistent across three separate rounds of testing over a 12-month period. The fact that the company hired a certified AWS consultant and worked directly with AWS engineers makes the “misconfiguration” argument difficult to sustain.

Getting back to AWS Firewall… So what is it really good at?

Despite these test results, I still recommend AWS Network Firewall to clients. Here’s why.

Domain-based outbound filtering. If you want to control which domains your workloads can reach, NFW does it well - especially with TLS inspection enabled to prevent SNI spoofing.

Network segmentation. VPC-to-VPC traffic control via Transit Gateway. IP and port-based rules. Basic allow/deny logic. This is exactly what most teams use it for.

Centralized logging. Full visibility of network flows thanks to native CloudWatch and S3 integration.

Zero operational overhead. No patching, no sizing, no HA configuration. It just works.

Compliance checkbox. For many compliance frameworks, having a firewall with logging and rules is a requirement - not a 99% score in an independent IPS test.

Cost. At $0.395/hour per endpoint (or $0.489 with TLS inspection), it’s four times cheaper than using Palo Alto VM-Series. And as of February 2026, AWS has eliminated additional data processing fees for advanced inspection.

These are real benefits. For a startup focused on basic outbound filtering or an internal application behind a transit gateway, NFW is the right tool. Just don’t confuse it with an IPS system.

So what do third-party firewalls do differently?

The difference between 0.59% and 99.61% isn’t due to budget or effort. It’s about the architectural approach.

App-ID vs. signatures. Palo Alto’s App-ID classifies traffic based on application behavior - payload inspection, behavioral patterns, protocol decoding - regardless of port. The AWS firewall classifies traffic based on port, protocol, and Suricata signatures. These are fundamentally different approaches. App-ID can distinguish legitimate HTTPS from reverse-shell tunneled protocols on port 443. To Suricata, both appear as TLS on port 443.

Full bypass support. Third-party firewalls reassemble fragmented packets, normalize protocols, and handle TCP segmentation before applying detection rules. This is why they withstand bypass tests. This is computationally expensive and causes delays, but that’s why you pay $3,000 per month instead of $750.

Continuous signature updates. Palo Alto’s threat intelligence team updates signatures daily, addressing new CVEs. AWS managed rule groups update less frequently and include fewer signatures.

One note: third-party firewalls aren’t perfect. For example, in CyberRatings’ Q4 2025 test, Palo Alto Networks’ PA-1410 firewall initially scored 0% in bypass resistance and a mere 46.37% in overall score. However, (credit where credit is due) after updating the Palo Alto operating system to version 11.2.10, its resistance increased to 100% and its overall score to 96.07%. The conclusion is simple: even companies dedicated to security must update their operating systems, and no vendor is always immune to vulnerabilities.

You’ve confused me… So what should I do?

If you’re using AWS Network Firewall, here are my recommendations:

1. Understand what you’re actually getting

NFW is a managed traffic filtering service. It’s great for domain allow/deny lists, IP rules, and network segmentation. It’s not an IPS that will detect known exploits. Adjust your expectations and security architecture accordingly.

2. Enable TLS Inspection

If you haven’t already, enable TLS inspection. This costs an additional $0.094/hour per endpoint, but it minimizes the SNI bypass vulnerability and provides real visibility into encrypted traffic.

3. Block QUIC

AWS NFW can’t inspect QUIC traffic. HTTP/3 relies on QUIC. Add a stateful rule to block UDP/443 and force clients to fall back to TCP/TLS, where the firewall can actually monitor traffic.

4. Don’t use stateless rules

Follow AWS best practices: set the default stateless action to “Forward to stateful rule groups.” Don’t configure stateless rules - they can interfere with stateful inspection. This is what CyberRatings says breaks the engine.

5. Layered Security

NFW shouldn’t be the only layer of security. Combine this with:

  • GuardDuty for threat detection (behavioral analysis, not signatures)
  • Security Hub for posture management
  • WAF against public applications
  • VPC endpoint policies to restrict access to services
  • SCPs to prevent misconfigurations at the organizational level

6. If you need a true IPS system, deploy an external NGFW

For regulated industries (PCI-DSS, HIPAA, SOX), environments processing sensitive data, or organizations with active threat models involving targeted attacks, consider deploying an external NGFW on AWS. Check Point, Fortinet, Juniper, Palo Alto, and Versa all achieved scores of 99.61-100% in the same test, running on the same AWS infrastructure.

Interestingly, this doesn’t mean you have to abandon NFW. I’ve actually seen many environments using both solutions. NFW for broad traffic filtering and an external NGFW provider in a centralized VPC for in-depth inspection.

The real question no one asks

The CyberRatings test measures how well a firewall detects known exploits and resists bypassing them. That matters. But it’s not the whole picture.

Most AWS security incidents I’ve seen weren’t caused by an attacker exploiting a CVE through the firewall. They were caused by:

  • Overly permissive IAM policies
  • S3 buckets with public access
  • Security groups open to everyone
  • Access keys that haven’t been rotated in 900 days

The 17 checks I perform during every AWS audit reveal more real risk than any firewall result. A team that fixes these fundamental issues and runs AWS NFW with TLS inspection is more secure than a team that deploys Palo Alto VM-Series but leaves its root account without MFA.

Security is a matter of layers. Firewalls are one layer. Don’t let the test result make you forget about the others.

Sources

Running AWS Network Firewall and want to understand your actual security posture? I audit cloud infrastructure for a living - from firewall rules to IAM policies to network architecture. Let’s talk.

Related posts

· 10 min read

AWS Network Firewall vs Palo Alto VM-Series - what I learned after deploying both in production

· 9 min read

17 AWS security issues I spot in almost every infrastructure audit

← Back to all posts