Introduction
I recently had an interesting conversation with a CTO of a company that was looking to get its SaaS application reviewed externally. As he felt their security maturity had improved internally over the past few years, he believed they were ready for an external web application penetration test (pentest).
Exciting times.
While exploring quotes, he was shocked at the pricing range he was seeing. One MSP quoted him under $1,000. A few other firms had quotes under $5,000. And yet several others were over five figures.
He was concerned. Why so extensive a range to hire an external security testing team?
In this article, I want to share with you what I told him. It all comes down to that adage that if you pay peanuts, you get monkeys.
But I am getting ahead of myself. Before we can even discuss the real costs of an application pentest, we need to level set what penetration testing is about and how apps and APIs differ in many ways from traditional network-based pentests that most firms offer when looking for security weaknesses.
So let’s dive right in.
What Is A Penetration Test?
So before we can get too deep into talking about pentesting, there are a few things we need to clarify because people usually confuse them.
There are some terms I’ll bet you’ve seen used interchangeably when discussing ethical hacking and finding exploitable vulnerabilities that shouldn’t be that way. The terms? Vulnerability Scanning, Vulnerability Assessment, and Penetration Testing.
Vulnerability Scanning
Vulnerability Scanning uses automated penetration testing tools to identify potential security vulnerabilities in your apps and infrastructure. Depending on the type of engagement, you may or may not have access to the source code, infrastructure, or app configuration, which may limit how deep automation tools can scan.
Vulnerability Assessment
Vulnerability Assessment takes the process a step further and has security professionals evaluate the impact of those identified issues. Besides vulnerability identification, ethical hackers may conduct security tests to infiltrate computer systems and highlight missing security measures. It includes things like risk ratings on security vulnerabilities found, explanations of false positives and negatives detected, and potential exploit scenarios.
Penetration Testing
Penetration Testing is a form of security testing that goes beyond vulnerability scanning and assessment. It allows for realistic simulated attacks mimicking external actors (or cybercriminals) who are looking to gain access to sensitive data or resources through security flaws. While some automated testing tools may be used during the engagement, penetration tests are far more methodical and manual. Using the same tools to show the effects of real world attacks, penetration testers can showcase a system’s vulnerabilities and the organization’s response capabilities.
To further complicate things, there are typically two kinds of penetration testing:
- Network Penetration Testing focuses on testing the security posture of the design, implementation, and maintenance of the network infrastructure, the hosts connected to it, and the services that run on top of it. During this engagement, ethical hackers are looking for unpatched, misconfigured, and/or vulnerable devices and services which may allow them to gain access to the infrastructure. This may include mobile devices along with maintaining access to any given computer system that may give them further access to public and private sources.
- Application Penetration Testing focuses more on the software, data, and security surrounding them, such as coding flaws and insecure use of resources and data. This usually includes targeted testing to exploit how the application works to leak sensitive information, abuse business logic, and take over the systems and services that host the application. This form of penetration testing isn’t just looking for security weaknesses in the infrastructure but the security posture of the applications themselves.
It is not uncommon for an application penetration test to be done after a network penetration test finds applications on hosts on a network that are within the scope of an engagement. But it is an entirely different set of skills required to conduct each kind of penetration test.
For the rest of this article, I will only focus on the discussion of application penetration testing. Not that network penetration testing isn’t important, but because web app and API pentests are approached entirely differently. So when I say pentesting, I’m talking about testing the security of the web apps and APIs and the direct infrastructure used to run them.
Why are Application Pentests Important?
So why is a penetration test important? Well, it gives you a realistic view of how your application would stand up if it were attacked by malicious actors. This is invaluable information for any organization looking to protect its sensitive data and resources.
Penetration testing also allows organizations to identify security weaknesses in their applications before those issues can be exploited. This can help the organization prioritize its security efforts to patch detected vulnerabilities and ensure they are investing in the correct type of security controls for its applications.
Finally, ethical hacking during a pentest can be used as a baseline for post-development internal testing, allowing organizations to ensure that any changes made to their applications won’t introduce new vulnerabilities or aggravate existing ones.
API Pentesting: A Different Beast
Now that we have a basic understanding of what penetration testing is let’s look at how it applies to APIs.
Unlike traditional web applications, APIs are typically decoupled from the frontend user interface and can be interacted with directly. This means it may be possible to bypass both client and server-side security controls (i.e., bypass authentication, authorization, or input validation tests) and interact directly with the API in unexpected ways.
Because of this, API pentests require a different set of skills and testing tools than regular application pentests. For example, API pentesters need to be familiar with not only HTTP protocols and Javascript, but the various standards used in API development, such as SOAP and RESTful APIs, Swagger/OpenAPI specs, GraphQL, and JSON-RPC, to name just a few.
They must also be familiar with the various penetration testing tools used in testing APIs, such as Postman, Burp Suite, Fiddler, OWASP Zap, and other similar testing tools. While some of these are also useful in web application testing, how it’s used with APIs differs and exposes new levels of complexity, especially when dealing with things like excessive data exposure and mass assignment vulnerabilities that are exploited through object property manipulation.
Types of Application Penetration Testing Methods
So, just to add to more complexity of all this, one other aspect needs to be considered when looking at how to conduct application penetration testing. And that is if the security team has access to the internal code structure of the application and infrastructure or not.
There are three key testing methods you should consider when conducting penetration tests to uncover vulnerabilities.
Black Box Testing
Sometimes called Behavioural Testing or Functional Testing, a penetration tester has no prior knowledge about the application’s internal structure in advance. They do not have access to critical systems or code or know how the application is packaged, deployed, and configured.
Penetration testers can use Dynamic Application Security Testing (DAST) to test from the outside in. Automated tools like the Burp Suite scanner can help with this, running blind testing in the background for you as you explore the application.
White Box Testing
This testing method includes access to source code or the internal workings of a software system, such as its code and infrastructure. This allows for static code analysis through Static Application Security Testing (SAST) and testing from the inside out.
I’ve discussed how to trace API exploitability through code review and taint analysis before. This is an important part of this testing methodology.
Penetration testers may also have access to the build and deployment processes (i.e., CI/CD pipelines), allowing them to better understand how the application works and how it may be (mis)configured. They have system information about the target system and target network and can also expose configuration errors in the IT environment or cloud environment hosting the application.
Gray Box Testing
The last method of testing combines both Black Box and White Box Testing. Hence the term Grey Box Testing. The pen tester has some understanding and access to the application’s internals or can get it by other means.
This might include gaining access to code and code structure through reverse engineering methods, which I’ve covered in past articles like on defeating a dockerized API to get access to source code.
How Do These Methods Affect The Cost Of A Pentest?
Depending on the access given to the security team, the depth of testing can vary. Black Box Testing is much harder to perform when you have to spend far more time in recon of the application to understand how it works. So more time is spent up front learning about the app than attacking it.
White Box Testing on the other hand allows for far more testing of not just the application functionality but about the architecture and code structure. While it becomes easier to hone in on more problem areas of the app, there is far more expected in reviewing how the apps and APIs work together.
In all cases, the more test cases you can perform to get more coverage takes more time. More time costs more money. Automation can only go so far; the more complex the application is and the less access you give to the pentester means it takes more time to test the attack surface of your apps and APIs properly.
Confidence in what is tested is key here though. This is where standards and methodologies for penetration testing can help.
Are There Any Standards Or Methodologies For Penetration Testing?
There are several standards and methodologies that have been developed specifically for penetration testing. Unfortunately, most of them are focused on network penetration testing.
NIST publishes SP 800-115, the Technical Guide to Information Security Testing and Assessment (TG-ISA). The PCI Security Standards Council publishes the PCI Penetration Guide. And the Penetration Testing Execution Standard (PTES) exists to offer an in-depth and comprehensive set of guidelines that cover the entire life cycle of a penetration test.
Yet none of these do a great job of clearly articulating how to cover an application security assessment end to end. And this is where OWASP comes in.
OWASP Testing Guides
The Open Web Application Security Project (OWASP) is a widespread open source project providing security guidelines for application development and testing. They have several guides on everything from setting up secure coding best practices to penetration testing methodologies.
I’ve discussed how to use OWASP guidance to build your own testing blueprint before. But let me zone in on a few things that can help you understand the time needed to complete a pentest engagement properly and how it affects the cost.
OWASP Top 10 and OWASP API Security Top 10
The OWASP Top 10 and OWASP API Security Top 10 are two of the most well-known lists that show what a security team should be testing during penetration tests. Both help to represent the current state of affairs with application security threats.
These lists are standard documents aimed at increasing awareness of prevailing security issues in web applications among developers, companies, and the software industry in general. While they offer general risk categories, at best, it helps define about 50 good application security practices (sometimes called controls) that you can test for. But they don’t define individual test cases that can be easily measured.
This is where things like the OWASP Web Security Testing Guide and the OWASP Application Security Verification Standard come into play.
OWASP Web Security Testing Guide
The OWASP Web Security Testing Guide (WSTG) defines a wide range of tests to use as part of a web penetration testing process. It covers areas such as authentication, session management, input validation, error handling, and logging. There are over 100 individual tests defined in the WSTG checklist that should be conducted during any engagement. Many of these can be automated; those that can’t take real time and effort to execute properly.
OWASP Application Security Verification Standard
The OWASP Application Security Verification Standard (ASVS) offers a comprehensive set of security tests that should be performed when testing the security of web applications as well. This standard covers areas such as authentication, access control, data validation, and secure coding. That means there is some overlap with WSTG. But the ASVS approach differs.
The primary aim of ASVS is to normalize the range in the coverage and level of rigor available in the market when performing Web application security verification using a commercially-workable open standard.
At the bare minimum, I believe that during any given Black Box pentest, you should ensure the 130 controls defined in ASVS Level 1 are tested. In other words, testing for just the OWASP Top 10 isn’t enough.
If your team has been given appropriate access for White Box Testing to review architecture, configuration, and code, you should strive to verify the 260 controls defined in ASVS Level 2. If you want a great ASVS checklist for audits consider checking this out.
The True Cost of an Application Security PenTest
OK, so let’s get back to the premise of this article. Remember that whole peanuts and monkey thing?
Ya, that.
Why did the CTO get a huge swath of quotes ranging from $1,000 to over $10,000? I’ll bet it comes down to what the firms thought a “pentest” represented. Their experience. And their understanding of the scope of work.
Remember, the objective of penetration testing is to identify vulnerabilities and to determine the exploitability of these vulnerabilities and their impact on your organization. To know the actual security risks and allow stakeholders to get a suitable risk assessment of what is exposed.
That objective is nullified if all you get is a report of useless findings that have no material impact on the real security of the application. Or worse yet, a clean report with no findings simply because the pen testing tools can’t assess the real security risks in the code and configuration.
When thinking of the true costs of a valuable pentest, I have a few questions I want you to ask yourself and your pentesting partner…
How Long Should A Pentest Take To Complete?
Depending on the number of applications, web pages, and API endpoints you have, along with the complexity of the application’s business logic, think about how much effort has to go into properly completing each test case.
Considering the combination of WSTG and ASVS controls, you could have hundreds of test cases to run. Even if half of them can be automated, there will still be hundreds that cannot. And some of those can take significant time to properly execute, review, and document. The more endpoints you have to test, the more time it will take to properly get the coverage and confidence you really need.
This is why timeboxed end-to-end pentests are difficult to estimate and why it’s usually better to focus on a specific area or feature when you first begin adding pentesting to your application security assessments. Instead of testing everything, focus on the attack surface with the most impact… usually the APIs that handle all the data and business logic.
It’s like that adage goes… the only way to eat an elephant is one bite at a time. In other words, while it may seem daunting, overwhelming, and even impossible… it can be accomplished gradually by taking on just a little at a time. You learn. You remediate and improve. And then you move on.
This then leads to the next question to ask…
How Often Should Pentests Be Performed?
You may hear that pentests should be done annually. If you follow my recommendations of scoping the effort into smaller engagements, that doesn’t make sense. You really should be considering continuous penetration testing or, at the very least, frequent testing in different areas.
However, if you decide to test everything all at once, I still don’t think yearly is appropriate.
The pentest report you receive as part of the engagement is a point-in-time reference to the security state of the areas under test. The day after the information is presented things may change. You will remediate issues in some areas. You may add new features to others. And as applications age, they become more brittle as time passes and the threat landscape changes.
As such, you need to consider the frequency of your engagements. With the typical meantime-to-remediation (MTTR) of reported vulnerabilities as part of responsible disclosure being 90 days, that seems like a good place to start.
That means at least quarterly. It should be even more frequent if you are part of a more agile team.
As your security maturity within your organization improves, you start to see the value of continuous security testing and maybe even the introduction of an application red team.
Conclusion
I hope I’ve been able to give you a glimpse into the API penetration testing pricing dilemma.
You will find firms out there willing to charge peanuts. You have to wonder though… who are the monkeys doing the work?
Are they just running some scanning tools and giving you the output? Do they really understand the effort required to get proper coverage for all the different endpoints under test? Can they give you a report that properly defines, documents, and demonstrates they have completed work that aligns with testing standards like ASVS and WSTG?
And are they capable of providing you with proof-of-concept (PoC) exploits demonstrating business impact and helping with remediation guidance and remediation testing to ensure exploitable vulnerabilities can be prioritized, fixed, and retested promptly?
You get what you pay for. If your pentesting partner isn’t discussing this with you and is offering you fixed pricing and fast turnarounds that seem too good to be true… well… it probably is.
Proper pentesting should be a regular part of your software development lifecycle (SDLC). It’s a real investment that costs real money. You have every right to ask for measurable results demonstrating coverage that gives you confidence in the efforts.
You may think you’re saving money by going cheap, but it will likely cost more in the long run. And if something does happen, you have no defense for why you weren’t properly assessing your applications for security vulnerabilities on a regular basis.
Choose wisely and take the time to do due diligence when selecting your pentesting partner. Do the hard work upfront, and you will reap the rewards in the long run.
This article was written to provide guidance on API penetration testing pricing, but it’s important to note that security should never be boiled down to just a dollar value. Investing resources into protecting your assets is an essential part of any successful business. From financial losses to reputational damage, the consequences of a security breach can be devastating. So ensure you invest in the right team capable of delivering results that meet your needs and expectations – no matter the cost.
Good luck!
Like what you are reading? Then consider subscribing to the API Hacker’s Inner Circle newsletter, where I share insights into the work and research I do each week.
