Application Security·~14 min read·3,179 words

Penetration test procurement and review: picking a firm, scoping a test, reading the reportPremium

Practitioner reference for CISOs, security leads, founders running a first pen test, and the procurement officer about to sign a statement of work they do not fully understand · 2026-05-25 · Written from twelve years of either commissioning pen tests, performing them, or being called in to tell a customer whether the report they paid for was actually any good

Why this guide exists

Most penetration test engagements I see are bad. Not bad in the sense that the vendor was incompetent — most reputable firms have competent testers. Bad in the sense that the customer scoped the test wrong, briefed the testers incompletely, accepted the report without challenge, and then had nothing to act on except a PDF that satisfied the audit requirement.

I have read perhaps three hundred pen test reports over the last twelve years. Maybe forty of them surfaced findings that mattered. The rest were a mix of automated scanner output, low-severity hygiene issues anyone could have found running a free tool, and the occasional finding that the customer was already aware of but had not yet remediated. The price tag on those engagements averages about ₹6-12 lakh in India and US$15,000-40,000 elsewhere. That is a lot of money for a PDF that nobody acts on.

The pattern that produces a useful pen test is not mysterious. It involves choosing the firm carefully, scoping the engagement so that the testers can find real problems, briefing them well, reading the report critically, and acting on the findings with the same discipline you would apply to any other internal finding. Each of those steps has predictable failure modes. This guide is what I tell clients when they ask me to help them get a useful pen test rather than a compliance artefact.

Why you are commissioning the test

The first question is the one most customers do not ask: what is this test for? The answer determines almost everything else. Three common purposes, each with different implications:

Purpose 1: Compliance / audit requirement. SOC 2, ISO 27001, PCI DSS, SEBI CSCRF, RBI guidelines, customer security questionnaires — many regulatory and customer requirements specify an annual pen test. The dominant goal is the artefact, and the test scope should match what the auditor or customer will inspect. The wrong move here is over-specifying scope for compliance; you end up paying for testing the regulator does not require.

Purpose 2: Pre-release assurance. A new application, a new product line, a new infrastructure migration. The dominant goal is finding the things engineering missed before they reach customers. Scope should be focused, deep, and behaviour-driven; the report is for internal action, not external display.

Purpose 3: Periodic assurance of the existing estate. Quarterly or semi-annual testing of the production environment. The dominant goal is detecting drift — new vulnerabilities introduced since the last test, changes in attacker techniques that affect existing controls. Scope should rotate across the estate, not test the same surface every quarter.

A single pen test can satisfy one of these well; trying to satisfy all three with one engagement is the most common procurement mistake. If you want both compliance attestation and real engineering value, plan two engagements. The cost is roughly double; the value is roughly five times.

Be honest with yourself about which purpose drives the test. If the procurement is happening because the auditor asked, the scope should match the auditor’s evidence requirement, the test should be fast, and the report should look the way the auditor expects. Pretending it is a deep technical exercise when it is really a compliance artefact wastes the testing team’s time and your money. Reverse is also true: if you want engineering value, do not let the engagement be steered by what the auditor will look at.

Picking the firm

Three categories of firm, each with different cost structures and value profiles:

The big-name security firms. PwC, Deloitte, EY, KPMG cybersecurity practices; the boutique consulting arms of bigger consultancies. They charge a premium (typically 1.5-3× the price of independent firms), produce polished reports, employ rotating staff including very junior testers, and rarely surface deep findings. You hire them when the customer or auditor requires the brand on the report cover. They are the wrong choice when you want engineering value.

The mid-tier specialist firms. Bishop Fox, NCC Group, IOActive internationally; Lucideus (now SAFE Security), Network Intelligence, Niki Infosec in India; many regional specialists. They charge fairly, employ experienced testers with deep specialisms, and produce useful reports. You hire them when you want engineering value and you have a customer or audit requirement that demands a recognisable firm. The challenge is that the named senior tester in the proposal may not be the one who actually does your work; ask specifically.

The independent practitioners and small boutiques. Individual CEH/OSCP/OSCE-certified consultants, two-to-five-person specialist shops, the testers who left a big firm to work for themselves. They charge the least, produce the most genuinely engineering-valuable reports, and have the worst-looking deliverables. You hire them when you want the test to find things and you do not need the brand. The risk is variability: the best independent testers are better than anyone at the big firms; the worst are running automated scans and lightly editing the output.

I have hired all three categories over the years. For compliance artefacts, the mid-tier firms are usually the right cost-value point. For engineering value, the right independent practitioner — vetted carefully — beats anything else. The big-name firms are almost always overpriced for the value delivered, but they win procurement-driven deals because the brand reduces buyer risk.

A few discriminating questions when evaluating any firm:

Who will actually be on the engagement? Get the named team, the CVs, the certifications, and the number of similar engagements they have delivered. If the named senior tester is “subject to availability” in the proposal, assume you will get whoever is on the bench when your start date arrives.

Can I speak to the lead tester before signing? A 30-minute call with the actual person who will do the work. They should be able to discuss your stack technically and ask good questions about your architecture. If they cannot, you have a sales engineer, not a tester.

Show me a redacted prior report at the same scope as mine. Not a marketing example, a real engagement report. Read the findings critically. Are there genuine business-logic or architectural findings, or only scanner output? Are the descriptions technically detailed, or templated boilerplate?

What is your methodology in detail, and what is your time budget per finding category? “We follow OWASP” is not a methodology; that is the table of contents. A real methodology answer covers reconnaissance time, automated scanning, manual testing depth per surface, business logic testing approach, post-exploit/privilege-escalation testing, and the time allocated to each.

Are you CERT-In empanelled (for India engagements requiring it)? If RBI, SEBI, or IRDAI inspectors will look at the engagement, the firm must be CERT-In empanelled. Verify on the CERT-In website. Without empanelment, the report is contestable on credentialing grounds.

Scoping the engagement

Scoping is where most engagements go wrong, and where most of the value is captured or lost. The standard scope categories:

External infrastructure pen test. Internet-facing IP ranges, exposed services, perimeter security. Largely automated by 2026 unless the firm goes deep on the few services found exposed. Usually low value unless you have a complex internet-facing estate. Worth doing annually for compliance, rarely worth deeper investment.

Web application pen test. Authentication and session management, authorisation and access control, input validation, business logic, API security. This is where most real findings live. Should always include manual testing of authentication and authorisation logic, business logic of the core workflows, and API endpoint enumeration. The most common deep finding category by far.

Mobile application pen test (iOS and Android). Client-side controls, local storage, certificate pinning, jailbreak/root detection, deep links, IPC. Underdone in most engagements because customers ask only for the iOS version or skip mobile entirely. If you have a mobile app handling sensitive data, mobile testing is mandatory; the threat model is different from web.

Internal network pen test. Lateral movement from a beachhead, privilege escalation, sensitive-system access. Useful for organisations with substantial on-premises infrastructure. Less relevant for cloud-native SaaS, though cloud-internal lateral movement is increasingly tested under the next category.

Cloud configuration assessment. AWS, Azure, GCP, Oracle Cloud configuration review against best practices. IAM policies, network rules, encryption settings, logging coverage, sensitive-data accessibility. Increasingly mandatory; SOC 2 Type 2 and ISO 27001 auditors now expect this for cloud-hosted businesses. Often delivered as a hybrid pen test + configuration review.

Red team engagement. Adversary simulation with no scope limits beyond rules of engagement. Multi-week, expensive (typically 3-10× the cost of a focused pen test), high value when done by a competent team. Most organisations are not ready for red team; they should be running better pen tests first.

Social engineering / phishing assessment. Targeted phishing campaign, optionally combined with physical access attempts. Useful as a board-level demonstration; less useful for engineering improvement. Often skipped in favour of phishing simulation tooling, which is cheaper and produces continuous results.

The scoping decisions that matter:

1. Test windows and rate limits. Production systems versus staging. If you only allow testing in a staging environment that does not mirror production, your test does not cover production. If you allow production but rate-limit aggressively, you will not find findings that require time to identify. Negotiate a production window with rate limits relaxed for the engagement.

2. Authenticated vs unauthenticated testing. Almost all real findings live in authenticated functionality. An unauthenticated pen test will find perhaps 20% of what a properly-credentialed test will surface. Provide test accounts at every privilege level: anonymous, signed-up free user, paid user, admin, super-admin, internal-employee. Every privilege boundary is where authorisation findings live.

3. The “out of scope” list. Every engagement has these; they should be deliberate and minimal. Common out-of-scope items: DDoS testing, social engineering against employees, physical access, hosted third-party SaaS not under customer control. Each item should have a reason. “Out of scope because it would be hard” is not a reason. “Out of scope because we cannot grant authorised access to a third-party’s infrastructure” is.

4. The time budget per surface. A two-week web application test that covers fifteen distinct functional areas will spend roughly half a day per area. That is not enough for any non-trivial area. For real value, the engagement should specify the deep-dive areas and allocate at least two days each. If your application has fifteen areas you care about, you need either a longer engagement or a smaller scope.

5. The retest. Findings get remediated; the retest verifies the fix. Insist on a retest within 30-60 days of report delivery, ideally at no additional cost. Without retest, the engagement produces findings but does not produce closure.

A scoping document I would recommend insisting on includes: a list of every URL, IP range, mobile app build, and cloud account in scope; a list of every authentication credential and privilege level the tester will use; the testing windows and rate limits; the rules of engagement (what is allowed, what is not); the deliverables (report contents, debriefing format, retest scope); and the success criteria.

Briefing the testers

Most customers do not brief their testers. They send the SOW, grant access, and wait for the report. This wastes about 30% of the engagement value.

A proper briefing covers, at minimum:

  • The application’s purpose, key users, key data flows, and the worst things that could happen.
  • Architecture overview: technology stack, hosting, authentication mechanism, sensitive data locations, integration points with third parties.
  • Known issues: things you already know are vulnerable or fragile. Yes, tell them. The point is not to “test if they find it”; the point is to direct their time to the things you do not already know about.
  • Recent changes: what shipped in the last 90 days, what is in flight, what is about to change. The new code is where the new bugs are.
  • Critical workflows: the three to five workflows where a vulnerability would do the most damage. Direct them to spend deep time on these.
  • People: who they can call when they find something, who needs to be looped in for production-impact discussions, who handles incident response if testing accidentally triggers something.
  • The rules of engagement detail: what destructive testing is allowed, what credentials they can rotate, what data they can exfiltrate to demonstrate impact, what they should stop on first sign of and notify.

The briefing should happen as a working session, not a document handover. The testers should ask questions. The questions they ask tell you whether they understand the engagement. If they do not ask, that is signal.

Reading the report

The report arrives. The instinct is to read the executive summary, look at the severity bar chart, count the criticals and highs, and route the findings to engineering. That instinct produces a workflow but not understanding.

How I read a report critically:

First, the methodology section. What did they actually do? How much time on each surface? What manual testing techniques did they apply, beyond automated scanning? If the methodology section is one paragraph of OWASP boilerplate, the report is light. If it is two pages of specific techniques, time budgets, and tools used, the engagement was real.

Second, the findings inventory by severity, but with my own judgment of severity. Pen test severity ratings are notoriously inconsistent across firms; one firm’s “critical” is another’s “medium.” For each finding, I re-rate it based on the business impact in our context: what data is exposed, what is the attacker’s ease of exploitation, what are the chained possibilities. A “high” finding the firm rated as low because they did not understand the data sensitivity gets re-rated up. A “critical” finding the firm rated high because it sounded scary in isolation gets re-rated down if it requires conditions we already preclude.

Third, the absence of findings. What is missing? If the web application report has no business logic findings, the testers did not test business logic; they ran scanners. If the cloud assessment has no IAM-policy findings, they did not audit IAM properly. Real business-logic findings are the differentiator between a useful test and a compliance artefact; if they are absent, ask why.

Fourth, the reproduction steps. Every finding should have a step-by-step reproduction with screenshots or HTTP requests. If reproductions are vague, the finding is questionable. If reproductions are detailed, you can verify and remediate confidently.

Fifth, the remediation guidance. Generic remediation guidance (“implement input validation,” “use parameterized queries”) is filler. Specific remediation guidance (“change the authorisation check at controller X to verify the user_id matches the resource owner_id”) is real. The quality of remediation guidance correlates strongly with the quality of the testing.

Sixth, the chained findings. Good testers do not just list isolated findings; they describe how findings chain together. “Finding 7 plus Finding 12 plus weak Finding 18 collectively enable full account takeover starting from anonymous user.” This is the analysis that separates a tester from a scanner.

If most of these elements are weak, the report is weak, and you should push back. The firm can usually do better if asked. A finding count and severity histogram is not a report; it is a summary.

Acting on the report

The hard part is what comes after the report. Most pen test findings get logged in JIRA and worked through ordinary engineering prioritisation. That is fine for the bulk of findings. The patterns where this fails:

The genuinely critical finding gets buried in the backlog. Pen test findings need a separate severity scale and separate SLAs. A critical pen test finding should have a 7-day fix SLA, not the standard engineering bug SLA. Without separation, real findings rot.

The “we accept this risk” finding is not formally accepted. Some findings will not be fixed — too costly, too disruptive, or genuinely low-impact in your context. These need formal risk acceptance: documented decision, business owner sign-off, compensating control if any, review date. Without this, the finding sits open forever and shows up in every subsequent audit as an unremediated pen test finding.

The retest gets skipped or done poorly. Retest is where the engagement actually closes. Without it, you have findings; you do not have remediation evidence. Schedule retest at engagement signing, not after the report.

The systemic findings get treated as individual findings. Five separate findings of “SQL injection in different forms across the application” are not five findings; they are one root-cause finding about parameterised queries not being enforced. The remediation is at the framework level, not the per-finding level. Most engineering teams remediate the five and miss the framework fix.

The findings are not fed back into the SSDLC. A finding that escaped to production indicates a gap in the pre-production process — code review, SAST, design review. If that gap is not closed, the same finding category will recur next year. Each pen test should produce process improvements upstream as well as fixes downstream.

A reasonable post-report cadence:

  • Week 1: Internal review, severity re-rating, JIRA tickets created with separate pen-test labels.
  • Week 2: Critical and high findings into the active sprint. Mediums into the next two sprints. Lows into hygiene backlog.
  • Week 4: First retest of critical and high findings remediated to date.
  • Week 6-8: Full retest of all findings the firm agreed could be remediated in that timeframe.
  • Week 10: Final report from retest. Risk acceptance documents for any remaining findings.
  • Week 12: Lessons-learned retrospective with engineering. Process improvements identified.

This sequence sounds heavy. It is heavy. It is also why pen test findings stay closed rather than recurring.

What this guide does not cover

Three areas worth their own treatment:

  1. Red team engagement specifically. Red team has a different procurement pattern, a different scope structure, a different reporting style, and a different operational impact. The principles in this guide apply directionally; the specifics deserve a dedicated guide.

  2. Specific tool selection. I have not recommended specific scanning tools, exploit frameworks, or testing platforms. The right tool set depends on the engagement and the tester’s familiarity. The tester should pick the tools; the customer should pick the tester.

  3. Bug bounty programmes as a complement. Pen tests and bug bounty programmes serve different purposes and complement rather than substitute for each other. Bug bounty operational mechanics — programme design, reward structure, triage discipline — are a separate topic.

This guide is a practitioner reference, not procurement advice. It reflects how pen test engagements typically work as of May 2026 in India and internationally, and is not affiliated with any specific testing firm. Compliance teams should validate specific obligations against the applicable regulatory and customer requirements driving the engagement.

ControlForge maps pen test scope and findings against ISO 27001, SOC 2, PCI DSS, RBI CSF, SEBI CSCRF, and other framework requirements; the synthesis surface helps trace which findings carry compliance weight across multiple regimes.