What to Do When You Don't Agree With Your Penetration Test Report

TL;DR:

• Read the penetration test report thoroughly to understand key findings and remediation work needed.
• Develop a plan, assign resources, investigate findings, and retest your environment.
• If you don't agree with the findings, talk to your penetration test provider and evaluate risk rating systems.
• Consider consulting a reputable third party for a second opinion if needed.
• Treat a negative penetration test as a learning opportunity to improve security posture and prevent future vulnerabilities.

Effective communication with the client is paramount in penetration testing to ensure a successful engagement. It begins with clearly defining the scope and objectives of the test, which helps manage expectations and align the testing process with the client's goals. Throughout the assessment, maintaining open lines of communication allows for addressing any concerns or unexpected findings promptly. When reporting results, it's crucial to provide a balanced view, highlighting both security shortcomings and areas where defences proved effective. This approach demonstrates the value of the client's existing security investments and provides a comprehensive picture of their security posture. Emphasizing partnership and fostering a collaborative relationship with the client enables testers to deliver more meaningful and actionable results. By prioritizing clear, ongoing communication, penetration testers can ensure that their findings are well-understood and valuable to the client's overall security strategy.

Read the penetration test report

The first thing you should do when receiving a penetration test report is to read it in its entirety. You need to understand the key findings, and how the testing was performed and get an idea of what remediation work needs to be done. When you don't agree with your penetration test report, it's crucial to understand the key findings.

Most penetration tests come with an executive summary which can be very useful for getting a high-level understanding of the test. You can also use this opportunity to evaluate how close your organization is to its final state. For example, if the test was related to a compliance standard then the report should let you know if you have the proper controls in place or what controls you are lacking.

Make a plan & implement

Now that you have investigated the findings and have identified what work needs to be done, the next step is to develop a plan and implement it. This plan should outline the order in which vulnerabilities should be fixed, who will be responsible for doing the fixes, the overall timeline for completion of all vulnerabilities and final validation.

Assign necessary resources for improvements

Once you have an overall understanding of the findings of the penetration test the next step is to assign resources. These resources will be needed to start working on remediating the issues identified in the report. This should be a combination of technical staff as well as management that will be needed to help drive the actions to completion and provide authorization to make required changes.

Thoroughly investigate the findings

Once you have the resources in place you can start investigating the findings outlined in the report. Typically, this should be done in order of severity with the highest severity issues being investigated and remediated first. In addition to finding and fixing issues, there should be attention given to identifying the root cause of the vulnerability and addressing that. For example rather than simply fixing a misconfiguration in a system you may want to look at the SOPs associated with configuring those systems and ensuring that the documented instructions specify secure configurations. Doing this extra step helps to reduce the likelihood of future vulnerabilities being introduced to the environment.

Conduct testing and retesting procedures

The final step is to perform a retest of your environment. This is critical to ensure that the fixes you implemented remediate the vulnerability and don’t leave any workarounds that can be exploited by a hacker. This retest should be done by professional penetration testers, not internal devs or staff.

What to do when you don't agree with your penetration test report findings

Make sure you understand the finding: If you don’t think a finding is accurate the first thing you should do is make sure you understand it correctly. Read the details of how the finding was discovered in the report and work with your internal technical staff to understand how this finding may be affecting your business. If you still disagree with the finding then you should contact the testers and speak with them to ensure that it is a vulnerability and not a false positive. The goal is to understand the full content of the finding, this includes the technical issue, sensitivity of data involved and potential business impact.

Take it as a learning opportunity: Any negative penetration test can be demoralizing, but any penetration test that finds a vulnerability benefits the organization by allowing it to improve its overall security posture. A negative penetration test should be seen as a learning experience, not a failure.

If you have evaluated the context, severity, and sensitivity of the data involved and the other measures, you can use the following steps to help further determine the potential impact of any finding in your pen test report:

Initiate a discussion with your penetration test provider

As mentioned above you shouldn’t hesitate to speak with your penetration testing provider if you have concerns about the report you were given. To do this first you should book a meeting with the penetration testing team. In this meeting, you can explain that you do not agree with the finding(s) provided in the report. Typically, penetration testing reports include the exact steps the testers used and how you can reproduce the same results. While this is included in the report, if you have been unable to reproduce the results yourself the testers can walk you through the steps during this meeting and help you troubleshoot why you were unable to reproduce these results yourself during your initial validation. They can also walk through the steps as to why they may have scored this vulnerability as a certain ranking. Many factors can affect the severity ranking of a vulnerability, many of which may not be apparent from the client’s point of view. Sometimes with context, a pair of low or medium vulnerabilities can become a high or even critical depending on the scenario and how they are used. This may be a reason why something that appears to be a small risk to the client may be given a high ranking because of how it can be used or combined with other vulnerabilities. If they walk through the entire story/scoring/reproducibility and you still do not agree, there are more steps you can take to verify the impact.

Assess the effectiveness of risk rating systems

Rather than focusing on vulnerability itself, you may want to look at the company’s risk rating system. Verify if your pentester uses multiple risk rating systems, and how they come up with their scoring, if they are only using one risk rating system this might be an issue. Ideally, the pen testing provider should have more than one rating system and they should have a method of scoring vulnerabilities that is objective and not done by feeling. If they use 2 that are well-known in the industry and do not score by feeling, there is more merit to their rating. You should have gotten a good picture of their scoring system already from the report itself, their follow-up call, or during the penetration testing process. If you were not privy to this information the entire time, this is a large red flag.

Seek advice from a reputable, independent third-party

If you have done these previous two steps, and still are not happy with the outcome, it might be useful to ask a third party to consult on the matter. This does not mean that you need to get a second penetration test, but another security consultant should be able to go through the report along with the pentesters notes, and reproducibility, and have your application context to determine whether the penetration test and its findings are credible. Depending on the feedback you get from the third party you can decide whether or not a second penetration test is a good idea or not.

How to maximize the value of a negative penetration test

• This is a learning experience and not a failure: If you get a penetration test that comes back with multiple vulnerabilities you should look at this as a chance to improve. The reality is those weaknesses were there before the test and now that you are aware of those issues, you have the opportunity to fix them before they could be exploited.
• Look into training your developers on secure coding: A common cause of security vulnerabilities in applications is a lack of security awareness among developers. Developers need to understand secure coding practices so that the amount of vulnerabilities that occur in company applications is reduced.
• Understand the issues found and how they came to be: It’s important to do a root cause analysis to identify the source of these vulnerabilities. Unless you find and resolve the root cause there is a possibility that similar vulnerabilities will surface in the future.
• Create a plan to make sure the same vulnerabilities are not found in future code snippets: Once you have identified the root cause you need to make changes to your standard operating procedures (SOP) to ensure that this doesn’t happen again. For example, this may mean modifying your current testing procedures to ensure that you account for these types of vulnerabilities going forward.
• Consider regular/consistent penetration tests to stay on top of potential vulnerabilities: Once you fix your initial issues you should consider doing regular penetration tests to ensure that new vulnerabilities in your environment are detected early and fixed. If you wait a long time between penetration tests then you give time for vulnerabilities to accumulate undetected.
• Proactivity is better than reactivity: Remember that going forward it’s always better to be proactive than reactive. Make a plan to do routine security activities like source code reviews and security awareness training for your employees to ensure that you prevent vulnerabilities rather than trying to fix them after a long time of them going undetected.

Conclusion

If you're not happy with your penetration test report, there are a few things you can do. First, try to understand where the testers were coming from. They may be considering different contexts and scenarios. Second, reach out to the testing company and ask for clarification. Let them walk you through their process and hear their explanation for their findings and rankings. Finally, if you still don't agree, you can always request a second opinion from a qualified third party. opinion from a qualified third party.