TL;DR:

• Pentesting in a production environment offers a realistic view of security but can disrupt customers and impact data integrity.
• Testing in production can be time-consuming and may not provide full coverage due to security controls.
• Unintended exposure, data pollution, and unexpected costs are additional risks to consider.
• Establishing an effective alternative environment for testing is recommended to minimize risks and maximize testing effectiveness.
• Carefully weigh the benefits and risks before deciding whether to pentest in a production environment.

Is it best to perform pentesting in a production environment?

You’re preparing for an upcoming penetration test and you need to provide a target environment to be used for the engagement. Pentesting in a production environment should be fine, right? While I can certainly admire the fortitude of such a choice and do appreciate there is sometimes a place for it, this is a complex topic with a lot of moving parts. As such, there are many factors which should be fully understood and considered before committing to such a choice.

Understanding What a Penetration Test Involves

Before we go any further, you’ll need a clear understanding of pentesting and how it might differ from other forms of testing. You may have heard the terms ‘penetration test’ and ‘red team’ mistakenly used interchangeably. However, there are significant differences in testing goals and methodologies which can impact your decision on the type of testing environment to use. When considering the risks and rewards of pentesting in a production environment, it's crucial to weigh all factors carefully.

Let’s imagine a scenario where you are in charge of a building containing top-secret government secrets. These secrets should never be accessible to an authorized party under any circumstances. They need to be maintained or the country could collapse.

A penetration tester would use any means necessary to break into the building, for example: entering an unlocked door, smashing a window with a brick, crow-barring the door, cutting the cable to the alarm if exposed, or even using a tank to shoot through the walls if they aren’t thick enough. This could be a fairly destructive test in nature with little regard for the integrity of the building or its contents.

While this example is an oversimplification of penetration testing, it is all to say a pentester will attempt to test all aspects of confidentiality, integrity, and availability on the entire exposed attack surface without hesitation and report all weaknesses found. There will be a large volume of both manual and automated malicious requests sent to the target in an attempt to find and exploit weaknesses in any known functionality/services/endpoints. In some cases, this may also result in unforeseen outages or other consequences. Any accessible data may be downloaded, altered, or deleted as a function of this process.

In contrast to penetration testing, a red team engagement might attempt to break into the building with more specific and covert methods like picking the lock in the dead of night or literally posing as the pizza delivery guy and then sneaking off to gain access without being detected. For red teaming, the job is typically finished as soon as they enter the building and have their objective achieved just once. While this is often a more surgical, less destructive, and more realistic attack scenario, it will not typically test all defences comprehensively and many other weaknesses go undetected/unreported. Both testing methodologies can add value to assessing your organization's security posture, but it’s important to understand the differences in this case.

Here are 5 reasons you should consider penetration testing in production, and 10 you shouldn’t.

5 Benefits of Conducting Pentesting in a Production Environment

1. No deltas in the test environment, it’s apples to apples.

The first benefit of testing in production is perhaps the most obvious. If you are testing the same environment as customers are using, it provides the most realistic view of security in that space. In some cases, minor configuration differences in an alternate environment (like staging, for example) can result in a missed vulnerability only present in production. Alternatively, it could also lead to vulnerabilities being reported which are only applicable to staging and not applicable in production. Therefore, it is not uncommon for staging or QA environments to have somewhat looser security restrictions to avoid impeding ongoing development work. In contrast, all vulnerabilities reported in a production test will be applicable.

2. Testing resiliency with less chaos

As mentioned previously, pentesting can involve managing thousands or millions of requests, often containing unexpected data. As such, it can be a good test of the target's resiliency against such large volumes of traffic. Many modern viewpoints highly value such testing of resiliency on production and suggest it can result in extremely robust systems. Netflix’s Chaos Monkey project is a great example where Netflix runs testing on their production networks.

3. Development environments can be fragile: “Can you fix that Dev environment for me, again?”

Back-end test environments are often not configured with as many resources as their production counterparts. This can often lead to additional outages testing in those environments, causing additional delays in testing and requiring more time/effort to support.

4. Stability, as the bleeding edge often isn’t stable.

If the environment in question is shared with QA or Dev teams, it can sometimes have more bleeding-edge code. This code is not always as stable as production code, which can cause gaps in testing vs. production if key features are not completely functional during the test window.

5. Test satisfaction: the real prize is more fun.

Last and probably least valuable, finding a critical vulnerability and popping a live production system with real customer data is generally more fulfilling to the tester. Though this comes selfishly from being a penetration tester myself, you shouldn’t let this advantage sway your decision.

10 Drawbacks of Pentesting in a Production Environment

1. Customer disruption. Don’t mess with the ones that pay our bills

I hope it is becoming clear that penetration testing is a very invasive exercise and there is a very real possibility of disruption to production systems and customers when testing is done in the production environment. Even if deliberate denial of service attacks are avoided in a particular engagement, I have seen many production systems taken down for long periods as a side effect of many different test activities, once from even a single byte of data. Depending on the use case of the target and the type of customer, this can sometimes represent thousands or millions of dollars in lost time. Even something simple like an account lockout as a result of credential stuffing attacks can be very disruptive to some customers if they cannot easily reset the account or password.

2. Data integrity. “Why is my account balance now 1234?”

Customer data is at risk of being exposed and losing its integrity. A large percentage of penetration testing activities involve attempting to manipulate the integrity of a system or introducing sample data to test against. Even if customer data is not manipulated directly in the production environment, there are sometimes cases where it is affected indirectly as a function of manipulating the system.

3. Time and coverage. To run or to tiptoe?

Testing in production is generally more time-consuming as testers tip-toe around doing their best not to intentionally affect customers on the system. This type of careful testing can take much longer, can cause test hesitancy, and as a result, there ends up being less test coverage within the limited test window.

4. Testing security in layers. To WAF or not to WAF?

Additional security controls enabled in production inhibit application layer testing. Controls like Web Application Firewalls (WAF) or Intrusion Prevention Systems (IPS) are a common part of production environments and will interfere with testing, but can be often disabled in staging environments if they are not already.

Keep in mind this activity is a pentest and not a red teaming engagement. As such, the primary goal is to test the security of the application/system comprehensively. Something like a WAF is a good thing to have in production as an extra layer of security, but should not be relied upon over a securely coded & configured application.

Consider the example of the building from earlier. A WAF might represent an electrified fence around the building. It is a sound extra layer of defence to stop intruders in production, but what if the power ever went out? Wouldn’t you also want to make sure the windows are closed and door locks cannot be picked? The WAF would no doubt inhibit some elements of the penetration test, so it would not be as comprehensive. If your security appliance is ever updated or changed to another vendor it may suddenly open up pathways to unknown vulnerabilities in the application layer which were previously masked by the appliance. That said, it is valuable to conduct some portion of a pentest with security appliances enabled, as they too can sometimes be the source of a unique vulnerability which might not otherwise be present.

Another disadvantage to consider is that testing through a WAF, while possible (see red teaming), is more time-consuming as the tester needs to massage all attacks to also bypass a WAF. Thus, leading to less overall testing.

Security in layers is almost always the most sound approach. Conducting some portion of a pentest with the WAF or IPS off is almost always advisable. Generally, it is difficult to do this effectively in production without introducing risk to your other customers.

5. Unintended exposure. “What's this random alert 1 box I am suddenly seeing?”

Another risk of testing in production is unintentional exposure of vulnerabilities to external parties/clients. You may think you have adequate segregation of different customer environments, but pentesting has been known to sometimes prove otherwise. Occasionally, evidence of a vulnerability could be propagated or unknowingly exposed to other external parties outside of the testing team in a common environment. Clients can then receive errors or observe other behaviours as a side effect of pentesting which could expose a weakness to them. Consequently, this unintended exposure negatively affects the user’s perception of the application.

Another concern with testing in production is data pollution. Pentesting can produce a lot of bogus data and settings in the target system. Effective manual cleanup is not always possible and it is sometimes easier to just destroy the entire environment. Alternatively, it could be restored to a backup after testing, which is often not an option in production environments.

7. Weaponized POCs: Making issues easy to reproduce is a risk.

One of the goals of any reputable penetration testing firm is to deliver a detailed report with reproduction steps to easily demonstrate the security issues identified to someone who isn’t necessarily a security expert. For more elaborate issues, pentesters sometimes include a script or executable to make this process easier to replicate. Now, imagine this script or tool is somehow leaked or inadvertently shared outside of your organization. If it’s against a real known production environment, this hot exploit is gold for an external attacker. The only thing they would need to do is run it as-is. If it’s targeting a back-end staging environment, perhaps one that is not easily associated with a real target, it may not be immediately evident where the weakness lies in a comparable publicly accessible asset. It would also mean some effort and expertise would be needed to retool any such exploit and it might be less useful to someone with less knowledge like a script kiddie.

8. Unexpected costs. “Why is my service provider bill 50x as usual?”

It’s uncommon nowadays for a development team to build every feature in an application or service in-house. Third-party vendors offer services for many common tasks like sending emails, SMS, data analytics, user management, scheduling, and multi-tenancy, among others. It’s easier to consume these services than build your own, though they usually also come with use-based fees or monthly bandwidth limits.

I’ve been a first-hand witness to a pentest blowing through a yearly allowance of such service in days, either causing a huge bill or a denial of service for the particular feature when limits are reached. To avoid this issue, it’s best to use separate services between production and development environments. It’s also easier to temporarily disable a particular problematic service, or devise an internal workaround to be leveraged during a pentest to avoid that unexpected bill.

9. Legal issues. “What privacy agreement?”

Remember that data privacy agreement you signed when you onboarded that huge government contract? Does it include any contingencies for when a pentester gains access to that customer's data during a pentest in your production environment? Are you required by law to disclose any such incident to said customer if it happens?

While you may assume your multi-tenancy or data segregation strategies make it impossible for any customer to access any other customer's data, this is a common weakness identified during pentests. For example, accessing data from the ‘test customer 2’ account instead of from an organization with a 3-letter acronym could be less of a headache if the pentesters are successful in that regard.

10. Masking the attack. “Don’t worry, there’s just a pentest going on right now.”

Last but not least, a pentest on a production environment could indirectly help mask a real attack. Pentesting can create a lot of noise for your operations team. It's unlikely, but if a real attack were to occur during the testing window, it might be very difficult to accurately detect and analyze.

Determining When to Evaluate Your Objectives for Pentesting in a Production Environment

No testing strategy is perfect for everyone. It’s important to consider and weigh all factors based on your own testing requirements, motivations and resources available. A few questions you should be asking yourself include:

• Can your production system handle the additional load of a pentest without impacting your business?
• How much impact could an outage in production have on your business?
• Do you already have one, or can you easily create a mirrored environment?
• Is the goal of the assessment to identify all potential weaknesses so they can be fixed, or to assess your current defences and risk of exploitation?
• How important is verifying resiliency against Denial of Service attacks for you?
• How sensitive is the information your customers are storing on production, and what would be the repercussions if a pentester were able to gain access to it or impact its integrity?

Consider these factors carefully when making a decision.

You may also be asking yourself what does Software Secured recommend?

The short answer is that we usually recommend performing the bulk of primary pentesting in a staging or test environment which is as closely configured to a typical customer in the production environment as possible. Then we recommend augmenting this with only targeted testing of any key differences or deltas on production.

LEARN MORE

Creating a Suitable Testing Environment as an Effective Alternative for Pentesting in a Production Environment

Examples of non-production environments that you can use for testing include an existing staging environment, development or QA environments, or even a dedicated security/pentest environment. To minimize some of the risks highlighted above, this environment should be configured as closely to production as possible. Ideally, it should be configured with comparable back-end resources. The build of any application should be the latest stable build as used in production unless there is a strong need to test something newer before release. If possible, having an environment which is not shared or used for any other resource-intensive testing or operations is ideal. Another good practice is to take a backup or snapshot of this environment before active testing starts, so it can be quickly restored in the event it is broken during testing. This will also help minimize any test interruption and support effort required for the test. Configuring any application with functional sample data/workflows comparable to your typical customer will usually also greatly improve any testing coverage.

Penetration testing is a critical activity in securing any system. Choosing whether or not to pentest in production is an important decision to make for the effectiveness of the test and potential impacts to your customers. While proper planning can help make pentesting in production environments more cost-effective and efficient, it is important to consider the potential risks and impact on business operations. Conducting pentesting in non-production environments may provide a more controlled environment, but it may not fully capture the complexities and vulnerabilities that can arise in a live production environment. Be sure to understand the benefits and risks, and choose wisely.