Maintaining compliance and minimizing security risks has become more complex than ever before. Regulatory frameworks such as GDPR, HIPAA, and SOC 2 require organizations to implement strict measures to protect customer data, secure their network and systems, and respond to audit investigations. Meanwhile, organizations increasingly rely on distributed, cloud-based infrastructure and leverage third-party dependencies in their code, which can make it difficult to attain full visibility into security and compliance issues. In addition, training AI and machine learning models often requires migrating large production datasets to development environments, which have less rigorous security and access control than their production counterparts. This has led to concern that unredacted sensitive data in these training sets, along with the ML models themselves, are at greater risk of exposure and attacks.
Traditional security tools and practices center around surfacing issues that are running in your live systems—but until these issues are discovered and remediated, they can leave your organization vulnerable to attacks or compliance violations. Applying a shift-left approach to compliance in addition to traditional runtime solutions enables you to proactively address risks earlier in the development cycle and identify issues before they are deployed.
In this blog post, we’ll discuss how taking a shift-left approach, in combination with runtime detection, can help you be better prepared to maintain regulatory compliance. We’ll outline several best practices for this approach, including:
- Redacting sensitive data that travels through non-production environments
- Identifying infrastructure misconfigurations before they are deployed
- Analyzing third-party dependencies and source code for vulnerabilities
Redact sensitive data that travels through non-production environments
According to recent research, 75 percent of organizations saw a growth in the volume of sensitive data stored in their non-production environments in 2024 compared to 2023—the leading contributor to this growth being the research and adoption of AI and ML models that are trained on consumer data. Most AI/ML models require large-scale data sets for training, and it can be difficult to guarantee the security of sensitive data used for training in development and test environments, which are typically accessible by a larger pool of employees than production environments and are configured with less rigorous identity and access management (IAM) controls. If PII is leaked, regardless of what type of environment it’s stored in, your organization is at risk of violating regulations such as GDPR, HIPAA, GLBA, and more, depending on the type of data and where it travels.
To ensure that data traveling from production environments into your non-production environments stays secure and does not risk exposing PII, we recommend adding a layer of sensitive data redaction to cross-environment data movement. Cloud-native extract, transform, and load (ETL) services such as AWS Glue can help you create automated pipelines to structure, cleanse, and transform data from your production environments in preparation for use in non-production environments.
Integrating ETL pipelines into your existing workflows enables you to sanitize data during cross-environment movement and format it to best fit your use case. Here’s an example workflow that uses AWS Glue to prepare production data for ML training in a development environment:
- Extract: Define a crawler that collects metadata from your distributed database—such as column names, data types, and table relationships—and stores it within a data catalog, which helps AWS Glue understand the structure of the data so it can be properly extracted.
- Transform: Use fine-grained sensitive data detection to cleanse data of PII and tokenize sensitive fields that need to be queryable. Convert the data into a column-based format such as Apache Parquet that is more optimized for storing and retrieving large datasets (such as those used for AI/ML training).
- Load: Store the data in cost-efficient storage such as S3 buckets. Secure it using S3 encryption, and configure IAM policies for the data according to the principle of least privilege. You can now load the cleansed and structured training data from S3 to your ML compute instances.
ETL pipelines can help you manage data in your non-production environments in a compliant manner—however, when data leaves these environments for downstream locations such as Datadog, it’s subject to different access controls, data retention policies, and compliance requirements. For example, PCI-DSS forbids certain PII such as credit card information from being logged, while GDPR and HIPAA restrict sensitive information from traveling across geographical boundaries. Adding an additional logging sanitization layer can help you address these compliance challenges before logs are sent downstream, propagated across different cloud regions, and made accessible by third-party integrations and SDKs.
Datadog Observability Pipelines enables you to redact sensitive data from your logs on-premises before they are sent to downstream destinations. When you create a pipeline, you can choose between various log sources and destinations, such as Datadog, Splunk, AWS S3, Google Cloud, and more. Datadog’s Sensitive Data Scanner processor will then use preconfigured and custom scanning rules to detect and redact PII, secrets and credentials, credit cards and banking information, and other sensitive data on-premises before it’s routed to your preferred destinations or stored at rest in the cloud. Using custom regex rules, you can also replace sensitive data with keywords or variables so it remains queryable for investigative or monitoring purposes while being scrubbed of sensitive information.
Identify misconfigured cloud resources before they are deployed
Today’s cloud environments are more dynamic than ever before, and it can be challenging to correctly configure and provision cloud resources across interconnected services. Misconfigurations of cloud resources in your production environment—such as overprivileged cloud workloads, open network flows, and publicly exposed cloud storage instances—can give outsiders access to sensitive data or systems. Failure to secure your network perimeter and sensitive data can compromise your compliance posture and potentially result in a data breach.
To address these issues, it’s important to scan your infrastructure resources that are running in production by using a solution like Datadog Cloud Security Management (CSM) Misconfigurations, which can notify you of overprivileged IAM users, publicly accessible EC2 instances attached to privileged IAM roles, and other misconfigurations that are actively jeopardizing the security of your data and systems. CSM’s out-of-the-box compliance rules are directly mapped to one or more compliance standards or industry benchmarks. You can explore your organization’s compliance posture for each of these frameworks on CSM’s Compliance page. By inspecting a framework, you can find more information on the most critical misconfigurations affecting your compliance posture, the status of rules across different control categories that apply to the framework, and trends in how your resources comply with different rules over time. From this page, you can also select rules to triage and assign to developers amongst your team—Datadog will provide more context into why the rule is important to maintain as well as recommended remediation steps to correct the misconfiguration.
Runtime scanning helps you triage and remediate critical misconfigurations, but it doesn’t help you prevent infrastructure misconfigurations from being deployed in the first place, especially if they are the result of human error. In addition to runtime detection, you can also shift left by statically scanning changes to your infrastructure-as-code (IaC) files. This helps you detect infrastructure misconfigurations much earlier in the deployment cycle so your developers can detect and remediate them before they’re deployed.
Datadog CSM offers IaC Security (available in Preview), which integrates with GitHub to scan your Terraform configuration files and flag misconfigurations, such as exposed network ports, directly in your team’s pull requests, as well as within the Explorers page in CSM.
Inspecting a misconfiguration gives you more detail into which lines of code are responsible for the finding—by clicking on the linked rule, you can learn more about the specific violation and how to address it. In the example below, IaC Security has detected a security group configuration that exposes a sensitive port to the entire network, which can be addressed by restricting network access to a trusted IP.
Similarly, policy-as-code tools can help prevent non-compliant configurations from being deployed. Policy as code enables you to define permissions as policies—i.e., you define the actions that different entities are explicitly allowed to perform or prohibited from performing on different resources. You can then use these policies to enforce compliant configurations in CI/CD, as well as dynamically authorize or deny requests for live systems. This combination of shift-left static analysis and runtime authorization helps you protect your production cloud resources from unauthorized access by enforcing role-based and attribute-based access control—for example, blocking users without admin permissions from performing write actions on records—and by preventing the deployment of configurations that violate your defined policies.
Shifting misconfiguration detection left by implementing IaC scanning and policy as code creates built-in safety checks within your deployment pipelines to help engineers catch misconfigurations before they affect your production cloud resources. This provides an additional layer of security and reduces the time and effort spent on remediation, since issues are typically much easier and quicker to fix before they go live. IaC and policy as code are also highly auditable, as both types of tools provide an immutable trail that shows when misconfigurations were first detected and what policy decision was made.
Analyze third-party libraries and source code for vulnerabilities
Each time your engineering teams incorporate new third-party libraries to help build applications, they introduce the potential for new security risks. In research conducted by Datadog, we found that over 60 percent of services with over 300 dependencies had at least one critical vulnerability (as defined by the Common Vulnerability Scoring System). Not only can vulnerabilities introduced by third-party libraries lead to data breaches, they can also result in financial repercussions—most compliance frameworks do not discriminate between first-party and third-party vulnerabilities, and they may levy fines if you fail to address known vulnerabilities by deploying an internal fix or updating to the latest security patch (typically within a month of release).
Datadog Software Composition Analysis (SCA) scans the open source libraries used across your repositories and running services to help you catch vulnerable code in development, in addition to identifying exploitable vulnerabilities in production. You can configure SCA to statically scan your repositories by directly running scans on Datadog’s infrastructure or by running as a GitHub Actions job. By integrating this step into your CI/CD pipelines, SCA can help you automatically identify commits that introduce vulnerable dependencies and notify your engineers so they can apply a fix before the changes reach production.
Still, healthy application code can become vulnerable over time through newly uncovered exploits, supply chain attacks, deprecated projects, and more. SCA helps protect your running services against new and emerging vulnerabilities in open source libraries by using a database of known issues sourced from Open Source Vulnerabilities (OSV), the National Vulnerability Database (NVD), GitHub advisories, and our security researchers’ own findings. You can find a real-time summary of your organization’s security posture in the SCA summary page, which groups open vulnerabilities based on risk—for example, those that are in production, at risk of exploitation, and are currently being used to attack your services—so you can prioritize your remediation efforts.
SCA also helps you simplify vulnerability resolution. After identifying a vulnerability to address, you can view a description of what exactly is responsible for creating the vulnerability, its impact across your services and repositories, and how to remediate the issue. Our recommended remediation steps highlight the exact lines within configuration files that need to be changed, or which packages you need to update via the package manager of your choice. If you’d like to learn more about the vulnerability, you can find resources in the More Information tab, which links out to the original advisories provided by the open source libraries.
In addition to third-party libraries, source code contributed by your engineers can also introduce vulnerabilities into your systems. You can detect and resolve insecure code earlier in the development cycle by implementing static application security testing (SAST) solutions, such as Datadog Static Code Analysis, to scan your application code changes prior to runtime.
You can use Datadog Static Code Analysis to evaluate your code against your organization’s custom rulesets or use our out-of-the-box rulesets, which encompass code style, best practices, and security for different coding languages to help your developers adhere to defined standards. Static Code Analysis scans each new commit and summarizes any detected vulnerabilities in the Vulnerabilities page. Upon inspecting a vulnerability, Datadog will provide context as to why your code presents a security risk as well as suggested code changes for remediation.
You can also enable your team to identify and correct vulnerabilities as they work by using Datadog’s IDE plugins, which scan source files as developers edit them on their local machines so best practices and security standards can be enforced in real time.
If your engineering organization wants to take a more stringent approach to blocking source code vulnerabilities and third-party vulnerabilities from reaching your repositories and environments (such as production or services that handle sensitive data), consider implementing quality gates. Quality gates enable you to block code from being merged if a known vulnerability or other type of issue has been detected. For example, the rule below from Datadog Quality Gates automatically blocks deployments that introduce a single code vulnerability violation to the repository.
Use Datadog to help your organization stay compliant
The shift-left approaches discussed in this post should not replace existing workflows, but rather build on them so that your organization can strengthen its defense-in-depth posture. For example, configuring ETL pipelines to sanitize data entering your non-production environments does not guarantee that the data traveling out of those environments will stay compliant when it reaches its destination, and using a static analyzer and enforcing quality gates won’t prevent all vulnerabilities from surfacing in your production environments. However, by implementing multiple layers of protection when sensitive data migrates and utilizing shift-left practices in combination with runtime detection, you can more confidently comply with data protection regulations and reduce critical vulnerabilities that threaten your compliance posture.
You can learn more about how Datadog helps support your compliance strategy in these blog posts, or learn about our latest security findings in this year’s State of Cloud Security report. If you don’t already have a Datadog account, sign up for a free 14-day trial today.