Editor’s note: Jeremy Garcia, VP of Technical Community and Open Source at Datadog, explains why fostering an organization-wide culture and practice of DevSecOps is essential for deploying resilient, secure applications and services.
Over the past decade, DevSecOps has become a popular buzzword in the tech industry. But even before that, it had established its role in addressing a long-standing problem: siloed workstreams between DevOps and security teams, which weaken an organization’s security posture and bog down software development and release cycles.
For many organizations, the practice of securing infrastructure and applications has typically been perceived as operating from within an ivory tower. Security practices, tools, and workflows were isolated from development and operations. Developers often saw security practices as blockers that pulled them away from their projects late in the game. Likewise, security engineers had no view or understanding of development workflows or exactly what was deployed into pre-production and production environments. The accelerated emergence of DevSecOps can be attributed to its effectiveness at breaking down these barriers.
Why DevSecOps?
Early in my career, I worked as an SRE at a few organizations that had siloed security functions. That was frustrating at the time, and in retrospect I can also see that it really hurt the organizations’ security posture. There was no shared vernacular, context, or goals among developers, operations, and security folks. The result was constant tension and a decrease in both software release velocity and security coverage.
Later, as an engineering manager who was responsible for a large portion of a company’s infrastructure, I inherited a small bit of the security function for it. As I tried to build relationships and connections with the main security org, it became clear that not only were these silos deeply rooted in culture but they were reinforced and exacerbated by disparate tools, data sets, and incentives. This disconnect created teams with unaligned goals, language, and context, which made effective and collaborative security workflows—including those that are part of setting up and scaling infrastructure, creating an observability strategy and implementing monitoring tools, and responding to incidents—an almost impossible task.
Like DevOps, which evolved to break down barriers between the obfuscated and specialized worlds of developers and operations, DevSecOps is more than just a set of tools and processes. It represents a fundamental shift in culture: making security a foundational part of all workstreams. Unifying workstreams benefits organizations in several ways. It shortens the feedback loop between engineering and security teams, enabling them to troubleshoot security issues faster. It empowers engineers to contribute to security guidelines for their features and to ask questions about how to make them more secure by design, from the start. And unified workstreams give security teams the context and understanding they need to conduct higher-quality investigations and security reviews, and easily share their findings with others. Together, these benefits result in applications and infrastructure that are more resilient. And when security issues are discovered and remedied earlier, preferably before they hit production, they are less expensive to address.
DevSecOps in practice
When we talk about making security an integral part of DevOps, we often don’t know what that should look like. In practice, security can take advantage of tooling that’s already in use, such as automation—which already plays an important role in successful DevOps. For example, collaboration between security and engineering allows for a common set of rules to be defined and appropriately applied in a CI/CD pipeline, minimizing friction. This prevents vulnerabilities from propagating throughout the runtime environment, where they would be more expensive to solve—not to mention the headache it causes developers to stop working on new features and hunt down and fix the offending code or dependency.
Integrating security with DevOps also applies to monitoring. Any system that creates shared context through a unified view of telemetry data across multiple teams will not only provide greater visibility into application and infrastructure health but also security posture. It also facilitates the cultural change needed to adopt DevSecOps successfully.
One example of DevSecOps in practice is embedding security engineers into platform engineering teams. When the monitoring context is consistent, embedded teams like this have a much higher chance of success. This transition also fosters programs like Security Champions, who can advocate for security best practices within individual teams. Programs like these have been extremely successful here at Datadog as well as for our customers.
DevSecOps inside Datadog
As with any trendy buzzword, it’s easy to be skeptical of it—and I admit that I have been somewhat skeptical of the term “DevSecOps.” But it’s a term the industry is now using and a shared vernacular that everyone understands has value. What I am not skeptical of is the benefit of security, development, and operations folks working together and not being in silos. And we’ve seen the results of this here at Datadog.
We know that security and observability go hand-in-hand in building reliable applications, so we take a holistic approach to fostering DevSecOps culture and tooling both in our internal operations and throughout our products. A core goal for us is to create strong communication channels across formerly isolated teams. Having shared context for evaluating the health of our systems in both pre-production and production environments is critical for supporting our organization’s continued transformation. This also helps us ensure that developers are responsible for and—more importantly—empowered to secure the code they write.
We’ve seen firsthand how shared context and collaboration is integral to successful security practices. Our engineering and product teams conduct many of their own security reviews and share their findings with our core security team, giving them a starting place for research. The security team then shares any additional insights with engineering and product, which informs their development decisions. We also integrate security into our overall site reliability goals by creating dedicated SLIs and SLOs for vulnerability management. All of this information is available in the Datadog platform, ensuring that every team is working with the same sets of data.
“At Datadog, our security teams are structured to optimize for automation and collaboration,” says Bianca Lankford, Datadog’s Vice President of Security Engineering. “Our approach to DevSecOps starts with leveraging Datadog’s security suite, namely Cloud SIEM, configured to our own business and security requirements. Using a tool like Cloud SIEM, optimized for automation, allows us to have a SOCless model. We are able to automate both detection and response by using fine-tuned security signals. We leverage Datadog Incident Management to help auto-declare incidents for critical alerts or cases for findings that need more attention in order to triage. Throughout the entire automation cycle, our security engineers collaborate with product teams to enhance and grow our tooling. Security teams also work closely with engineering to create better auto-detections, and collaborate closely with SRE teams to ensure that security is part of the entire production support life cycle.”
These are just a few examples that show how a DevSecOps culture encourages a bidirectional pathway between security and infrastructure teams, making security more accessible throughout our organization.
Break down security silos with Datadog
With today’s evolving threat landscape, integrating and streamlining security workflows has become more important than ever. We’ve already seen the benefits of this internally, as well as its value in protecting our customers’ applications and infrastructure. To learn more about Datadog’s security offerings, check out our security documentation. If you don’t already have a Datadog account, you can sign up for a free 14-day trial today.
