Taming the complexities of hybrid cloud

The United States Department of Agriculture’s (USDA)’s Digital Infrastructure Services Center (DISC) is responsible for managing and providing essential IT services for more than a dozen US federal government departments and bureaus. DISC has operated as a federated data center since 1973, meaning they employ a network of distributed data centers that work together seamlessly.

Over the years, DISC has embarked on a modernization journey that aims to enhance its own capabilities and improve the operational efficiency of its customers across the federal community. As a result of its recent modernization initiative, DISC has adopted a hybrid cloud infrastructure that utilizes both on-premise and cloud-based resources, seamlessly integrating new capabilities with existing legacy technologies. This has led to significant efficiencies like heightened scalability, improved resource utilization, and streamlined deployment processes. However, this transformation introduced new complexities and challenges. Overcoming these hurdles would require more sophisticated monitoring capabilities than what DISC's existing tools could provide. Moreover, compliance with the 2021 Executive Order (EO) focused on improving the nation's cybersecurity rendered many legacy monitoring tools noncompliant with federal security regulations.

“When the cybersecurity EO was issued in 2021, it caused us to stop and reevaluate the security and integrity of our entire software supply chain,” says Chris Condon, Director of Enterprise Observability at ECCO Select, the trusted MSP supporting the contract.

“Suddenly, our ‘good enough’ monitoring solution was no longer compliant and we were forced into a state of change. We took the opportunity to think about our ideal state and began evaluating best-in-class observability platforms,” he says.

Enhancing DISC operations with Datadog's federally authorized observability

To meet DISC requirements, ECCO Select evaluated FedRAMP®-authorized monitoring and observability solutions and selected Datadog as its end-to-end observability platform, allowing it to monitor and manage all aspects of its IT environment, from its foundational infrastructure to the delivery of end-user experiences.

With Datadog in place, ECCO Select can offer essential managed services that provide real-time visibility into the health, performance, and configuration of IT services and assets. These systems are not only observable at a macro level by DISC, but individual tenants can use the platform’s self-service features to securely access and monitor their specific infrastructure resources and applications. DevOps teams across DISC’s portfolio of customers now have unified visibility into the health of their IT systems and can observe the performance of systems, servers, applications, and other resources within the enterprise technology stack.

Attaining rapid and tangible results

Within 75 days of the project start date, the ECCO Select team completed an automated deployment of the Datadog Agent to 4,800 hosts running in both on-prem data centers and cloud-hosted environments. This gave them full monitoring coverage of over 95 percent of their infrastructure, including around 4,000 containers as well as network and storage devices and databases. They were also able to seamlessly transition over 1,100 monitor templates, covering infrastructure services, logs, and synthetic tests, from the legacy monitoring system. This process was validated through a carefully phased approach to ensure accuracy and reliability.

“We now have a comprehensive solution that not only speeds up root cause analysis when there’s an issue, but continuously provides the visibility we need to keep our systems secure and resilient.”

ECCO Select’s infrastructure support team realized the value of Datadog almost immediately when it helped them detect and resolve a recurring memory issue that had been plaguing them for months. They had received more than 700 service desk tickets, each requiring manual investigation. The longer the problem persisted, the more time and resources were needlessly spent at all levels of the Enterprise Service Desk team.

Whereas the legacy monitoring tool’s limitations left them in the dark, only offering "high" or "low" thresholds for memory utilization, Datadog immediately revealed a problematic process caught in a loop due to a (at the time) seemingly harmless change in a disaster recovery location. The change disrupted replication efforts, causing a widespread memory issue affecting the system enterprise wide.

Armed with this information, the team quickly identified the problem and used insights from Datadog to correlate standard high/low memory alerts with process log information to implement a straightforward fix. The previously problematic process, undetectable with older monitoring tools, was successfully identified and permanently resolved.

“Reflecting on our journey, we started with limited visibility—unable to see beyond top-level alerts, facing spotty access, and lacking meaningful log data,” says Condon.