Improve developer experience and collaboration with Service Catalog schema version 3.0

As software ecosystems grow more complex and fragmented, organizations are finding it harder to manage the thousands of interdependencies that make up their environments. For starters, engineers are collectively struggling to uphold security and reliability standards throughout their organizations because they lack a shared view of these complex software landscapes. They also find it challenging to work together during incidents as they look for upstream and downstream software components related to the ones directly impacted. And engineers who are new to these organizations have trouble onboarding, learning how these complex systems work, and contributing to incident resolution.

Datadog Service Catalog today already helps solve these problems with respect to all the services in an organization’s software ecosystem. But engineers need even more flexibility in how they define the software assets that make up their tech stacks, to allow their colleagues to discover important elements beyond just services during incident investigations or onboarding. Engineers also need more flexibility in how they define the contextual information (such as ownership) about all these various assets. For example, some components, such as databases and messaging queues, normally have shared ownership across multiple teams—and it’s important that engineers can quickly recognize complex realities like these.

Datadog’s Service Catalog schema v3.0, now available in private beta, provides teams with an improved framework for modeling their software environments. It helps eliminate knowledge gaps and silos by making it possible for every team to easily view, discover, and understand the many interrelated components—services and others—that make up their organization’s software ecosystem. Through enhancements such as extended entity types and the ability to define relationships manually, the new version of the schema helps engineers gain an even clearer view, shared by all teams, of their tech stacks. This in turn helps teams better cooperate during incidents and more easily communicate architecture knowledge to new team members.

In this blog post, you will learn how you can do the following with the Service Catalog schema v3.0:

• Leverage developer knowledge of complex architectures to improve collaboration across teams
• Create and automatically validate your entity definitions with the IDE plugin
• Automate new entity setup with Software Templates

Leverage developer knowledge of complex architectures to improve collaboration across teams

Service Catalog gathers information about services automatically from APM and USM, which helps teams gain a quick overview of their environments. Service Catalog schema v3.0 now also allows users to manually supplement and override this automatically derived data. With v3.0, teams can update Service Catalog metadata to better reflect institutional knowledge and an organization’s shared understanding of their software environment.

Define a range of entity types

Service Catalog schema v3.0 allows you to define additional types of software entities beyond services. In this newer schema version, services are just one of three possible component types that you can define, alongside datastores and queues. These three component types can also be added to the definitions of new system entities, which act as logical containers to help sort and organize components. Finally, all these various entities—services, datastores, queues, and systems—are defined within metadata as values assigned to the kind field:

• kind:service
• kind:datastore
• kind:queue
• kind:system

For instance, the configuration in the following YAML file defines a datastore component:

apiVersion: v3
kind: datastore
metadata:
  name: mydatastore
  displayName: My Datastore
  owner: AppTeam
  additionalOwners:
    - name: DbaTeam
      type: operator

This ability to define a range of entity types gives your teams greater flexibility to define your Service Catalog in whichever way makes the most sense within the context of your organization. And it lets you give all teams more granular detail about the essential components that make up your organization’s software environment. These features benefit incident response by promoting more efficient research, better information sharing, and faster MTTR.

Map relationships

In version 3.0 of the Service Catalog schema, relationship mapping now combines automatic and manual capabilities. As with the previous version of the Service Catalog schema, the relationships and dependencies are first mapped automatically by using APM and USM data. But the 3.0 version of the schema now also supports manual configuration for services, datastores, and queues to augment the auto-detected application topology. This ability to update relationships manually helps ensure that relationships among these entities are represented in the way that best reflects common knowledge among teams.

For services, datastores, and queues, you can manually define dependency information by specifying values for the following new keys within the spec section:

• dependsOn (specifies a dependency, i.e., serviceA depends on serviceB)
• ownedBy (i.e., serviceA is owned by TeamA)
• partOf (i.e., serviceA is a part of systemA)

For example, the following service definition specifies a value for the dependsOn key:

apiVersion:v3
kind:service
metadata:
 name:web-store
spec:
  dependsOn: 
    - service:cws-webapp

Because the dependency cws-webapp is configured manually in the config file, it appears as such in Service Catalog even when the Include Detected option is disabled, as shown below:

And when the Include Detected option is enabled, cws-webapp appears above all the auto-detected dependent services, as shown here:

Define systems, components, and inheritance

With the previous version of the Service Catalog schema, metadata had to be defined at the service level, i.e., via individual service definitions. With schema v3.0, we now allow inheritance, so you can specify the ownership information at the system level and have the system’s child components inherit its metadata.

To define components within a system, you can specify values for the components key in the spec field of the entity’s definition. An example of this is shown in the entity.datadog.yaml file below, which creates a system entity payments-core and defines five component services:

apiVersion:v3
kind:system
metadata:
  name:payments-core
  displayName:Payments
  tags:
       - tag:value
spec:
components:
- service:api.payment.com
- service:fraud-prevention-processor
- service:purchase-processor
- service:payments-backend
- service:fraud-detector

The following image shows what appears in the Service Catalog as a result of this configuration: the user-defined system payments-core is made up of the five component services defined in the YAML file.

Components that are defined within a system’s metadata automatically inherit that system’s metadata. You can also explicitly define inheritance from another component by specifying a value for the inheritFrom key in an item’s metadata definition in the metadata field.

These new inheritance features make it much easier for teams to populate data for entities at scale because now, it’s no longer necessary to configure metadata for each entity individually.

Specify multi-ownership

Service Catalog schema v3.0 allows you to assign multiple owners to any entity within your system by defining values for owner and additionalOwners in the metadata section, as shown below:

apiVersion: v3
kind: service
metadata:
  name: shopping-cart
  displayName: Shopping Cart
  inheritFrom: service:otherService
  tags:
    - tag:value
  owner: myteam
  additionalOwners:
    - name: opsTeam
      type: operator

This new feature helps ensure that, when components have multiple owners, the ownership can be accurately captured. It also helps ensure that teams have multiple people to contact when any one contact is unavailable.

Configure custom filters for logs and events

Within the datadog section in v3.0 schema, you can specify custom filters for logs and events associated with each entity. For example, a custom filter such as status:warn AND system:shopist would allow you to display the results of that complex search in the catalog and retrieve the associated logs and events for that system. This new capability enables teams to easily surface the telemetry data that they need the most.

apiVersion: v3
<!-- snip -->  
datadog:
  events:
    - name: "Elastic Search Activities"
      query: "site:shopist.io source:elasticsearch"
    - name: "Chef Pipelines"
      query: "env:prod source:chef role:corp-site"
  logs:
    - name: "critical logs"
      query: "source:java status:error env:prod"
    - name: "ops logs"
      query: "service:(web-store OR address-service) status:info"

Note that all systems come with a default query out of the box. This default query collects logs and events linked to the service tag values. When you customize the queries, Service Catalog will automatically display the queried logs and events for a tailored view of your systems, as in the following example:

Create and automatically validate your entity definitions with the IDE plugin

Datadog’s IDE plugin follows the JSON schema specification, which enables your IDE to autocomplete and validate your metadata as you edit it. The plugin has been updated to support the v3.0 schema and includes information that will tell your IDEs which filenames it applies to—specifically, the files entity.datadog.yaml, entity.datadog.yml, and entity.datadog.json. This update makes it easy to quickly and accurately complete your entity definition file without needing to switch contexts and consult documentation. In the example below, you can see how the IDE automatically creates keys as you fill out the entity definition.

The IDE plugin also automatically validates your entity definition to prevent errors and help ensure that your values are formatted correctly. This helps you get ahead of issues with your entity definition before it reaches the Service Catalog. In the screenshot below, for example, Visual Studio Code has recognized an issue with the contact email address, allowing you to fix the formatting before publishing the file.

Automate new entity setup with Software Templates

As engineering teams define and enrich their software components with more contextual information through the v3.0 schema, it’s important that they use proper syntax in entity definitions. But with more manually declared data allowed in v3.0, there’s more potential for human error. Using Datadog’s new Software Templates feature helps mitigate this concern. Through Software Templates, you can automate end-to-end processes that (among other capabilities) enable dynamic and self-service workflows integrated with Service Catalog. In the context of v3.0, Software Templates allows engineers to create entities through a template that can pre-populate information and ensure that metadata is captured correctly.

You can use App Builder to set up a new software template and guide developers through a self-service process of creating a new component according to your best practices.

In the example below, a simple form on App Builder helps guide a developer through the process of creating a new software component with the proper metadata. This form populates data fields based on the user inputs, which are automatically translated to proper syntax. This reduces the possibility for misconfigurations and enables the new component to correctly appear in Service Catalog on day one.

You can access the Software Templates feature by signing up for the private beta.

Give developers a comprehensive and customizable view of your software environment

The enhancements in Datadog’s Service Catalog metadata schema v3.0 give your teams a more comprehensive approach to monitoring their software environment. By extending observability to a wider range of software assets, introducing more manual metadata controls to supplement automatic detection, and enabling metadata inheritance for components, the new schema empowers teams to spot and address problems in their environments more quickly and accurately.

You can use our documentation to learn more about the features of the Service Catalog schema v3.0. Or you can view the full schema on GitHub.

If you’re not yet a Datadog user, you can sign up for a 14-day free trial.