Improve developer experience and collaboration with Software Catalog

As software ecosystems grow more complex and fragmented, organizations are finding it harder to manage the thousands of interdependencies that make up their environments. For starters, engineers are collectively struggling to uphold security and reliability standards throughout their organizations because they lack a shared view of these complex software landscapes. They also find it challenging to work together during incidents as they look for upstream and downstream software components related to the ones directly impacted. And engineers who are new to these organizations have trouble onboarding, learning how these complex systems work, and contributing to incident resolution.

Datadog Service Catalog has long helped solve these problems with respect to all the services in an organization’s software ecosystem. But engineers need even more flexibility in how they define the software assets that make up their tech stacks, to allow their colleagues to discover important elements beyond just services during incident investigations or onboarding. Engineers also need more flexibility in how they define the contextual information (such as ownership) about all these various assets. For example, some components, such as databases and messaging queues, normally have shared ownership across multiple teams—and it’s important that engineers can quickly recognize complex realities like these.

That’s why we’re evolving Service Catalog into Software Catalog. Software Catalog provides teams with an improved framework for modeling their software environments. It helps eliminate knowledge gaps and silos by making it possible for teams to easily view, discover, and understand the many interrelated components—services and others—that make up their organization’s software ecosystem. Through enhancements such as extended entity types and the ability to define relationships manually, Software Catalog helps engineers gain an even clearer view, shared by all teams, of their tech stacks. This in turn helps teams better cooperate during incidents and more easily communicate architecture knowledge to new team members.

In this blog post, you will learn how you can do the following with Software Catalog:

• Use developer knowledge of complex architectures to improve collaboration across teams
• Create and automatically validate your entity definitions with the IDE plugin
• Automate new entity setup with self-service actions

Use developer knowledge of complex architectures to improve collaboration across teams

Software Catalog gathers information about services automatically from APM and USM, which helps teams gain a quick overview of their environments. Now users can supplement and override this automatically derived data. Teams can update Software Catalog metadata to better reflect institutional knowledge and their organization’s shared understanding of their software environment.

Define a range of entity types

Software Catalog allows you to define additional types of software entities beyond services. In this newer schema version, services are just one of three possible component types that you can define, alongside datastores and queues. These three component types can also be added to the definitions of new system entities, which act as logical containers to help sort and organize components. Finally, all these various entities—services, datastores, queues, and systems—are defined within metadata as values assigned to the kind field:

• kind:service
• kind:datastore
• kind:queue
• kind:system

For instance, the configuration in the following YAML file defines a datastore component:

apiVersion: v3
kind: datastore
metadata:
  name: mydatastore
  displayName: My Datastore
  owner: AppTeam
  additionalOwners:
    - name: DbaTeam
      type: operator

This ability to define a range of entity types gives your teams greater flexibility to define your Sofware Catalog in whichever way makes the most sense within the context of your organization. And it lets you give all teams more granular detail about the essential components that make up your organization’s software environment. These features benefit incident response by promoting more efficient research, better information sharing, and faster MTTR.

Map relationships

Now in Software Catalog, relationship mapping combines automatic and manual capabilities. Previously, in Service Catalog, the relationships and dependencies were first mapped automatically by using APM and USM data. But now Software Catalog supports manual configuration for services, datastores, and queues to augment the auto-detected application topology. This ability to update relationships manually helps ensure that relationships among these entities are represented in the way that best reflects common knowledge among teams.

For services, datastores, and queues, you can manually define dependency information by specifying values for the following new keys within the spec section:

• dependsOn (specifies a dependency, e.g., serviceA depends on serviceB)
• ownedBy (e.g., serviceA is owned by TeamA)
• partOf (e.g., serviceA is a part of systemA)

For example, the following service definition specifies a value for the dependsOn key:

apiVersion:v3
kind:service
metadata:
 name:auth-dotnet
spec:
  dependsOn: 
    - service:user-auth

Because the dependency auth-dotnet is configured manually in the config file, it appears as a dependency in Software Catalog even when the Include Detected option is disabled, as shown below:

And when the Include Detected option is enabled, auth-dotnet-postgres appears as an auto-detected dependent service, as shown here:

Define systems, components, and inheritance

Previously, in Service Catalog, metadata had to be defined at the service level, i.e., via individual service definitions. With Software Catalog, we now allow inheritance, so you can specify the ownership information at the system level and have the system’s child components inherit its metadata.

To define components within a system, you can specify values for the components key in the spec field of the entity’s definition. An example of this is shown in the entity.datadog.yaml file below, which creates a system entity payments-core (display name Payments) and defines three component services:

apiVersion:v3
kind:system
metadata:
  name:payments-core
  displayName:Payments
  tags:
       - tag:value
spec:
components:
- service:api.payment.com
- service:fraud-prevention-processor
- service:purchase-processor

The following image shows what appears in Software Catalog as a result of this configuration: The user-defined system Payments is made up of the three component services defined in the YAML file.

Components that are defined within a system’s metadata automatically inherit that system’s metadata. You can also explicitly define inheritance from another component by specifying a value for the inheritFrom key in an item’s metadata definition in the metadata field.

These new inheritance features make it much easier for teams to populate data for entities at scale because now, it’s no longer necessary to configure metadata for each entity individually.

Specify multi-ownership

Software Catalog allows you to assign multiple owners to any entity within your system by defining values for owner and additionalOwners in the metadata section, as shown below:

apiVersion: v3
kind: service
metadata:
  name: shopping-cart
  displayName: Shopping Cart
  inheritFrom: service:otherService
  tags:
    - tag:value
  owner: myteam
  additionalOwners:
    - name: opsTeam
      type: operator

This new feature helps ensure that when components have multiple owners, the ownership can be accurately captured. It also helps ensure that teams have multiple people to contact when any one contact is unavailable.

Configure custom filters for logs and events

Within the datadog section, you can now also specify custom filters for logs and events associated with each entity. For example, a custom filter such as status:warn AND system:shopist would allow you to display the results of that complex search in Software Catalog and retrieve the associated logs and events for that system. This new capability enables teams to easily surface the telemetry data that they need the most.

apiVersion: v3
<!-- snip -->  
datadog:
  events:
    - name: "Elastic Search Activities"
      query: "site:shopist.io source:elasticsearch"
    - name: "Chef Pipelines"
      query: "env:prod source:chef role:corp-site"
  logs:
    - name: "critical logs"
      query: "source:java status:error env:prod"
    - name: "ops logs"
      query: "service:(web-store OR address-service) status:info"

Note that all systems come with a default query out of the box. This default query collects logs and events linked to the service tag values. When you customize the queries, Software Catalog will automatically display the queried logs and events for a tailored view of your systems, as in the following example:

Create and automatically validate your entity definitions with the IDE plugin

Datadog’s IDE plugin follows the JSON schema specification, which enables your IDE to autocomplete and validate your metadata as you edit it. The plugin has been updated to support the new Software Catalog and includes information that will tell your IDEs which filenames it applies to—specifically, the files entity.datadog.yaml, entity.datadog.yml, and entity.datadog.json. This update makes it easy to quickly and accurately complete your entity definition file without needing to switch contexts and consult documentation. In the example below, you can see how the IDE automatically creates keys as you fill out the entity definition.

The IDE plugin also automatically validates your entity definition to prevent errors and help ensure that your values are formatted correctly. This helps you get ahead of issues with your entity definition before it reaches Software Catalog. In the screenshot below, for example, Visual Studio Code has recognized an issue with the contact email address, allowing you to fix the formatting before publishing the file.

Automate new entity setup with self-service actions

As engineering teams define and enrich their software components with more contextual information through Software Catalog, it’s important that teams define the right metadata for their components. But with more manually declared data allowed in Software Catalog, there’s more potential for human error. Through the new Self-Service tab in Software Catalog, you can automate end-to-end processes that (among other capabilities) enable dynamic and self-service workflows integrated with Software Catalog. Self-service actions allow engineers to create entities through a template that can pre-populate information and ensure that metadata is captured correctly.

You can use these self-service actions, for example, to set up a new software template and guide developers through a process of creating a new component according to your best practices.

In the example below, a simple form created through App Builder and made available through the Self-Service tab helps guide a developer through the process of creating a new software component with the proper metadata. This form populates data fields based on the user inputs, which are automatically translated to proper syntax. This reduces the possibility for misconfigurations and enables the new component to correctly appear in Software Catalog on day one.

Give developers a comprehensive and customizable view of your software environment

The expanded capabilities of Datadog’s Software Catalog, formerly Service Catalog, give your teams a more comprehensive approach to monitoring their software environments. By extending observability to a wider range of software assets, introducing more manual metadata controls to supplement automatic detection, and enabling metadata inheritance for components, Software Catalog empowers teams to spot and address problems in their environments more quickly and accurately.

You can use our documentation to learn more about the features of Software Catalog. And if you’re not yet a Datadog user, you can sign up for a 14-day free trial.