Create Golden Paths for your development teams with Datadog App Builder and Workflow Automation

Improving the developer experience is a chief concern for many orgs who must maintain highly complex software architectures and platforms supported by an intricate web of internal processes. Platform engineering for Golden Paths seeks to address this by providing self-service tools, capabilities, and processes to help engineers start new projects in a more standardized, less mistake-prone way.

Project scaffolding tools like cookiecutter support a user-friendly approach for templatizing the creation of new projects to help you create Golden Paths for your organization. You can use Datadog Workflow Automation and App Builder in conjunction with cookiecutter to create an extensible, self-service UI for creating projects from Golden Path templates.

In this post, we’ll discuss some of the key advantages of Golden Path project scaffolding, explore how to use Datadog Workflows to both generate new cookiecutter templates and branch off existing ones, and show how you can use App Builder to create a self-service UI for scaffolding new projects.

Simplify project setup by defining Golden Paths

The complexity of the DevOps technical ecosystem and its plethora of deployment patterns and requirements can introduce friction for development teams. When developers own infrastructure, monitoring, and operations logic (in addition to application code), using templated infrastructure and configurations can help them create new projects more efficiently. Particularly in microservices-oriented architecture owned by distributed teams, standardized project scaffolding can help collapse knowledge silos, streamline onboarding, and make it easier for teams to get their services working together.

Golden Path templates address this challenge by offering a framework for organizations to maintain prepackaged compositions of well-integrated code and infrastructure capabilities. Golden Path templates typically provide:

• A repository template that engineers can quickly branch off of and get all the key resources and configurations they need
• A pipeline that can build and deploy the templated repo, including all deployment manifests (such as Helm charts, Kustomize files, etc)
• Default observability settings for traces, logs, custom metrics, and more

By defining best practices for project initialization and making it easy for developers to follow them, Golden Paths promote development velocity, standardization, and process optimization. You can help engineers at your org adopt Golden Paths by offering a workflow for creating them in your organization’s internal developer platform (IDP), alongside the service catalog and other DevEx resources.

Next, we’ll walk through how you can use Datadog Workflows and App Builder in conjunction with cookiecutter to create your own Golden Paths tool.

Create project templates with cookiecutter and Datadog Workflows

cookiecutter is a popular open source library for project templating. You can use cookiecutter to create Golden Paths that include key template assets, including:

• Infrastructure and observability manifests
• Testing frameworks
• Documentation templates
• Code formatting guidelines

You can create templates with cookiecutter in a few different ways, including the CLI, Python package, and API. Cookiecutter templates live in GitHub and can be generated from a few lines of code and a set of input parameters. For example, the following code uses the Python package to implement a Lambda function that creates a set of project files defined by the template and user parameters. Note that the helper functions recreate_template_directory and serialize_project_directory have been omitted.

def lambda_handler(event, context):
    # serialized array of template contents. Files are encoded as base64
    template_json = event['template']
    # key/value map of template variables
    template_vars = event['variables']
    
    # use request_id in the folder name to prevent leaking across requests (just in case)
    rid = context.aws_request_id
    basePath = os.path.join('/tmp/', rid)

    # deserialize template contents
    template_dir = recreate_template_directory(template_json, basePath)
   
    # config reply directory
    replay_dir = os.path.join(basePath, 'cookiecutter_reply')
    os.makedirs(replay_dir, exist_ok=True)
    # custom config file
    configPath = os.path.join(basePath, 'cookiecutter')
    os.makedirs(configPath, exist_ok=True)
    configFile = os.path.join(configPath, 'custom-config.yaml')
    with open(configFile, 'w') as f:
        f.write('replay_dir: "{}"\ncookiecutters_dir: "{}"'.format(replay_dir,basePath))
    os.environ['COOKIECUTTER_CONFIG'] = configFile

    # generate project using cookiecutter
    output_dir = cookiecutter(template_dir, extra_context=template_vars, no_input=True, output_dir=basePath)

    # serialize the generated project
    generated_project_json = serialize_project_directory(output_dir)

    # cleanup
    shutil.rmtree(template_dir)  
    shutil.rmtree(output_dir)

    return generated_project_json 

By using Datadog Workflows, you can automate the process of branching off of an existing cookiecutter template to create a new project for your engineers. Let’s create a Workflow that implements the following steps:

• Download a project template from GitHub
• Generate new project files from the template using cookiecutter
• Deploy the completed project back into GitHub, ready for development

For the first step, we’ll create a Workflow that uses a series of GitHub API requests to fetch the template repo’s tree object and files, then uses a script to write all these files into a local working directory according to the hierarchy specified by the tree. The following screenshot shows this workflow.

Next, we’ll create a Lambda function to run the Python code we showed earlier to generate the project using the downloaded template files and user parameters.

Finally, we’ll create another Workflow that uses the GitHub API to create a new branch in the target repo, uploads the generated cookiecutter project files, and opens a pull request that a team member can approve to merge in the new project.

We can also add a Slack step to the end, so that the user is notified in a direct message once the PR is open.

Now, we have a complete workflow that lets team members generate new projects using cookiecutter’s templating logic at the touch of a button. Next, we’ll explore how to use App Builder to wrap this functionality in a more accessible, user-friendly UI.

Create a templating app for your team with App Builder

With Datadog App Builder, you can create a single app that leverages all of the workflows we’ve discussed so that your team members can generate new projects from a self-serve UI. App Builder can call Workflows to take specific actions by feeding it the user-submitted parameters.

Let’s build a UI using App Builder that users can submit all their project details through. We’ll add forms for the project info (cookiecutter template path, project name, description, and owner) and destination (repo, path, and slug).

On the backend, users should be able to press a button and submit this info to the app, which will then pass it all through our templating workflow. The following query triggers the workflow and passes in all the info from the form. It’s set up to run when the “Generate Project” button is pressed.

Our app hides all the complexity involved in passing in parameters to the workflow, running the cookiecutter script, and performing actions with the GitHub API. Now, users need only to input a few details to create a new project within your organization’s systematized templating.

Make your templates discoverable

Let’s say your organization has several templates with different versions that are owned by disparate teams. We can make the app even more useful by adding a layer to the UI that enables users to sort and filter the template list to find templates without knowing the specific repository URL offhand.

Let’s add a list that sorts the templates by ID, name, author, version, and creation date, and a search bar for querying templates. By adding a column to sort the templates by creation date, we can enable users to quickly surface recently created templates.

Next, we’ll add the template creation form we created earlier as a modal that launches when a user clicks the “Use Template” button for the in the “Actions” column.

To support this on the backend, we need a simple database for storing all the project information in our list. App Builder’s included DynamoDB extension facilitates this by letting us query a DynamoDB instance to populate the table.

Now, even as your organization defines templates for hundreds of standardized development flows, your App can help engineers launch any of them. To make the app more discoverable and convenient to use, you can embed it in a dashboard. Your dashboard can include template variables that enable users to filter the template list to the ones for their team, for example.

Improve your organization’s developer experience

By defining organization-wide best practices for project scaffolding and making it easier for developers to follow them, Golden Paths help your development teams ship code faster and with fewer hiccups. Workflows and App Builder can help you implement Golden Paths for your organization by building a simple self-service interface for creating new templated projects. The Golden Paths app we’ve demonstrated in this post can help your teams more easily access templates and ensure that engineers can spin up new projects more efficiently. You can find it on GitHub.

Datadog Workflows and App Builder are generally available for all Datadog customers. For more help getting started with project scaffolding, see the dedicated blueprints for both App Builder and Workflows.

Additionally, you can use Service Catalog to track relevant templates at the service level alongside SLOs, monitoring data, scorecards, software delivery insights, and more. You can access Service Catalog’s new Software Templates feature by signing up for the private beta.

If you’re brand new to Datadog, sign up for a free trial.