Monitoring Rails applications with Datadog

Rails is a Ruby framework for developing web applications. It favors the Model-View-Controller (MVC) architecture and includes generators that create the files needed for each MVC component. Rails applications consist of a database, an application server for running application code, and a web server for processing requests. Rails provides multiple integrations for its supporting database (e.g., MySQL and PostgreSQL) and web server (e.g., Apache and NGINX). It also supports a few different application servers: Passenger is a popular choice, as it offers more features than Rails’s built-in application server (Puma) and integrates easily with both Apache and NGINX. Passenger acts as a module for the web server, running application code while the server handles HTTP requests.

In this post, we’ll show you how to use Datadog to collect metrics, request traces, and logs from a Rails application running on Passenger, Apache, and MySQL. (Though we will focus on these integrations in this guide, you can also monitor NGINX servers and PostgreSQL database systems with Datadog.) This post will cover:

• Installing and configuring the Datadog Agent
• Capturing Passenger and Rails logs
• Enabling Datadog’s Apache integration
• Enabling Datadog’s MySQL integration
• Creating custom dashboards and alerts

Note that this guide includes commands for Linux hosts, so some commands may differ if you are using another operating system.

Getting started: Installing the Datadog Agent

In order to begin collecting logs, traces, and metrics from your Rails application, you will need to download and install the Datadog Agent on the application host. This step requires an API key; if you have a Datadog account, you can find the key in your account. Or, if you’d like to follow the steps in this guide but don’t yet have a Datadog account, you can start a free trial.

Run the following script to install the Agent:

DD_API_KEY=<API_KEY> DD_AGENT_MAJOR_VERSION=7 bash -c "$(curl -L https://raw.githubusercontent.com/DataDog/datadog-agent/master/cmd/agent/install_script.sh)"

Check the Agent status with:

sudo datadog-agent status

If the installation was successful, you will see information about the running Agent similar to the following output:

Getting the status from the agent.

===============
Agent (v7.39.2)
===============

  Status date: 2022-11-01 15:11:19.37 UTC (1667315479370)
  Agent start: 2022-11-01 15:10:09.39 UTC (1667315409390)
  Pid: 5171
  Go Version: go1.17.11
  Python Version: 3.8.14
  Build arch: amd64
  Agent flavor: agent
  Check Runners: 4
  Log Level: info

  Paths
  =====
    Config File: /etc/datadog-agent/datadog.yaml
    conf.d: /etc/datadog-agent/conf.d
    checks.d: /etc/datadog-agent/checks.d

  [...]

Note that Go and Python versions may vary depending on which version of the Agent you installed.

Configure the Agent to collect request traces and logs

Datadog APM provides detailed insight into service-level performance by enabling you to trace requests across every service in your application. Tracing comes prepackaged with version 6 of the Datadog Agent and is enabled by default.

The Datadog Agent also collects logs generated by your application and forwards them alongside your metrics and request traces. Log collection requires version 6+ of the Datadog Agent. You can configure both APM and log management by editing the datadog.yaml file located in your host’s /etc/datadog-agent/ directory. It’s a good practice to add an environment tag in the APM configuration to better categorize the data coming from your application:

# Trace Agent Specific Settings
apm_config:
  enabled: true
  env: staging

The Agent doesn’t collect logs by default, so you will need to enable log collection in the same datadog.yaml file:

# Logs agent
logs_enabled: true

Save the file and restart the Agent to apply your recent changes:

sudo service datadog-agent restart

Set up Rails application performance monitoring

Datadog APM auto-instruments your Rails app with minimal setup so you can quickly start tracing activity such as server requests, database calls, and template renderings.

Datadog uses the ddtrace gem and a Rails initializer file for instrumenting your application. Add the following gem to your Gemfile:

source 'https://rubygems.org'
gem 'ddtrace', require: 'ddtrace/auto_instrument'

And then install it with Bundler:

bundle install

Next, create a datadog.rb file in your application’s config/initializers directory:

require 'ddtrace'

Datadog.configure do |c|
    c.tracing.instrument :rails, service_name: 'staging-rails-app'
end

The service_name parameter sets the name for each service you see in Datadog. After you save the file, you should start seeing services for your running Rails application on the APM services page of your Datadog account. Since you specified an environment in your APM configuration (env:staging), you can select that tag to find your Rails services.

In the service list above, you can see that Datadog is now collecting request traces and performance metrics from two services: the application server (staging-rails-app) and the MySQL database (staging-rails-app-mysql2). For integrations such as MySQL, Datadog will automatically append an integration alias to the service name. In Datadog, a service is simply a set of processes that accomplish the same job, like serving up a web application or querying a database. Datadog APM automatically generates dashboards that display key metrics such as latency, errors, and request rates for every instrumented service.

APM also collects performance metrics from your application at the resource level. Resources are operations related to a service and can include MySQL database queries or Rails application controller actions. APM captures and groups these resources accordingly, giving you a more comprehensive view into how your application is handling specific actions and different types of requests.

If one of the resources raises an error, you will see that reflected in the Errors and Error Rate columns. In the example image above, the MicropostsController#destroy resource shows a 66.7 percent error rate. You can click on the resource to view more information about latency, error rates, and request counts for that specific resource. Each resource includes a list of recent traces where you can see the status and latency breakdown for individual requests.

You can then select an individual trace to inspect a flame graph and stack trace of the associated error, which enables you to pinpoint the exact source of the issue. In the example below, you can see that an error is thrown when the destroy method at microposts_controller.rb:17 is called.

You can navigate to the microposts_controller.rb file and drill down to the exact line number shown in the stack trace to find and address the statement causing the error. In this case, the error shown in the example above is caused by a custom raise "Could not delete post" statement used to force application errors for testing and is no longer needed. With Datadog APM, you can easily monitor the performance of your application and troubleshoot errors as they occur. You can also build on the out-of-the box tracing that Datadog APM provides for Rails applications by instrumenting your code to add more detail to your request traces. Individual request traces, like what is shown in the image above, include a Logs tab that shows any relevant logs the Agent collects. Logs give you greater visibility into the state of each of your application’s integrations.

Capture Passenger and Rails Logs

You can set up Datadog to collect database, server, and application logs. As you’ll see later in this guide, many Datadog infrastructure integrations support log collection out of the box, and the Agent already includes the configuration files needed to start collecting logs. For other sources (like Rails and Passenger), you can create your own config files to enable log collection with the Agent. Rails and Passenger logs capture activity such as database queries and Ruby exceptions.

First, create a new directory and YAML file named after your log source in the Agent’s conf.d directory. This applies to both your Passenger and application logs:

cd /etc/datadog-agent/conf.d/
mkdir ruby.d passenger.d
touch ruby.d/conf.yaml passenger.d/conf.yaml

Include the following information in the ruby.d/conf.yaml file:

# Log Information
logs:
  - type: file
    path: /path/to/app/log/development.log
    service: staging-rails-app
    source: ruby

The source parameter is connected to Datadog’s log integration pipelines and facets so we recommend setting this to ruby for your Rails logs. This will install the built-in Ruby pipeline and integration facets so you can group, search for, and customize all the logs coming from the application. The service tag connects logs to request traces in Datadog APM, so make sure that the value you set for your logs matches the associated service in APM (staging-rails-app, in this case). This enables you to quickly jump from an individual log to correlated performance data in APM.

In the passenger.d/conf.yaml file, add the following snippet to configure log collection for Passenger:

# Log Information
logs:
  - type: file
    path: /path/to/app/log/passenger.3000.log
    service: staging-rails-app
    source: passenger

This will connect Passenger logs to the same staging-rails-app service and set Passenger as the source for any processing rules. For sources like Passenger, you can create custom processing pipelines to capture all associated logs based on the passenger source parameter. This will set custom rules around Passenger logs only so you can easily separate them from your application logs, giving you greater control over how Datadog parses them.

Restart the Agent to load the new configuration files:

sudo service datadog-agent restart

You should start seeing Passenger and application logs in the Log Explorer as soon as they are generated. Datadog automatically transforms key attributes about your logs, like host and service, into facets, which you can use to easily search and explore your logs in the Log Explorer.

Fine-tuning your Rails logs

Datadog automatically parses JSON-formatted logs for you, but not all applications generate logs in JSON format. Rails generates plain-text, multi-line logs, so they may require additional processing in order to capture all the data generated from one log entry. For example, a multi-line development log may look like this for a single request:

Started HEAD "/" for 127.0.0.1 at 2018-05-25 21:36:18 +0000
Processing by StaticPagesController#home as HTML
  Rendering static_pages/home.html.erb within layouts/application
  Rendered static_pages/home.html.erb within layouts/application (692.2ms)
  Rendered layouts/_shim.html.erb (0.7ms)
  Rendered layouts/_header.html.erb (1.6ms)
  Rendered layouts/_footer.html.erb (1.0ms)
Completed 200 OK in 914ms (Views: 894.0ms | ActiveRecord: 7.03ms)

But you can use the Lograge gem to clean up the log like this:

{
   "method":"GET",
   "path":"/",
   "format":"html",
   "controller":"StaticPagesController",
   "action":"home",
   "status":200,
   "duration":914.00,
   "view":894.0,
   "db":7.03
}

Lograge is a popular library for Rails applications as it converts multi-line request logs to JSON format and provides custom formatting options. You can also use other libraries such as Log4r and Logging to format your application logs, though Lograge is specifically designed for use with Rails applications.

To start pre-processing your logs with Lograge, add gem 'lograge' to your Gemfile and install it with:

bundle install

Then, create a lograge.rb file in your project’s config/initializers folder and include the following:

Rails.application.configure do
    config.lograge.enabled = true

    # Generate log in JSON
    config.lograge.formatter = Lograge::Formatters::Json.new
    config.lograge.custom_options = lambda do |event|
      { :ddsource => ["ruby"],
        :params => event.payload[:params]
      }
    end
  end

This configures Lograge to convert each multi-line Rails log into a single line in JSON. You may need to restart the application server to see your Rails logs reported in the new format.

You can jump to dashboards related to the log’s host or source by clicking on either the host or source links at the top of the log entry. These dashboards are scoped to the time when the log was generated to provide further context around what happened. And with the service link, you can view all traces related to the service that generated the log.

Datadog automatically extracts relevant data from your JSON logs and sets them as attributes you can search and filter by, such as status code or HTTP request method. Note that some JSON logs do not have a message attribute so you may not see a message for a selected log in the Log Explorer. Learn more about customizing Datadog’s processing rules in our pipeline documentation so you can get the most out of your logs, regardless of their format.

We’ve successfully begun collecting logs and traces from a Rails application, but that’s just one part of fully comprehensive monitoring. Datadog includes integrations with MySQL, Apache, and 200+ additional technologies so you can monitor your application and infrastructure in one place. Next, we’ll cover extending your monitoring coverage by enabling MySQL and Apache integrations to collect metrics and logs from your database and web server.

Enable Datadog’s Apache integration for metrics + logs

Navigate to your account’s Integrations page and click on the Apache integration tile to enable it. The Datadog Agent collects metrics exposed through Apache’s status module, so ensure that mod_status is enabled for your Apache server and that the server’s status page is available at /server-status. To configure the Agent to collect metrics and logs from Apache, you will need to create a configuration file in the Agent’s apache.d directory. You can use the provided config template to get started quickly:

cd /etc/datadog-agent/apache.d
cp conf.yaml.example conf.yaml

Edit the new conf.yaml file to add the server status URL for your application and enable log collection from Apache’s access and error logs. The final result should look similar to the following:

init_config:

instances:
  - apache_status_url: http://localhost/server-status?auto

## Log Section (Available for Agent >=6.0)
logs:
   - type: file
     path: /var/log/apache2/access.log
     service: staging-rails-app
     source: apache
     sourcecategory: http_web_access

   - type: file
     path: /var/log/apache2/error.log
     service: staging-rails-app
     source: apache
     sourcecategory: http_web_access

As with the Passenger and Rails logs above, make sure the service parameter matches the name of your service in APM so you can easily pivot between correlated data in Datadog. Setting the source parameter to apache automatically installs Datadog’s built-in Apache pipeline. The sourcecategory parameter is optional but enables you to set global tags for your services. With this, you can search for all applications, databases, or web servers that share the same sourcecategory tag.

Save the file and restart the Agent:

sudo service datadog-agent restart

The Datadog Agent will start collecting metrics from your application’s Apache server and automatically create an out-of-the-box screenboard in your dashboard list. You can view related logs by hovering over and clicking on a datapoint from one of the graphs on the screenboard.

Your Apache logs will show up in the Log Explorer alongside your Rails and Passenger logs so you can easily track processed requests from your web server with any applicable responses coming back from your application server.

Learn more about customizing logging for your web server, including how to monitor request latency, with our Apache monitoring guide.

Enable Datadog’s MySQL integration for metrics + logs

To start collecting metrics and logs from your MySQL database, you first need to create a Datadog user with replication rights in your database, as explained in our documentation. You can create a password for your user by navigating to your account’s MySQL integration, enabling it for your account, and clicking on the Configuration tab. After you generate a password, execute the following commands in a terminal session:

sudo mysql -e "CREATE USER 'datadog'@'localhost' IDENTIFIED BY '<GENERATED_PASSWORD>';"
sudo mysql -e "GRANT REPLICATION CLIENT ON *.* TO 'datadog'@'localhost' WITH MAX_USER_CONNECTIONS 5;"
sudo mysql -e "GRANT PROCESS ON *.* TO 'datadog'@'localhost';"
sudo mysql -e "GRANT SELECT ON performance_schema.* TO 'datadog'@'localhost';"

MySQL logging is disabled by default, so you’ll need to locate your database server’s configuration file (/etc/mysql/conf.d/mysql.cnf on Ubuntu) to enable it. Depending on your host’s operating system and version of MySQL, the name and location of the file may differ. Edit the file with the following to enable the general, error, and/or slow query logs:

[mysqld_safe]
log_error=/var/log/mysql/error.log
[mysqld]
general_log = on
general_log_file = /var/log/mysql/mysql.log
log_error=/var/log/mysql/error.log
slow_query_log = on
slow_query_log_file = /var/log/mysql/mysql-slow.log
long_query_time = 2
# Threshold query time in seconds for logging slow queries

Notes on collecting MySQL logs

There are two things to note with MySQL logs:

• The dd-agent user needs read access to the /etc/mysql/ and /var/log/mysql/ directories in order to forward any database metrics or logs to Datadog.
• Logs take up disk space, so Oracle recommends rotating MySQL logs in order to save space on your server, especially when you deploy an application to production. Datadog can help you determine when a server will run out of disk space with metric forecasting.

Save the file and restart the MySQL server:

service mysql restart

Next, create a configuration file in the Agent’s mysql.d configuration directory using the existing configuration template:

cd /etc/datadog-agent/conf.d/mysql.d
cp conf.yaml.example conf.yaml

Edit the new conf.yaml file to include your new database user and log paths:

init_config:

instances:
  - server: 127.0.0.1
    user: datadog
    pass: '<GENERATED_PASSWORD>'
    options:
        replication: 0
        galera_cluster: 1

#Log section
logs:
     - type: file
       path: /var/log/mysql/error.log
       service: staging-rails-app
       source: mysql
       sourcecategory: database

     - type: file
       path: /var/log/mysql/mysql-slow.log
       service: staging-rails-app
       source: mysql
       sourcecategory: database

     - type: file
       path: /var/log/mysql/mysql.log
       service: staging-rails-app
       source: mysql
       sourcecategory: database

Datadog provides an out-of-the-box pipeline for MySQL as well, so set the source parameter to mysql for your database logs to automatically route them through the appropriate pipeline. If your database generates multi-line logs, you can add additional processing rules to your log configurations. For example, for a multi-line log that begins with a yyyy-dd-mm timestamp, you can include the following rule:

 - type: file
       [...]
       log_processing_rules:
           - type: multi_line
            name: new_log_start_with_date
            pattern: \d{4}\-(0?[1-9]|1[012])\-(0?[1-9]|[12][0-9]|3[01])

sudo service datadog-agent restart

Once the integration is set up, you will immediately see key metrics from your database populating Datadog’s out-of-the-box dashboard for MySQL.

In the Log Explorer, you will see entries from your database along with logs from Rails, Apache, and Passenger. View any log by searching for facets like your service or source. For example, search for source:database to see only MySQL logs.

You can always confirm your logs are enabled with the Agent’s status command:

sudo datadog-agent status

You will see the following output if the Agent is able to tail each log successfully:

==========
Logs Agent
==========

  apache
  ------
    Type: file
    Path: /var/log/apache2/access.log
    Status: OK
    Inputs: /var/log/apache2/access.log

    Type: file
    Path: /var/log/apache2/error.log
    Status: OK
    Inputs: /var/log/apache2/error.log

  mysql
  -----
    Type: file
    Path: /var/log/mysql/error.log
    Status: OK
    Inputs: /var/log/mysql/error.log

    Type: file
    Path: /var/log/mysql/mysql-slow.log
    Status: OK
    Inputs: /var/log/mysql/mysql-slow.log

    Type: file
    Path: /var/log/mysql/mysql.log
    Status: OK
    Inputs: /var/log/mysql/mysql.log

  passenger
  ---------
    Type: file
    Path: /path/to/app/log/passenger.3000.log
    Status: OK
    Inputs: /path/to/app/log/passenger.3000.log

  ruby
  ----
    Type: file
    Path: /path/to/app/log/development.log
    Status: OK
    Inputs: /path/to/app/log/development.log

With Datadog collecting metrics and logs from your Rails application, MySQL database, and Apache server, you can monitor your infrastructure at a glance with dashboards and custom alerts.

Go further with Datadog dashboards and alerts

Datadog provides built-in dashboards for your hosts and all your integrations on the Dashboard list. Instead of looking through multiple dashboards, you can also create a custom one that includes all the data you deem most important. For example, you can include widgets for APM service summaries for your Rails application and log streams from any key components. Check out our guide on creating a custom timeboard or screenboard to learn more.

You can also set up Datadog to alert on infrastructure metrics, service-level metrics, or processed log data with monitors. For example, you can configure alerts to notify you when they detect sudden drops in application requests or a high error rate for individual resources. Datadog can even automatically notify you of metric outliers and anomalies and forecast trends as your application grows. This is especially useful for establishing a process for rotating logs in order to save disk space on your servers.

Get started with Rails + Datadog

With Datadog, you can monitor metrics, traces, and logs across every layer of your Rails stack. By correlating data between customizable dashboards and alerts, you can easily identify and troubleshoot any performance problems that arise. Sign up for a free trial to get started if you don’t yet have a Datadog account. And if you’re already using Datadog, you can read our documentation for more information on how to monitor, debug, and optimize your Rails applications and their underlying infrastructure.