Real Docker Adoption
Is Up 5x in One Year

At the beginning of September 2014, 1.8 percent of Datadog's customers had adopted Docker. One year later that number has grown to 8.3 percent. That's almost 5x growth in 12 months.

Docker Now Runs on 6%
of the Hosts We Monitor

This is an amazing fact: one year ago Docker had almost no market share, and now it's running on six percent of the hosts we monitor. Six percent may not sound high as an absolute number, but since we monitor 120+ different technologies, it represents a meaningful part of our users' stacks.

However, as seen below, growth as a percentage of hosts seems to have stalled last quarter. This could be seasonal; major infrastructure initiatives are often paused until after summer vacations. We will be interested to see what happens this quarter.

Larger Companies
Are the Early Adopters

This one bucks the stereotype that larger companies are slower to move. The more hosts a company uses, the more likely it is to have tried Docker, and the more likely it is to be have adopted Docker. This fact is particularly surprising since the more hosts a company uses, the higher the number of Docker containers the company must use to be considered an “adopter”. As discussed in Methodology below, this finding is resilient to different infrastructure-size segmentation thresholds.

2/3 of Companies
That Try Docker Adopt It

Good news for Docker just keeps coming. We were surprised to find how many companies who try Docker actually adopt it, and fast. Most companies who will adopt have already done so within 30 days of initial production usage, and almost all the remaining adopters convert within 60 days.

Users Triple the Number
of Containers They Use
within 5 Months

Both adopters and dabblers approximately triple the average number of running containers they have in production between their first and sixth month of use. This phenomenal increase in usage—even amongst dabblers—is great news for Docker.

The Most Widely Used Images Are
Registry, NGINX, and Redis

The most common technologies running in Docker are:

• Registry: Fully 25% of companies running Docker are using Registry, presumably instead of Docker Hub.
• NGINX: Docker is being used to contain a lot of HTTP servers, it seems. It is interesting that Apache (httpd) didn’t make the top 10.
• Redis: This popular in-memory key/value data store is often used as an in-memory database, message queue, or cache.
• Ubuntu: Still the default to build images.
• Logspout: For collecting logs from all containers on a host, and routing them to wherever they need to go.
• MongoDB: The widely-used NoSQL datastore.
• Elasticsearch: Full text search.
• CAdvisor: Used by Kubernetes to collect metrics from containers.
• MySQL: The most widely used open source database in the world.
• Postgres: The second-most widely used open source database in the world. Adding the Postgres and MySQL numbers, it appears that using Docker to run relational databases is surprisingly common.

Docker Hosts Often Run
Four Containers at a Time

The median company that adopts Docker runs four containers simultaneously on each host. This finding seems to indicate that Docker is in fact commonly used as a lightweight way to share compute resources; it is not solely valued for providing a knowable, versioned runtime environment.

VMs Live 4x Longer
Than Containers

At companies that adopt Docker, containers have an average lifespan of 3 days, while across all companies, traditional and cloud-based VMs have an average lifespan of 12 days.

As discussed in fact #7, it is common to run 4 containers per host simultaneously. So you might expect that the median VM would run 16 containers total in its lifetime (4 generations of 4 simultaneous containers). But due to uneven distributions, the median VM actually runs 9 containers in its life.

Containers' short lifetimes and increased density have significant implications for infrastructure monitoring. They represent an order-of-magnitude increase in the number of things that need to be individually monitored. Monitoring solutions that are host-centric, rather than role-centric, quickly become unusable. We thus expect Docker to continue to drive the sea change in monitoring practices that the cloud began several years ago.

METHODOLOGY

Population

As noted in the introduction, we compiled usage data from a sample of 7,000 companies, so this is probably the largest and most accurate review of Docker adoption that has been published to date. However, Datadog's customers skew toward "early adopters" and toward companies that take their software infrastructure seriously. All the results in this article are biased by the fact that the data comes from our customers, an imperfect sample of the entire global market. Caveat lector.

Averages

When we talk about average numbers within our customer base (for example, the average container lifespan) we are not actually talking about the mean value within the population. Rather we compute the average for each customer individually, and then report the median customer’s number. We found that when we took a true average, results were unduly skewed by unusual Docker practices employed by relatively few companies. For example, using a container as a queueable unit of work could cause individual companies to use thousands of containers per hour.

Adoption Segments

This article categorizes companies as Docker "adopters", "dabblers", and "abandoners". Each company is recategorized at the end of each month, based on their Docker activity that month.

• Adopter: the average number of containers running during the month was at least 50% the number of distinct hosts run, or there were at least as many distinct containers as distinct hosts run during the month.
• Dabbler: used Docker during the month, but did not reach the “adopter” threshold.
• Abandoner: a currently active company that used Docker in the past, but hasn't used it at all in the last month.

Note that the adoption-segmentation thresholds are not derived from a natural grouping within the data; the data covers a continuous spectrum of use. Rather we used numbers that we felt would be intuitively meaningful to our readers.

Interestingly, the findings in this article are surprisingly resilient to different adoption-segmentation thresholds. For example, regardless of whether the adopter threshold is lower (25% containers on average, or 0.75x distinct containers compared to hosts) or higher (75% containers on average, or 1.5x distinct containers compared to hosts), most findings are unaltered:

• Fact #1: Real adoption is still up 5x in one year.
• Fact #2: Adoption segmentation is not relevant to these findings.
• Fact #3: Large companies are still 2–4x more likely than smaller companies to be Docker adopters or dabblers. Relative to one another, the graphs are almost unchanged.
• Fact #4: Findings are almost unaltered: adoption percentage changes only by ±3%.
• Fact #5: Adopters still triple their container use between month 1 and month 6.
• Fact #6: Adoption segmentation is not relevant to these findings.
• Facts #7, 8: Results are not altered.

We also ran this analysis with absolute-usage thresholds, using segmentation rules such as “adopters had at least 20 containers running on average during the month”. While most of our findings were surprisingly similar (almost exactly the same, in fact), this strategy made it too likely that we would consider small companies to be dabblers when in fact a large percentage of their infrastructure was running Docker.

Counting

Containers running only the Datadog Agent were excluded from this investigation, so Docker hosts that were only running the Agent were also excluded.

Fact #1

We thought maybe we were seeing such a large increase in Docker use on Datadog precisely because Datadog is good at monitoring Docker. Maybe new Docker growth was fueled by new customers who needed Docker monitoring and came to Datadog specifically for that purpose. However, when we created cohorts of longtime Datadog customers, we saw almost identical adoption percentages.

Fact #2

For each technology we monitor, we exclude from this calculation the organizations that are in the top 1% of its users. In other words, if a small number of companies use a particular technology in an unusual way, and use it quite heavily, they are excluded from the calculation.

Note, too, that the same basic shape of the “Percent hosts running Docker” graph persists even if we limit our population to Docker-using companies, or if we exclude the top 5%, 10%, or 25% of the Docker-using companies. There is a distinct flattening of the percent of hosts running Docker during the last quarter.

Fact #3

This finding is resilient to different infrastructure-size cut-points. Several different cut-points are below. The middle set of cut-points were used for this article.

SIGN UP

Get Started with Datadog

To monitor your Docker environment, your own custom metrics, or any of Datadog's other 120+ pre-built integrations, sign up for a free 14-day trial account below.