This article was originally published October 2015. This research was last updated June 2016.

With thousands of companies using Datadog to track their infrastructure, we can see software trends emerging in real time. Today we're excited to share what we can see about true Docker adoption—no hype, just the facts.

Docker is probably the most talked-about infrastructure technology of 2016. We started this project to investigate how much Docker is used in production, and how fast real adoption is growing. We found the answers to these questions—and more that we discovered along the way—to be fascinating.

The research that follows was based on a sample of 10,000 companies and tracks real usage, not just anecdotally reported usage. As far as we know, this is the largest and most accurate review of Docker adoption that has ever been published.

Throughout this article we refer to companies' adoption status: "adopted", "dabbling", or "abandoned". Our method for determining adoption status is described in the Methodology section below.

ONE

Real Docker Adoption
Is Up 30% in One Year

At the beginning of May 2015, 8.2 percent of Datadog's customers had adopted Docker. One year later that number has grown to 10.7 percent. That's almost 30 percent market-share growth in 12 months.

TWO

Docker Now Runs on 10%
of the Hosts We Monitor

This is an impressive fact: 18 months ago Docker had about 2 percent market share, and now it's running on 10 percent of the hosts we monitor.

However, the graph below illustrates two other interesting findings. First there was a distinct local maximum of percent hosts running Docker around August 2015. This could be caused by Datadog reaching new customers who did not use Docker yet, or by organizations cooling on Docker while the technology matured, or by both. Second, growth as a percentage of hosts seems to have transitioned from "extremely rapid growth" to "rapid growth" after that time.

THREE

Larger Companies
Are Leading Adoption

This one bucks the stereotype that larger companies are slower to move. The more hosts a company uses, the more likely it is to have tried Docker, and the more likely it is to have adopted Docker. This fact is particularly surprising since the more hosts a company uses, the higher the number of Docker containers the company must use to be considered an "adopter". Notably, in the eight months since this article was originally published, this finding has only strengthened.

Editorial conclusion: Docker is solving problems felt most acutely by companies with larger numbers of hosts.

FOUR

2/3 of Companies
That Try Docker Adopt It

Good news for Docker just keeps coming. We were surprised to find how many companies who try Docker actually adopt it, and fast. Most companies who will adopt have already done so within 30 days of initial production usage, and almost all the remaining adopters convert within 60 days.

FIVE

Adopters 5x Their Container
Count within 9 Months

Docker adopters approximately quintuple the average number of running containers they have in production between their first and tenth month of usage. This phenomenal internal-usage growth rate is quite linear, and shows no signs of tapering off after the tenth month.

SIX

The Most Widely Used Images Are
Still Registry, NGINX, and Redis

The most common technologies running in Docker are:

Registry: 21% of companies running Docker are using Registry. While Registry is still in the #1 position, its share has declined from 25% eight months ago.
NGINX: Docker is being used to contain a lot of HTTP servers, it seems. It is interesting that Apache (httpd) doesn't make the top 10.
Redis: This popular in-memory key/value data store is often used as an in-memory database, message queue, or cache.
Elasticsearch: Full-text search moved up from the #6 position in the last revision of this article.
MySQL: The most widely used open source database in the world moved up from its previous position at #9.
Logspout: For collecting logs from all containers on a host, and routing them to wherever they need to go.
Quay: With Registry losing a bit of ground, Quay charts on the leaderboard for the first time.
etcd: A new addition to the leaderboard, etcd is the only distributed configuration store to chart.
Postgres: The second-most widely used open source database in the world. Adding the MySQL and Postgres numbers, it appears that using Docker to run relational databases is surprisingly common.
CAdvisor: Used by Kubernetes to collect metrics from containers.

Knocked off the leaderboard:

Ubuntu: Ubuntu would have retained its ranking, but was removed because a variety of very different images were built on this default base image.
MongoDB: The widely used NoSQL datastore.

SEVEN

Docker Hosts Often Run
Five Containers at a Time

The median company that adopts Docker runs five containers simultaneously on each host, up from four containers eight months ago. This finding seems to indicate that Docker is in fact commonly used as a lightweight way to share compute resources; it is not solely valued for providing a knowable, versioned runtime environment. Bolstering this observation, 25% of companies run an average of 10+ containers simultaneously.

EIGHT

VMs Live 6x Longer
Than Containers

At companies that adopt Docker, containers have an average lifespan of 2.5 days, while across all companies, traditional and cloud-based VMs have an average lifespan of almost 15 days. These numbers have changed from 3 and 12, respectively, just 8 months ago. One possible explanation is that as more companies move dynamic workloads to Docker, the average container lifespan shrinks, leaving traditional hosts to run for longer periods of time.

As discussed in fact #7, it is common to run five containers per host simultaneously. So you might expect that the median VM would run 30 containers total in its lifetime (6 generations of 5 simultaneous containers). But due to uneven distributions, the median VM actually runs 14 containers in its life.

Containers' short lifetimes and increased density have significant implications for infrastructure monitoring. They represent an order-of-magnitude increase in the number of things that need to be individually monitored. Monitoring solutions that are host-centric, rather than role-centric, quickly become unusable. We thus expect Docker to continue to drive the sea change in monitoring practices that the cloud began several years ago.

METHODOLOGY

Population

As noted in the introduction, we compiled usage data from a sample of 10,000 companies, so this is almost certainly the largest and most accurate review of Docker adoption that has ever been published. However, while Datadog's customers span most industries, and run the gamut from startups to Fortune 100s, they do have some things in common. First, they take their software infrastructure seriously, and second they tend to be public and private cloud users. All the results in this article are biased by the fact that the data comes from our customers, a large but imperfect sample of the entire global market. Caveat lector.

Averages

When we talk about average numbers within our customer base (for example, the average container lifespan) we are not actually talking about the mean value within the population. Rather we compute the average for each customer individually, and then report the median customer’s number. We found that when we took a true average, results were unduly skewed by unusual Docker practices employed by relatively few companies. For example, using a container as a queueable unit of work could cause individual companies to use thousands of containers per hour.

Adoption Segments

This article categorizes companies as Docker "adopters", "dabblers", and "abandoners". Each company is recategorized at the end of each month, based on their Docker activity that month.

Adopter: the average number of containers running during the month was at least 50% the number of distinct hosts run, or there were at least as many distinct containers as distinct hosts run during the month.
Dabbler: used Docker during the month, but did not reach the "adopter" threshold.
Abandoner: a currently active company that used Docker in the past, but hasn't used it at all in the last month.

Note that the adoption-segmentation thresholds are not derived from a natural grouping within the data; the data covers a continuous spectrum of use. Rather we used numbers that we felt would be intuitively meaningful to our readers.

Interestingly, the findings in this article are surprisingly resilient to different adoption-segmentation thresholds. For example, regardless of whether the adopter threshold is lower (25% containers on average, or 0.75x distinct containers compared to hosts) or higher (75% containers on average, or 1.5x distinct containers compared to hosts), most findings are unaltered:

Fact #1: Real adoption is still up ~30% in one year; actually 33% for both of the alternate thresholds.
Fact #2: Adoption segmentation is not relevant to these findings.
Fact #3: Large companies are still 2.4–2.9x more likely than smaller companies to be Docker adopters or dabblers. In fact, the graphs change very little with different thresholds.
Fact #4: Findings are almost unaltered: adoption percentage changes only by +1.5% (lower threshold) and -3% (higher threshold).
Fact #5: Adopters still 5x their container use between month 1 and month 10.
Fact #6: Adoption segmentation is not relevant to these findings.
Facts #7, 8: Results are not altered.

Counting

Containers running only the Datadog Agent were excluded from this investigation, so Docker hosts that were only running the Agent were also excluded.

Fact #1

We thought maybe we were seeing such a large increase in Docker use on Datadog precisely because Datadog is good at monitoring Docker. Maybe new Docker growth was fueled by new customers who needed Docker monitoring and came to Datadog specifically for that purpose. However, when we created cohorts of longtime Datadog customers, we saw almost identical adoption percentages.

Fact #2

For each technology we monitor, we exclude from this calculation the organizations that are in the top 1% of its users. In other words, if a small number of companies use a particular technology in an unusual way, and use it quite heavily, they are excluded from the calculation.

Note, too, that the same basic shape of the "Percent hosts running Docker" graph persists even if we limit our population to Docker-using companies, or if we exclude the top 5%, 10%, or 25% of the Docker-using companies. There is a distinct local maximum around August 2015 followed by nearly linear growth to a new high.

Fact #3

This finding is resilient to different infrastructure-size cut-points. Whether the cut-points are halved or doubled, the relative shape of the graph hardly changes, and the conclusion is exactly the same.