Platform teams aren’t like other teams
Hi, so in my role as an engineering manager or manager of engineers for years, I’ve dealt with variety of different teams.
There have been different technical problems that my teams have had to solve.
I’ve had teams that are non-engineering teams entirely.
I worked with turning some of the troublesome teams around, as well as working with various software teams.
But I think that the most interesting and challenging part of my work has been working with platform teams, and specifically growing platform teams.
Platforms, and we’ll talk about platforms in a little bit, aren’t like other teams.
Platform and products teams differ in many, many aspects.
Your customers are very different, the way that you treat your backlog is very different, and the composition of people and problems on the team ends up being very, very different.
Your day-to-day operational stuff and concerns is very different.
It’s too early to call a success on our platform effort at Flatiron, we started about a year-and-a-half ago, but I think that we have a lot of very interesting learnings to take out of it, and that’s what I want to talk to you about today.
Just as a side note, you’ll see a lot of Legos in this presentation.
When I first started at Flatiron, our company was located in the Flatiron district of Manhattan right above the Lego store,
and hence the collection of Legos in my house has about tripled, and I figured this is the best way for me to visualize what I think about platform teams.
What platform means at Flatiron
So you likely have already heard a few of the platform definitions today.
If you look at Merriam-Webster Dictionary there is about 10 different definitions that are ranging from the declaration of principles by which a group of persons adheres to, like a political platform, all the way down to like a thick sole of a shoe.
All the way on the bottom there is a very way technical definition that talks about that being sort of like an operating system or a computer architecture.
And I think every company probably defines platforms very differently, so I figured I’d throw this definition up here of how we think about platforms at Flatiron.
So we think about it as a way to describe a technology and infrastructure as a product, as well as a set of software capabilities that are enabling us to efficiently test ideas, scale our delivery, or provide new capabilities, internally and externally to our physicians and researchers.
So why am I talking about physicians and researchers?
In order for me to tell you the platform story and kind of put it into context for you, let me tell you a little bit about Flatiron and what we do.
Flatiron mission
So our mission is to improve lives by learning from the experience of every cancer patient.
But what does that actually mean?
It means that we’re using oncology data and technology to accelerate research, which includes data discovery, time of oncology treatments to market, understanding of various treatments, in how are they doing in the real world. As well as to empower physicians at the point of care.
That includes providing them with effective and smart electronic health records system.
Includes enabling a patient in any small community oncology practices to have same access to trials as they would at a larger institution.
As well as, to help the practice to operate more efficiently, thus cutting the costs of care for the actual patients.
As you can see, the variety of the products that we have is very large, and they’re all powered by the data generated by our electronic health records system, as well as by the data that we’re ingesting from the outside.
Platform Team responsibilities
And our data platform teams inside Flatiron provide the means by which this data rests and moves all around the Flatiron.
We’re responsible for data hosting, structuring, discoverability, transport, and quality.
The infrastructure tools and frameworks that actually support this data is also part of our data platform.
But we started off, as you can kind of see on this picture, with a series of legacy pipelines that are developed across many different product teams.
Some are potentially reusing some of the same building blocks, or Legos, or libraries as some might call them.
But most of them are kind of running in silo, delivering data to very specific teams, for very specific purposes.
So what our platform team has then set out to do is to take ownership of this infrastructure, combine all these different disparate data sets and data processing tools and create an analytical data warehouse, support a set of distributed data processing tools, as well as standardize and enable the data movement through standardization.
Govern how data is packaged and received, as well as own some of the interfaces around handoffs of the data.
So really, our product can be viewed as sort of this massive amount of organizational tech-debt that we’ve taken on.
Platform beginnings
Before I jump into the meat of the stock, on the details of building out a platform team, let’s talk for a couple of seconds on how platform teams are born.
So timing is very different for different companies.
You don’t start building a small dataset for customers by investing into an analytical data warehouse immediately.
And if we would have done that, we probably wouldn’t have ended up with the diversity of the offerings that we have today.
So you start an experimentation, then you move into sort of spending the time of building quickly a wide variety of different products.
Your small engineering teams can be pretty much experts at everything at this point, and you can’t really afford to develop further than a step away from your customer.
Once your engineering team has grown to a certain size, they’re all working in parallel on large number of products, you may start thinking about what your technical needs are all across the company.
It really will depend on the size of your teams, the maturity of your products, and the types of platforms that are necessary.
By this time you’re very good at working cross-functionally, at having amazing ideas, translating those ideas into proof of concepts and products, and delivering at your current scale and complexity.
But, your customers have now established, and you need to actually really rapidly scale your effort.
So, in order for you to scale, you can’t hire 20 times the engineers.
So in order for you to deliver the 20 times with two times the effort, this is the time that you start building out the actual platform
that would not only allow you to scale rapidly, but would also ensure continuous stability of your vast portfolio of products.
So we talked about kind of when to develop it.
About six months before I came on board into Flatiron, my CTO Gil came out in front of the whole company and declared sort of our mission of developing an interoperable technology platform.
Thus, securing both engineering and executive buy-in before we actually even started doing this.
This was a very first, important step, kind of fanfare before the product actually exists.
And you’ll see later in kind of what I’m describing that timely and appropriate communication is actually very important in every step of developing a product.
Other considerations
So what are some of the other important things to consider when we’re talking about platforms?
Things that we’ll talk about is, past that initial fanfare, what are the first sort of underwater stones of taking on platform ownership.
We’ll talk through some of the techniques around getting out from underneath the large support burden.
We’ll cover sort of turning your organization’s technical landfill into an actual viable product.
Cover securing buy-in from your stakeholders, from engineering teams, from executives,
and finally, kind of look a little bit ahead of how to actually grow the platform teams themselves.
Taking ownership
So, we’ll start with taking on ownership.
Platforms very frequently are built not kind of in an empty space from the scratch, in the beginning they’re built on legacy code, many components which are developed and maintained frequently in silos.
These legacy systems most of the time lack owners.
We frequently no longer have anyone in the company who actually understands these systems.
There’s a general fatigue and disengagement that you will feel, as well as development teams are distrustful and dissatisfied with a lot of these common components.
And they will continue building new products replacing existing ones and silos.
Frequently the systems, for example, for us, for Flatiron, these systems have been maintained during our quarterly two-day hackathons.
I’m actually missing the hackathon today, in which hopefully we’re no longer hacking on things that should be platforms and are actually hacking on fun product ideas.
So what happens upon creation of the platform is you get the dump all of these different legacy components that you now all of a sudden take ownership of.
And then the first thing that you should understand is that the platform team is not a legacy dump.
When you find yourself in this situation you slowly start plowing through it.
Do we own all of the legacy frameworks?
Do we own all of the tech debt at Flatiron?
Should we focus on a specific area?
You need to define the mission and vision for you teams that allow you to figure out how to set these appropriate boundaries and how to scope what you’re doing.
We decided that it’s better for us that this vision is not purely technical.
For that, we needed to identify, what do we want to unlock for the rest of the company by creating these platforms?
So, we defined our data platform as a learning healthcare platform that actually allows us to use our ever-increasing dataset that continuous being enriched by various products and unlock the ability for physicians and patients to benefit immediately and directly from all the data processing enhancements, analytics, that happen across our product set.
This vision has allowed us to create objectives that very easily translate from overall company goals down into our platform goals.
It allowed us to prioritize and select the infrastructure that we want to continue investing into, as well as to limit the scope of ownership.
Once you defined what’s in scope, you also have to define what’s not in scope, and think about why it should and should not be.
You need to determine whether it’s time to stop supporting some of the features entirely, or potentially hand them off to other teams.
For example, at Flatiron we have an ETL data processing framework.
It’s developed over many, many years, it continued growing various features,
it actually has multiple versions of it living simultaneously on various downstream teams.
The future development in this framework for us has been blocked by existence of all these different older versions.
So we had to actually set the time, develop a plan, and retire some of the versions that are continuously being used by other teams.
So stopping to support the specific functionality has allowed us to unblock future development on the system.
Lessons learned from taking ownership
Some of the things that we learned in the process of handoffs and retirements is that things that we need to do is to identify the dependencies.
The dependencies aren’t always obvious.
Things that you find out along the way of, “Oh, somebody has been actually using this,” happens very frequently, retrospectively.
Identify the stakeholders and establish a very effective sign-off process.
You can’t just find one person on the downstream team and say like, “Oh, can you sign off on this plan for me?”
You actually need to ensure that this person is communicating to their team in entirety, that everyone’s aware.
You need to communicate early and seek feedback frequently.
So it’s not just sort of outward communication, you need to kind of ping somebody say like, “Hey, have you actually heard about this?
Are you going to get back to me at a specific time?"
And then have clearly defined timelines and milestones that are continuously communicated on and dependent of progress.
Just because things are going as planned on doesn’t mean you shouldn’t actually periodically check-in and remind people that things are going as planned.
If things aren’t going as planned, if something actually came up along the way, make sure you notify your stakeholders.
Retiring and relinquishing ownership is not very much unlike rebuilding the infrastructure, in terms of how it should be handled.
And rebuilding the infrastructure itself very frequently feels like replacing the foundation while the house is still standing.
In our case our customers, the downstream teams, are providing a software to the physicians that are treating patients on daily basis.
And so, our inability to continue providing them with daily data has direct impact on ability of this physician that we as a platform team may not actually understand.
So establishing not just effective rollout plans, but also taking into account scheduling of customer-facing teams, taking into account their needs specifically when we’re actually creating our plan is very, very important.
What to communicate
Establishing continuous and stable communications, cadence of the communication is very important.
And then creating a set of tools that allows your downstream consumers to continuously monitor the state of your systems.
So it’s not that you want this monitoring the state of your systems, your consumers actually have to have an understanding of how you’re doing.
During this period, we’ve learned that we need to communicate clearly and frequently, and that continuous communication sometimes is more important than flawless execution.
Things that we’ve communicated are: ownership boundaries, retirement plans, rebuilding plans.
We’ve gone on road shows for requirement gathering, we’ve done early feedback on design and planning, not just on the requirements, as well, as handoff communication is something that’s been very important.
Support burden
So we got rid of half of the stuff that we initially owned, but we continued to be unable to get out from underneath a heavy support burden.
When you first take over the support, it’s majority of your team’s time.
We continue spending much of our time putting out fires, instead of steadily working on forward-looking progress.
There’s a lot of operational tasks that can easily eat up a full team’s time:
ensuring uptime, answering downstream teams’ questions, responding to feature requests from downstream teams, and deploying code.
Establishing effective on-call process will solve some of those burdens.
And you, I think are hearing like a talk on on-call process, either earlier today or later today and there’s books that have been written on this.
So really, what I just wanted to point out is kind of the differences between what does the on-call process look like for platform teams versus downstream customer-facing teams.
So what’s different for the platform teams?
On-call process for platform teams
Your immediate customers are internal.
You’re playing a support role that is likely handled by multiple teams for the customer-facing teams.
The downstream teams come to you with questions, with feature requests, and with concerns.
So, you’re kind of like an account manager, a product manager, and a customer support team all at once, while you’re still trying to continue to do your engineering work.
You are removed from the end consumer of your company’s product, so you don’t always kind of, as we talked about, understand the impact that the changes that you’re making on a platform have all the way down on the physicians or the patient.
And as an example, sort of like when you’re delivering data on nightly basis, you don’t understand that even if, you know, something failed overnight you redeliver in the middle of the day, without knowing exactly what happens at the clinic and the fact that in the morning the roster of all the patients that are eligible for trials is getting printed out and is being used throughout the rest of the day; you don’t understand that redelivering at 10 A.M. versus 8 A.M. is going to make a huge difference in the way that the clinic is operating.
And then you also need to be aware of all the different dependencies, incase of incident management.
So when something happened, you are not letting just your consuming teams know, you potentially are letting downstream account management,
you need to set up with all these various trees of what all dependencies are and who needs to be notified of what.
Supporting legacy code and SLAs
Establishing effective on-call process won’t necessarily solve all of your problems immediately.
For platform teams that are dealing with legacy code, there’s a continuous vicious cycle.
All of our SLAs are continuing to break daily, many customers are unhappy and miserable, like when you’ve first taken over all these legacy platforms.
Likely your team already has some preexisting SLAs that they are very unhappy with because they just can’t fit within them.
We frequently go in with an assumption that we can’t negotiate on these SLAs.
However, what we can do is find out why and how they’ve been put in place.
So as an example, we have a 4:00 a.m. SLA on daily data delivery to our downstream teams.
The reason that 4:00 a.m. SLA has been put into place is the fact that they need to deliver their data to customers by 8:00 in the morning.
Some of their pipelines can run for as much as four hours, so that sort of created the 4:00 a.m. and the 8:00 a.m. deliverable SLAs.
We then could step back and look at the whole process holistically and then reevaluate what needs to happen when
and maybe some things need to be tweaked downstream and not necessarily within the platform, to be able to fit within the 8:00 a.m. data delivery.
Additionally, in that process we found out that some of the SLAs weren’t actually motivated by anything, they’ve just been established once upon a time, and are no longer actually necessary and you don’t have to wake up in the middle of the night to fix it.
What to communicate
Once you’ve actually negotiated some of these SLAs, once you reestablish them, communicating these renegotiated commitments is very important, because people may still have certain expectations that have been built in over time.
Communicating what your service level objectives are against those SLAs is also very important because people are going to look at the trends of how much are you improving within that commitment.
Setting the goals around, and monitoring, your service level as it continues to go up.
Here within establishing of this communication, one of the things that I wanted to mention is you want to establish as a platform team, your team communication API.
So what are the different means of other teams contacting you and connecting with you?
What are the things that they need to know?
Like, do you have a specific Slack channel that they’re pinging you on?
Is there a single Jira board that they are actually adding requests on?
All of these things really will become your sort of interface for the rest of the world and will become your API.
Platform as a product
So, you’ve gotten out from underneath the support burden.
Our daily processes now move from like, kind of Brownian motion of particles, to more of a resemblance of a conveyor belt.
This is the time that we can actually start thinking about converting our massive tech debt into an actual product.
And in order for us to make it a product, we need to start thinking about building technical roadmaps.
Chances are, you don’t have a technical product manager, who comes in with sort of like two suitcases of tips and tricks on how to build a successful roadmap and do all of these things.
So you need to define who your product owners are.
In the short term, most likely your tech lead or mature tech lead can hack that.
But eventually, you may need to carve specific roles to avoid burnout, because very frequently pushing that responsibility on a tech lead along with minding the team and doing all these other things, becomes very, very burdening on them.
We found that introducing a program or a delivery manager may help to, at the very least initially, reduce that burden.
They can help you establish variety of different processes lowering the operational burden on your team.
There’s probably a separate conference that covers product management, so I’m not going go into things like strategic planning and roadmaps and OKRs, and all those other things.
But some of the things that are worth mentioning that are specific to platform teams, is the fact that you want to establish effective feedback protocols with your customers.
Your customers aren’t miles away, kind of the way that they would be for downstream teams.
So you don’t just talk to them about requirements, you can actually check in with them with regard to your design plans.
They can weigh in on your POCs and they can actually test things out early.
Something you don’t necessarily think about when you’re dealing with sort of the customers that are much further removed from you.
Another important aspect of product management is ability to actually measure how successful you are at delivering that product.
Are you building the right thing?
Are you delivering value?
Are you prioritizing the right things?
Are you solving the most important problems?
And this is where it becomes very difficult when you’re working on a platform team, because there aren’t immediately dollars or revenue associated with it, there aren’t clicks and things that allow you to measure things easily.
What to communicate
So how do you actually measure the increased throughput on downstream teams as a result of you delivering the product to them?
What’s an equivalent of an Mbps with the internal teams?
The key learning within this area for us is that you won’t collect metrics unless you need to communicate them, so unless you’ve committed to specific things that you are continuously giving to your consumers.
So, communicate your commitments early, tell them like, what are you going to report on,
and then communicate them all the time, even if you haven’t actually had a chance to measure them.
This sort of holds you more accountable towards it.
It allows you to figure out whether or not certain things are important, if people are actually going to start asking for that metric that you’ve been communicating as an N/A for the last month.
One thing that’s important within this communication to note, is that since you have direct communication with your consumers, you can communicate changes to your priorities relatively early to them.
You can also set up feedback loops that allow you to … for them to actively feed into your priorities.
Another thing is as we’re communicating metrics, metrics are great but they only tell story in a slice of time.
Communicating trends within this, and not just specific metrics like, “Oh, we have 25 backlog Jira tickets this month,” doesn’t tell anything to your downstream consumer.
Telling them that you’ve gone from 50 to 25 in the last month, telling them that the downstream teams have spent, instead of 700 hours, 25 hours delivering monthly deliverable, thanks to some of the things that you’ve put in place, actually makes a huge difference.
If you just communicate 25 hours or 700 hours in a document, doesn’t make a difference.
Getting buy-in
Remember, I said in the beginning that my CTO had announced the existence of platforms very early, six months before we actually came in.
Well since then our team has actually spent numerous hours on stage, talking in conference streams, talking at theater stages.
Here in this picture you’ll actually see me and Rohit, who’s currently playing my technical product manager role, talking to 500 people in the audience, to our whole company, trying to explain to them what the platform actually means to them.
So this is our finance department and accounting department trying to understand what our data platform actually does.
Our CEO has been reciting our data warehouse wins in our company meeting.
Our on-call engineer sends out weekly summary notes.
We have spent hours on Wiki pages, newsletters, release notes, and progress updates.
Why are we doing all this?
Why are we not, you know, spending this time actually working actively in development and continuously writing all these different things or talking to people?
We found that we need to be known by engineers, engineers need to know what is it that we’re doing, how we’re building it, why are we building it?
Engineers very frequently have a very narrow outlook, they’re so focused on the problems that they’re dealing with at hand right now, so finding every opportunity where somebody has picked their head up above the water to actually communicate to them has been very important.
Evangelism though will take you only so far.
There are two additional concepts that I wanted to cover as part of the buy-in.
One of them has been governance and the other is trust.
And when you’re thinking about governance and trust sometimes people say, “Oh you only need one,” I feel like you need both.
Governance
Governance is necessary when you’re creating tools that are used by everyone,
it’s very difficult to do in a non-homogeneous environment, so you need to at least set some sort of standards that other people will be adhering to.
Saying like, “Hey, if we’re creating services we’re going to be using RESTful APIs, or if we’re working within certain relational databases, our supported choice is RDS.”
You’re not necessarily dictating to teams what they should be using, but you can tell them “Hey, we will only support you in a situation when you’re using these set of support tools.”
There are going to be situations where you might need to kind of have a stricter form of governance and you actually will need to talk about very specific things.
Like for example, we deal with personally identifiable information, so there’s going to be things that we will have to set clear rules about that can’t be broken,
but for the rest of the things your standardization really talks about, “Here’s a set of supported technologies that we use.”
Trust
Okay, I’ve set up all these rules, that should get me through. But as I said, not quite.
Remember we talked in the beginning that there is all these different orphaned frameworks and tools that we’ve been handed.
Everyone is using them, no one is supporting them, and then you come in and make a statement like, “Yes, from now on we own them, we support them, this is ours.”
Well, to your downstream teams, in the very beginning, this is going to feel like somebody came in and took their toys away.
This actually happens in my house daily.
Your consumer teams don’t trust that you know what they need; your consumer teams actually don’t trust that you know what you’re doing.
My team when I first came into Flatiron was pretty much all brand new engineers at Flatiron, who were just still learning the things that they’re doing kind of on the job.
And so, we didn’t necessarily have that trust developed with the rest of the engineering team.
And then they also don’t think that you have the capacity or motivation to prioritize their problems.
So what are the steps that you might take to win their trust back?
One of the things is, you need to continuously provide them a clear view into your roadmap.
So, once again, all these things that we don’t necessarily take the time to spend on updating and creating Gantt charts and all those other things, actually are important for them to know exactly what you’re working on.
And then they need to know what are you working on today, why aren’t they your number one priority?
So you need to actually moderate what your number one priority is and why.
And then, simple things like access to your trends and historical dashboards, not just kind of occasionally sending it to them, but them being able to at any point take a peek at what you’re doing.
And then establishing a documented feedback loop, how do you tell me what you need?
Growth
Finally, to scale all of Flatiron, we must be able to scale as a team ourselves.
By the time your stakeholders love you, your engineers love you, you have more work than you can imagine, you need to scale your team, you need to grow it.
As a platform team, your best hires are actually poached internally.
As I said just a second ago, having a brand new team working on a set of platform products is very difficult, you actually haven’t felt the pains that your downstream engineering teams have felt.
So ideally, what you want to do is you want to hire engineers from within, who already know some of the products, who don’t have to actually learn the things from the beginning, because as a platform team you need to know a little bit more than each one of the specific product teams does.
So in order for you to recruit internally, your image should be that of superheroes, and not sweepers or plumbers who are just kind of sweeping tagdat under the rug.
And in order for you to create that image first, you have to establish that image within the team.
In a cycle of continuous improvement, and it’s very difficult for us to remember and celebrate small wins.
We’re continuously, you know, we’re sending out retros and postmortems and all those little things of how we’re doing, you know, this and then that and so on and so forth.
We kind of forget to stop and think about like, oh you know, this is great, we’re no longer actually spending 20 hours a day on-call, or we’re not spending all this time doing support tickets and so on and so forth.
Creating an internal mantra and ensuring that this communication is happening internally, as well as externally, will set you up on that path to success.
Team communication API
Once again, as your team grows, that communication API that you initially have set becomes very, very important.
So within our teams we recently have gone through a transition between going from two teams that compose our data platforms into three different teams that compose our data platforms.
What happens in this situation very frequently is that you haven’t actually created a proper API, what you’ve created is exposure into your internal functionality to other teams.
So what we’ve done that has been wrong is that we had a team that was responsible for data requests, and we had a team that was responsible for infrastructure.
And so we’ve had people directly pinging those teams for very specific needs, and we’ve had sort of all of our API set up in a way that we expect two different data teams.
Now that we’re three different data teams, and a lot of our internal things have been shuffled, it’s impossible for our downstream consumers, again to find, “Who do I tell what?”
So in retrospect, what we should have done is sort of created a single API that we don’t have to version as we upgrade into different number of teams, or reconfigure internally,
where for this specific need, you go to a Slack channel that is actually focused on just this,
and what our team is servicing that Slack channel is really not important.
What to communicate
So just to kind off close of this area, the communications here that are important for growth are: wins, and not just troubles and priorities,
communicating internally as well as externally is very important, as well as kind of establishing and keeping stable that team communication API.
As you noticed, a lot of it has been around communication, and not just execution and delivery, and this is probably one of our biggest learnings throughout of actual like the platform team growth.
That’s all that I have, so if anybody has any questions, now is a good time or you can find me in the hallway.
Questions
Audience: Hi.
What is the size of a platform team where you woul start to actually hire for a TPM or a PM?
Stacy: What’s the size of the platform team where you actually start hiring a TPM or a PM?
Audience: Yeah, we are a team of three engineers so we’re pretty small for a PM, you know.
Stacy: I think when it became relatively difficult for us to handle things internally is when we did that initial split from one team into multiple teams, when there is actually no longer things that every single person within the team sort of has a handle on.
So I will say for us has been like probably around eight or nine engineers was a time when it became really apparent that we needed to do it this.
And just to be clear, we didn’t hire an external TPM, we’ve actually taken one of our data insights engineers and have converted them into that role, where it became their full-time role.
There is a question over there.
Audience: Hi.
My question is, like you mentioned quality several times in your whole slides.
So is your team also made up of like some QA or like automation engineers or for whatever kind of test frameworks or support tools as well?
Stacy: So we have been focusing on quality quite a bit, but it’s been sort of an engineering problem for us so far.
It hasn’t been as much, we don’t have dedicated quality engineers.
We thought about an engineer in test role for a bit, but for now really what we’re focusing on is building out these larger frameworks that are allowing us to measure the quality of our data more easily.
And once we’ve actually built out the frameworks then we can focus on whether or not we need a person who maintains and thinks about these things changing over time.
Audience: Hi, thank you.
Sorry, I missed the beginning, we were trying to get in.
So if you already covered this, but I’m curious what are some of your background experience or resources that led you to be able to be so effective in your current role?
Stacy: That’s a good question.
I think just having experience working on products teams definitely helps a lot because I’ve been sort of on the receiving end quite a bit.
I’ve been both working on and leading teams that are focused on customer-facing products, and teams that are actually using some of the common platform components from other teams.
I’ve sort of experienced the pain of being a consumer of products that a platform team that’s working in a silo that’s not communicating to the downstream engineering teams is producing.
And I kind of understand, sort of from a customer perspective, what have been some of the downsides of that.
So I think that that could have been one of the helpful things for me.
Audience: You already mentioned it but maybe you can deepen details. It’s very thin border between your customer and between you.
And when something going wrong, its very hard to figure out, is these guys using it wrong way, or you’re really doing something wrong, and how you handle these situations?
Stacy: Absolutely.
It’s actually quite interesting,
part of it is also perceptions that kind of build around it.
So, very frequently our customer success teams and our account management teams, they’re continuously saying like, “Hey, our customers are getting data wrong, there’s something wrong with our data platforms because continuously our customers aren’t getting their data on time, etc., etc.”
And so what’s been important for us is to kind of try to figure out how we can provide more measurements at each step.
So one of the things that we just talked a second ago about data quality and data quality measurement platforms, we’re trying to figure out is there a framework that we can apply at every step of data processing, every time sort of the data makes the leap to the next team or to the next consumer and so on and so forth.
And we can apply the same set of tests and the same set of metrics for this data, which makes it easier for us to kind of understand where something is no longer fresh or has transformed in a way that we don’t expect it to.
Outside of like having some of these pre-established tools, a lot of it has been somewhat manual where we actually would manually run a variety of reports and continuously compare the reports, like this our downstream report, this is our report, do these two things match up?
And then we find out where something could have gotten lost.
A lot of the stuff has gone from very manual, like let’s figure out manually what we need to do, to now and we actually have time to sit down and think about building a platform to do this, without our or direct manual involvement.
I think that’s the last question.
Audience: Where did you get the Lego pictures?
Stacy: So some of the Lego pictures are stock images, but like a bunch of them I actually, my son just got an animation studio for his birthday, and I’ve used that to shoot the Lego images.
That was fun evening.
Thank you.