• about

• process

• case studies

• contact

Problem

The goal of Model Ops is to enable data science work to be delivered from a data science team to data science stakeholders. As we attempted to mature our offering, we were encountering some issues that pointed to us needing a more robust vision for the future of the product.

Issues:

• Duplicate work: There were 5 existing methods for hosting and delivering models. Each had been implemented independently. This meant that every new feature (like autoscaling or performance monitoring) had to be implemented 5 times.
• IA complexity: The 5 methods each had their own page. We had some methods that were not shown at all. And we had new methods we wanted to add. The nav could definitely not continue to scale in this way.
• User confusion: In research calls, users seemed confused about the differences between the various hosting methods.

Two hosting methods

"Model APIs" and "Apps" are two ways of hosting and delivering models. The main difference is that Apps can be viewed in a web browser. However, in our product, the UI and config were completely different.

I wanted to take a step back, understand user goals and workflows more deeply, and create a vision that would allow us to scale our product to new technologies, usecases, and personas while increasing the value we could provide to our customers and decreasing their time to value.

Process

Research

I started by interviewing data scientists to learn about how they approached delivering their work. I wanted to better understand their goals and workflows and, most importantly, how they interacted with other stakeholders inside and outside of their team to deliver their work to others in their organizations.

Clearer problem statement

In terms of customer limitations, I identified two major problems with our existing approach:

1. Inadequate representation of user deliverables
2. Broken continuity across user workflow

Inadequate representation of user deliverables

For a data science team to provide value to the rest of their organization, they must deliver predictive models to other stakeholders in a way those stakeholders could consume. Our product was oriented around our implementation of a delivery method rather than on who needed to consume a model and why.

This was causing a number of issues:

• Incomplete representation of delivery: Only some delivery methods were shown in the product. Others required non-obvious workarounds that involved manually exporting the model. From there, a user had no visibility into what was deployed, to whom, or how.
• Unnecessary complexity: Each delivery method had its own separate UI with its own name. Users struggled to understand what the differences were, and no screen in the product could show a holistic view of what was being delivered. Further, these UIs were built and maintained separately. Every new delivery method we implemented required a new UI.
• Lack of continuity for model-consumers: Model consumers needed to consume a model at a consistent, durable location, regardless of version. If the model delivery URL changed, they had to make code or process changes to reflect this. However, the hosting method was built to create a new URL every time a change was made. Because the durable delivery point was not a first class citizen, there was nowhere to go to see what had been delivered over time or its current status.

Broken continuity across user workflow

The connection from doing data science work to delivering it was implicit. This was causing a number of problems:

• Difficult iteration: Data science work is highly iterative. Often a model in production needs to be tweaked and redeployed. We had no way of representing what version of a model was already deployed. This made it very difficult for a user to take a deployed model, make specific changes, and redeploy. Where a single version was deployed in multiple ways, it was impossible to see that these deployments were using the same version, or for example, that a single version had progressed from a staging environment to production.
• Race conditions: Because model versions were implicit, a data scientist would not see the version they were deploying until after it was deployed. This meant that if someone else on their team committed new work during the deploy flow, they might accidentally deploy the new commit without meaning to or even realizing it had happened.
• Unusable work: Our product had flows that would implicitly move a data scientist's work from their repo's main branch to a topic branch. The publishing flow only published from the main branch, and it did so without saying what exactly was being published. So work that had been moved to a topic branch would silently not deploy. To get this work to deploy, a user needed to know about this limitation, leave our product to manually merge the branch, and then deploy.

Synthesis

As I thought through this, I saw that we had collapsed two user workflows into one. This combined UI was oriented around our implementations of hosting rather than around the underlying user goal of delivering a model to a specific model-consumer.

My next step was to map out the two user workflows so that I could separate them.

Two user workflows

1. Model development

Data scientists develop models through an iterative process. The end of the process is creating a version of a model that can make predictions with accuracy. That version is deployed, and as time goes by, it encounters limitations that require it to be further iterated on and deployed again.

2. Model delivery

A given stakeholder consumes a given model in a relatively fixed way. A team that needs predictions will use the same resource over time, not caring or needing to know about the iteration that is happening internal to the data science team.

For example:

• An insurance underwriter will need an estimate of a what a given customer will cost in order to know what rate to quote them. As they work, they will use an app to find out the right rate for each customer. They will bookmark this and find it in the same place every day.
• An app that relies on a prediction will consume a model as an API. It will make a specific query to a specific URL to obtain predictions.
• An analyst will use a PDF report to understand trends in some area over the last month. They always want the latest report, and they will bookmark it and expect to find the latest whenever they navigate to it.

In all of these examples, the model consumer needs continuity. As long as the model works and can be found in the same place, it is irrelevant to them what updates and changes happen on the data science team's side.

Design

I created a design that solved these issues by 1. separating Versions into its own separate entity and 2. combining the various Hosting methods as config within a single entity. I went through 5 rounds of user testing and iteration. After each round, I showed my findings with PM and Engineering leads and we collaborated to refine the solution.

By the end, we had a design that was very successful in supporting the user goals and tasks we had identified. Engineers were excited because this outlined clean solutions to the technical limitations we had encountered. PMs were excited because the proposed design had a clear, low-effort path to scaling to future features and to new personas we hadn't supported before.

I worked with the PM and Eng leads to plan an incremental approach to implementing the vision. To make a case for prioritizing this work, they used some existing initiatives on both the Eng and Product sides. On the Product side, we had partnerships that would require us to scale to new deployment types, and on the Engineering side we had a tech debt initiative that would require rewriting some UIs and the underlying APIs.

Solution

My solution oriented around the two workflows outlined above, creating one entity (and corresponding section of IA) for each workflow.

Model Versions

One portion of the UI was focused on managing the connection from model development to deployment.

It created an explicit model version object that could be moved between deployments and reopened in other parts of the product to analyze and iterate on.

Deployments

The other portion of the UI was focused on how models would get into the hands of model consumers.

Each deployment is a durable representation of a model being delivered, at a specific location for a specific usecase. There are clear indicators for whether it is being delivered successfully. A settings section contains the hosting-specific config, bringing consistency across different hosting types. This reduced complexity for users and allowed the UI to scale to any future hosting technology.