• about

• process

• case studies

• contact

Problem

As I was working on MLOps tools, I noticed a pattern in how Data Scientists talked about their relationships to others in their businesses. I got the sense that there was often tension in their relationships with non-technical stakeholders, and I hypothesized that Data Scientists were focusing on statistics-oriented metrics rather than the business outcomes that other stakeholders valued.

Knowing the amount of impact data science can have for businesses, and knowing how much data science teams cost, I thought there might be an opportunity to solve a valuable problem for companies.

Process

Generative research

I started by reaching out to Melissa Federoff, a researcher who I really enjoy working with. We kicked off a set of interviews with 5 Data Scientists and 5 of their business stakeholders.

We found a clear signal that there was a problem: Lots of data science projects were getting thrown away and completely redone before being successfully delivered. From the stories and accompanying estimates that we were hearing, it sounded like 50-60% of data science resources were getting wasted.

So, I quit my job to start working on this full time!

Problem definition

As we worked to define the problem, we identified two main personas with related goals and problems.

Data Scientist

• Goal: Understand what needs to be modeled and deliver it to the business stakeholder
• Problems:
  • Lack of clarity about what the business stakeholder needs
  • Models don't get used
  • Wasted work redoing or discarding models
• Quote: “It breaks our hearts when they don't use what we build"

Business Stakeholder

• Goal: Enable any role or team member to have more impact by helping them make better decisions (using ML)
• Problems:
  • Lack of background needed to formulate requests to data science team
  • Difficult to interpret data science output
• Quotes:
  • “There should have been one single score. Instead there are all of these plots”
  • “Data Scientists get super detailed about things that seem unimportant—they don’t have business judgment.”

Initial design

Foundational work

I started by outlining data science project workflows based on the stories we had heard and on my existing domain knowledge. I wanted to use the workflows to help us align on the main user actions that our product would support and how the features and objects in our product would need to flow across them.

User flows

Our key user flows contained three key phases: requirements, development, and delivery.

Domain model

Together with this, I defined a domain model showing the objects that would support the user flows. The key object was a project. Projects would contain "decisions" which represented the core pieces of alignment.

Combining these two assets, I set the bones of the product: Projects would flow through a lifecycle, existing in one phase at a time. Each phase would have its own project UI, corresponding to the unique goals of that phase.

Solution ideas

I had read the book Data Science for Business and seen that it outlined similar problems, as well as a set of approaches that helped bridge the gap between data science and business outcomes. A main innovation in the book was the "expected value framework". With permission from the author, I adapted this to software.

If folks followed it, I was certain that it would help the business stakeholders map their problems to data science process and help Data Scientists understand the business context. The first version was pretty complicated....

Requirements gathering UI

A requirements gathering UI that frames questions both personas should understand. It also outlines a set of decision-making steps to help ensure alignment.

Expected value framework UI

A UI for guiding both personas to collaborate on the "Expected Value Framework"

Validation

When we showed this design to users we heard two main themes.

All participants found value in the concept

We heard:

• “I would love this for every project Iʼm on”
• “I could see what the data scientist is working on without having to badger them”
• “It would fill a gap for something that isnʼt being done today”

There was hesitancy around adoption

We also heard:

• “We could use your prompts in an existing tool and get the same benefits”
• “We would only want to use this if it avoids duplicate work across projects. We would need integrations."

Our takeaway was that our problem is real and that users see value. We decided to ship first and then iterate to get to something that was differentiated and more feature-rich.

In hindsight, we should've paid more attention to the "integrations" note. (*Foreshadowing...*)

Beta

We found a technical co-founder and developed this into a beta. We got 50 customers to sign up and try it. None of them continued to use it regularly.

This was highly disappointing. I fully expected to get things wrong and to learn. But having 0% adoption was a heavy blow.

Iterating to V2

As we talked to customers, the story we heard was: "This is just a Google doc."

I identified three main opportunities that were coming through in the research:

• Support model delivery: We were only addressing project definition. We could also help stakeholders to find, understand, and use models. For us as a business, solving this would mean we have a full story, supporting an end-to-end workflow, rather than just the start of a lifecycle.
• Reduce cognitive load: Our solution was seen as high effort both to create and to consume. We could move from "wall of text" to something more visual and interactive. As a business, this would help us to differentiate from existing documentation and collaboration tools.
• Help Data Science Managers: We were not helping Data Science Managers address their personal pain points. We could help them identify problem areas and bubble up status to execs. As a business, this would possibly lead to more sales by directly solving a pain point of our buyer persona.

We decided that differentiating from existing tools was the key thing that we could do to try to get adoption. This would also avoid prematurely expanding our surface area. Instead, we would improve our existing features by making them more visual and adding richer functionality.

Solution

V2 of our beta was a visual flowchart of project requirements, which focused on a few points of improvement:

• Add a visual/flowchart representation of the project requirements
• Support inline commenting within the requirements
• Use generative AI to create an image of the deliverable
• Support live collaboration, with the ability to see other stakeholders and their work in real time

Flowchart-based requirements gathering

We reimagined the background information and expected value framework as an interactive flowchart with inline comments and live collaboration

Diff-mode

This also made it much easier to represent the changes that happened between two versions of the project

After shipping this, we still struggled to gain adoption. We repeatedly heard that folks would need more features and more integrations in order to adopt. We didn't have runway to build these and so ultimately we cut our losses.