A Guided Framework For Evaluating Defensible AI Strategies for Products & Startups

This post provides a series of cohesive steps for evaluating the AI strategy of your product, startup, or any context where you're considering implementing AI. It reflects how we think through our AI strategy at Sentur , and I hope it will be useful to others.

Note: This is the first attempt at putting this framework together. This is not a perfect framework but rather a general guideline, so I welcome feedback, refinements, and further contributions. Feel free to poke holes or add insights. I will try to incorporate your feedback and comments as they come in and expand and refine the model for future versions.

So let's start.

Step 1: Scoping

Before diving into each step, it is important to outline few foundational steps that are critical to applying this guided framework.

1. Acknowledging the Hype: AI today is a VERY big hammer looking for nail. The hype around AI in the market makes it really hard to think rationally about it. There is a huge FOMO across the industry with little guidance. Let's first acknowledge that fact.

2. Scoping the Problem: To ensure our mindset is focused on what truly matters, it’s crucial to orient ourselves around problems rather than solutions. As engineers, our enthusiasm often gravitates towards creative solutions without spending enough time pondering the real problem that needs a solution. We don't spend enough time asking: How critical is this problem to our customers? Do we even need to solve it now? Are we using a sledgehammer to crack a nut?

3. Defining the Use-Cases: Now that we are oriented around problems, let's zoom-in further. Let's now outline these problems as use-cases with outcomes rather than features. What are the use-cases that are enabling your business value-proposition? Think of the hardest use-cases to implement regardless of whether implementing them is in the realm of magic! Here are a couple of examples of use cases:

• Identifying real-time fraud detection in financial transactions. This use-case is directly tied to a critical business need — reducing fraud.

• Identify tumors in an X-Ray or MRI enabling 10x faster, 10x accurate diagnosis of cancers. This is a very clear use-case.

Step 2: The Outcome / Accuracy Spectrum

Once you have the use-cases outlined, it is important to try to map out the critical success factors of each use case. What are you driving towards? Accuracy or Outcome.

Below is a visual that maps the two axes and few example use-case domains.

Identifying which axis you are striving towards is an important step as it will help guide the next step of your strategy: Data vs. Integrations

Step 3: Data vs. Integrations

If you are striving for accuracy, then Data becomes the most critical piece of your AI strategy and you need to be able to evaluate the following questions:

1. Do you have enough quality data to deliver on your use case? If not, do you have the means to collect that data?

2. How do you match against others in the market? Do they also have access to the same data you have access to?

The answer to both questions above will give you an indicator to how solid your strategy is. If you don't have access to quality data and/or you are using the same public data that others are you using, you really are in a bit of a tough spot and you should re-consider your use-cases. Otherwise, you are likely to have a somewhat defensible strategy.

If you are striving for outcomes, then the integration of AI as part of the use-case becomes the most critical piece.

1. Where in the use-case is AI delivering 10x or 100x productivity or speed?

2. Where in the use-case are you using AI to solve a problem that was not solvable before?

3. Where in the use-case are you delivering a capability that was never possible before?

In outcome driven use-cases it is all about where and how AI is being integrated as part of the use-case.

Step 4: Metrics & Rate Of Learning

I think of AI a bit like a living being; it is either growing healthier or growing sicker. There is no static state that will hold true over time.

So, whether you are aiming for Accuracy or Outcome, relying on an existing knowledge base—be it your own current data or pre-trained public models like ChatGPT-4 or LLAMAx—you need to build a feedback loop to allow you to harness continuous learning and improvement. This feedback mechanism ensures that your AI systems adapt over time, correcting biases, refining algorithms, and enhancing performance as new data comes in. By actively integrating these insights, your AI can evolve, remain relevant, and deliver more precise and effective results.

In addition to having clarity on the feedback loop and it's mechanism, you are in need to also define the 'Metric' that will be used to evaluate whether your accuracy is going up or down OR whether you are driving outcomes in a positive or negative direction.

So to summarize, you need:

1. The evaluation metric. i.e. speed, % accuracy compared to a benchmark...etc.

2. The learning mechanism.

Step 5: Big Providers Threat

I'm including this step because it likely weighs on the minds of every founder in the AI space:

Will OpenAI or another major player render my value proposition obsolete in a future update?

Operating under the looming threat of giants like OpenAI or Google can feel daunting. However, utilizing the framework outlined earlier can help you identify and concentrate on the unique value you provide. It’s crucial to be able to clearly articulate this value whether it is exclusive data, a specific use case, a superior rate of learning, or all of the above.

Sam Altman in a recent podcast shared a bit of their high-level guidance to startups: He did share that if you are betting that the models today are insufficient and you are focusing your efforts on surpassing these technologies in knowledge/skill, then you will be outpaced out of business.

I will add two caveats to that statement:

a) If you have access to data they don't, then you have an edge of specialization.

b) If you dominate a use case within a specific domain and provide comprehensive integration that they do not, you can sustain your competitive edge.

In closing, I hope you find this useful and insightful. Feel free to reach out with comments and feedback!

Enjoy

At Sentur we are on a mission to generate vast social change by revolutionizing mental health care through the comprehensive treatment of trauma. We invite you to be a part of this transformative journey. Subscribe to this newsletter / follow me or the Sentur LinkedIn page.