---
title: "Box and Operator Collective on Measuring AI ROI | Olivia Nottebohm and Mallun Yen"
episode_number: 8
canonical_url: https://enterprisealignedai.com/episodes/box-and-operator-collective
md_url: https://enterprisealignedai.com/episodes/box-and-operator-collective.md
last_updated: 2026-11-20
duration: "37 min"
youtube_url: https://www.youtube.com/watch?v=dX1uuHEX9GI
spotify_url: https://open.spotify.com/episode/0U3Ti5RZj0ZXmGJj7t8h65
apple_url: https://podcasts.apple.com/us/podcast/box-and-operator-collective-on-measuring-ai-roi/id1890781384?i=1000802866218
guests: ["Olivia Nottebohm", "Mallun Yen"]
tags: ["Agentic AI", "AI Native", "Measuring ROI", "Governance & Regulations", "Organization & Talent"]
---

# Ep. 8: Box and Operator Collective on Measuring AI ROI | Olivia Nottebohm and Mallun Yen

**Summary:** Host Aparna Sinha talks with Olivia Nottebohm, COO of Box, and Mallun Yen, founder and CEO of Operator Collective, about what AI native means for a 21-year-old enterprise and how companies measure AI ROI. Box has 150,000 customers, including over 65% of the Fortune 500. On the go-to-market side, Box runs role-based agents for repeated tasks. Olivia explains why customers now need a content strategy before they deploy AI, how Box treats personal productivity agents and enterprise agents differently, and how Box proves ROI to its customers. Operator Collective surveyed 123 operators for its State of AI Transformation report. 50% call their companies AI native, and 91% feel personally responsible for AI transformation. Mallun explains why CFOs will ask for proof of AI ROI. They also cover AI pricing, a 2x2 for choosing AI use cases, agent reliability, and how Operator Collective invests in AI.
## Why listen

Olivia runs go-to-market at a 21-year-old company that sells AI to regulated industries, so she speaks to both sides: how Box changes its own work with agents, and what its customers need before they deploy. Mallun brings data from Operator Collective's survey of 123 operators on who owns AI and how they measure it. If you are deciding how to govern agents or how to show AI ROI to your CFO, both perspectives apply.

## Key takeaways

- 90% of the data in the enterprise is unstructured, and AI lets companies draw insights and run workflows on it for the first time
- Because of breaches and rogue agents, customers need a data strategy and a content strategy before they deploy AI
- Box separates personal productivity agents, which see only what the user sees, from enterprise agents with shared instructions and fixed integrations
- Box's value engineers prove ROI to customers, and the ROI usually rises over time as the cost of AI falls
- Olivia's 2x2 for AI use cases: start with low-lift tasks that repeat often, because tight cycles let you tune the agent and reach ROI sooner
- In Operator Collective's survey of 123 operators, 50% call their companies AI native and 91% feel personally responsible for AI transformation
- Mallun defines AI native as being willing to blow up how you work and re-engineer it around AI
- Most survey respondents left the ROI question blank, and Mallun expects CFOs to ask for receipts
- Box's agents report confidence scores against a ground truth, at 98% to 99%, compared with human error rates in the low 90s
- Operator Collective puts about 75% of its portfolio in pre-consensus companies and 25% in consensus companies

## Timestamps

- 0:00: The Enterprise AI Reality Check
- 0:42: Meet Olivia Nottebohm (Box) and Mallun Yen (Operator Collective)
- 2:31: Box Is Not Just File Storage Anymore
- 4:08: Operators Built Operator Collective
- 5:45: Making a 21-Year-Old Company AI-First
- 7:27: Box on Box: 68 Go-to-Market Agents
- 8:33: The Biggest AI Blocker Is Your Content Strategy
- 10:07: AI Insurance Claims Processing: Sensitive Data Stays Governed
- 11:21: AI Native Means Willing to Blow Up How You Work
- 12:55: A 10-Person VC Firm That Works Like 100
- 14:11: Pricing AI Products: Seats and Consumption
- 17:00: Box's AI Agents Replaced 2,000 Meetings
- 19:24: Who Owns AI? 91% of Operators Feel Responsible
- 21:09: The AI ROI Dilemma: The CFO Will Ask for Receipts
- 22:31: Personal AI vs. Enterprise Agents
- 24:30: Measuring Personal AI Productivity With Lanai
- 25:27: AI ROI Goes Up Over Time as AI Costs Fall
- 26:11: Choosing High-ROI AI Use Cases
- 28:26: Do You Need a CMO? AI Is Blurring Roles
- 29:32: Making Agents Reliable: Confidence Scores and Ground Truth
- 30:59: Helping Enterprises Adopt AI: In-House Devs or Services
- 31:47: AI Workflow Efficiency: Prompts, Tokens and Context Retrieval
- 32:57: Operator Collective's AI Bets: 75/25
- 34:42: Regulated Companies Go Full AI in 24 Months
- 35:59: Why People Still Matter Most

## Clips

Short segments published from this episode. Each is citable on its own.

- **The Biggest AI Blocker Is Your Content Strategy**
  - URL: https://www.youtube.com/watch?v=AZLABfiv7Gw
  - Speaker: Olivia Nottebohm
  - Occurs at: 9:07
  - Duration: 0:22
  - Claim: Customers have to get the data layer right.
- **91% of Operators Feel Personally Responsible for AI**
  - URL: https://www.youtube.com/watch?v=-2ZuXIZeSTk
  - Speaker: Mallun Yen
  - Occurs at: 19:44
  - Duration: 0:20
  - Claim: 91% of operators feel personally responsible.
- **ChatGPT and Claude Agents vs Enterprise Agents**
  - URL: https://www.youtube.com/watch?v=xw1IYvSknxE
  - Speaker: Olivia Nottebohm
  - Occurs at: 22:31
  - Duration: 0:55
  - Claim: Box hardens the agents it puts in production.
- **AI Native Means Willing to Blow Up How You Work**
  - URL: https://www.youtube.com/watch?v=AvypHm-43As
  - Speaker: Mallun Yen
  - Occurs at: 11:41
  - Duration: 0:38
  - Claim: AI native is a mindset.
- **AI ROI Goes Up Over Time as AI Costs Fall**
  - URL: https://www.youtube.com/watch?v=zS52By0ZIoU
  - Speaker: Olivia Nottebohm
  - Occurs at: 25:27
  - Duration: 0:34
  - Claim: ROI usually rises as AI costs fall.
- **The Jensen Token-Maxxing Era Is Over: Now Measure AI ROI**
  - URL: https://www.youtube.com/watch?v=B2yRBcLqWHo
  - Speaker: Mallun Yen
  - Occurs at: 22:08
  - Duration: 0:14
  - Claim: The CFO will ask for receipts.

Olivia Nottebohm is COO of Box, where she runs the global go-to-market organization. Mallun Yen is founder and CEO of Operator Collective, a venture firm whose limited partners are mostly operators. Olivia is also an LP in Operator Collective. The conversation covers how an established enterprise becomes AI native, and how companies measure what AI returns.

## The content layer comes first

Olivia sees customers move from asking what AI can do to asking whether their data is ready:

> "They deeply understand that they need a data strategy and even they need a content strategy. And if you were to ask people two years ago, 'Do you think the biggest blocker will be to deploying AI?' They were not saying, 'Oh, we need to be really clear about our content strategy.' But where we are at today, because of everything we're seeing in how AI is getting employed and where you see breaches and where you see rogue agents, it is imperative that customers get that data layer right and the access to the data layer right."

## AI native is a mindset

Operator Collective surveyed 123 operators. Half described their companies as AI native, including companies more than 20 years old. Mallun explains what that means:

> "AI native is being willing to blow up how you did work before, to say, if we have now AI, it's not about doing the same thing faster, it's about re-engineering in a way that is optimized when you use the AI tools."

## Who owns AI

The survey asked who is responsible for AI transformation. Answers ranged from the CEO to a committee to every employee. Mallun:

> "Regardless of what the structure was, 91% of the respondents said, 'I feel personally responsible.'"

## Personal agents and enterprise agents

Olivia separates two kinds of agents. A personal productivity agent, such as Gemini, ChatGPT or Claude, only has access to what the user has access to. Box hardens the enterprise agents it puts into production:

> "It's been hardened because it has those shared instructions, but it's also been hardened because it's got fixed integrations into the appropriate other data sources that we approve. So by definition, it can't run amok."

## Measuring AI ROI

Most survey respondents left the ROI question blank. Mallun expects that to change:

> "I think you can feel the value of AI before you know how to measure it, but it doesn't mean you shouldn't be doing AI if you don't know how to measure it yet. But at some point, the CFO is gonna ask for receipts."

Box has to prove ROI to its own customers, and its value engineers track it over time:

> "Because the cost of AI is getting less and less, usually what we see is the ROI increases over time, which is really wonderful to see."

## Choosing AI use cases

Olivia maps use cases on two axes: how hard the task is, and how often it repeats. She recommends starting with low-lift tasks that repeat often. A bank's loan decision can drop from two weeks to one hour, and it runs thousands of times a day, so the team can tune it quickly. A monthly research report is a big lift that runs rarely, so it takes longer to get right.

## The next two years

Olivia expects some customers to run nearly every process agentically within two years, with humans in the loop. She sees 24 months as the point when regulated and security-oriented companies move to a full AI world. Mallun closes on people:

> "Companies succeed because of the people who build them."