Conversational AI operational chatbot

EVA is a conversational AI agent that turns operational data into decision-ready answers with built-in trust and verification, allowing Managed Services team leaders to validate metrics on demand. This was an internal tool meant to support the unique and established workflows of the Managed Services team.

I was the sole product designer, working with a PM, and an engineer. This project took around 2 months to test a working MVP prototype.

Key insights

My PM interviewed the Head of Managed Services and several Operations Managers about how they monitored team performance. Everyone landed in the same place: leadership reviewed metrics via static reports, workbooks, and exports, but data entry was time-consuming and disorganized. Tracking performance was also slow, inconsistent, and sometimes just inaccurate.

Two very different jobs to be done fell out of that research:

1) Leadership needed a macro-level view they could defend in a meeting, without doing the calculations themselves.

2) Operations managers needed fast, granular answers broken down by team, queue, case type, or individual, so that they could act on it.

I audited high-caliber AI products like Perplexity and AI UX pattern libraries to see how they build trust. I found that trust grew from a solid retrieval policy, not the interface. For conversational AI design, substrate-level decisions—which sources get prioritized, how conflicting data gets ranked, what guardrails are in place—are the deciding factor, which informed my design approach.

“Be skeptical of the marketing claims being made by AI tools... Unnecessary AI chatbots and features can harm rather than help users.”

I read as much of that research as I could find, plus articles and podcasts on designing for AI, before designing a single flow. That set the bar: this only shipped if it was more trustworthy than the status quo—not if it was more novel.

The first version was intentionally simple: a raw text field and one shared view for every user. Ask a question, get an answer—like your typical AI chatbot.

It worked... but also broke fast. First, raw text input meant anyone could ask out-of-bounds questions, which was expensive and unsupported by our data. Secondly, one generic role view couldn't simultaneously provide a five-minute glance for Managed Services leaders and a granular analysis for Operational Managers.

I disabled raw text input entirely and replaced it with pre-configured questions. This meant every question that could be asked was one EVA could answer with verified data. I also split the chatbot into two role views. Leadership accessed higher-level questions whereas operational managers accessed more granularity.

Even if EVA gave the right answer — why should anyone trust it? My first instinct was citations. I explored two approaches.

1 — Appended citations

Rejected

Gave no context for what the data actually said on the same page.

2 — Inline citations

Rejected

Took two or more clicks to verify anything, working against the speed leadership needed.

Looking closer, citations weren't solving the actual problem. For an operational metric, users needed more specificity: how did you get this number, and can I defend it if someone challenges me on it?

Chain-of-thought disclosureShipped

Users can view how the model reasoned its way to the answer. However, reasoning in real time still didn't give leadership enough to validate an operational decision.

Proof modeShipped

Users can expand any answer, in place, into its calculation details and the data it was sourced from. This closed the gap chain-of-thought alone couldn't: leadership could validate a number before using it in an executive discussion, and operations could trace exactly where a number came from.

Suggested follow-up questionsShipped

I included suggested follow-up questions after a new chat initiated to raise the success rate per prompt rather than the number of prompts. Every additional prompt is a real token cost to the business, so "more engagement" opposed EOX's business needs.

Before

After

With pre-configured questions, the finished chatbot looks constrained, almost simple. That was intentional. EVA was never meant to be a general-purpose AI assistant. As an internal tool, it was built specifically around the EOX Managed Services team's existing workflows, data, roles, and constraints. Open-ended prompting would have let people ask questions the data couldn't support, and every one of those queries would have cost token money for no guaranteed value.

After demoing our MVP to our target users, we received the following feedback:

• Reduced ad-hoc reporting requests to analysts, with improved trust in the numbers themselves
• Strong signals of adoption and alignment across both roles

For a complex enterprise product, especially for an internal tool, we weren't trying to max out AI chatbot capabilities. For EOX Vantage, good design looked like a product that understood organizational workflows, technical architecture, buesiness needs, and different user needs. In the end, that might look like a simpler, more constrained design, but that was the solution that worked for EOX's users, and what moved the needle.