Voice Scheduling Agent over MCP

What it is

A proof of concept for scheduling imaging appointments over the phone. A caller talks to a Google CES voice agent, and the agent does the actual work by calling tools on an MCP service. The service verifies who is calling, finds the physician’s imaging order, searches appointment availability, and books the slot the caller picks. It runs on synthetic patients only.

I built the MCP service, connected it to the voice agent, and built the dashboard used to follow each call.

The call flow

The voice channel passes in the caller’s phone number, which the agent uses to look up a caller record. The phone number is treated as context, not authentication. The agent still confirms the caller’s name and date of birth before it can see any patient-specific orders.

From there it identifies the requested imaging service, reads the ordering office’s notes and screening questions, searches locations and open slots, and reads the exact appointment back to the caller. It only books after the caller confirms, then reads the appointment back from the scheduling system to verify it.

Why the MCP service is the boundary

The agent never talks to the scheduling API directly. Every action goes through a small set of MCP tools served over streamable HTTP with FastMCP. That keeps the rules in one place: identity verification before any order is exposed, a write guard that has to be explicitly enabled before anything can be booked or cancelled, and service codes that come from the service catalog instead of being made up by the model.

Two backends, one contract

The service runs against either the live FHIR-style scheduling API through OAuth, or an offline mock that keeps the same method signatures. The live test environment could return availability while failing on booking or cancellation, so the mock is what made it possible to test the agent and dashboard end to end. Switching backends is one environment variable, and nothing on the agent side changes.

Demo bookings also take a temporary lease of about 20 minutes. Cleanup uses the lease to cancel the appointment and reset the order so the same demo can run again.

Journey dashboard

A small FastAPI app with plain HTML, JavaScript, and CSS. It is read-only. It shows the conversation list, caller and order context, screening answers, the booked appointment, and a timeline of every event in the call, read from the shared state the MCP service writes to Cloud Storage. It never books anything itself.

Results

In demos, calls took about 3 to 5 minutes. The agent held up against testers trying to throw it off, and every call that went all the way through booked the appointment the caller asked for. The scope was deliberately narrow, a handful of imaging services and one task, which is part of why it held up.