AI Scheduling Agent Evaluation: Build a Hard-Meeting Test Suite

Build the hard-meeting evaluation in eight steps

Use one base fixture: a sixty-minute customer kickoff inside a five-business-day window. Four people are required, one is optional, two companies use Google and Microsoft, and one required person will not connect a calendar. Every test mutates this fixture while preserving one meeting request ID.

1. 1

   Define terminal success before writing prompts

   Write a machine-checkable completion contract: one canonical event, approved start and end, meeting time zone, correct organizer, every required participant, the final optional-attendee decision, and verified invitation dispatch. Also define terminal non-booking outcomes such as required decline, deadline expiry, revoked authority, and no overlap after permitted changes.
   • An HTTP 200, tool success, or model success sentence does not pass the evaluation.
   • A bounded exception names current evidence, one owner, and one allowed next action.
   • Cancellation and rescheduling use separate authority and completion contracts.
2. 2

   Freeze a typed meeting-intent fixture

   Store meeting_request_id, revision, organizer, purpose, duration, date window, deadline, IANA time zone, location, required people, optional people, approved substitutes, working-hour rules, buffers, outreach policy, approval mode, and allowed calendar writes. Keep the source instruction, but grade the system against typed state rather than its interpretation of old prose.
   • Reject a fixture with no organizer, duration, bounded window, deadline, or required participant.
   • Every proposal, approval, message, and booking attempt points to the current revision.
   • Mutating required people, duration, or date bounds invalidates stale proposals.
3. 3

   Build the participant and availability matrix

   Vary required versus optional status, Google versus Microsoft, connected versus unconnected calendars, known versus ambiguous time zones, and approved versus denied outreach. Normalize provider free/busy and participant-provided windows into intervals with source, freshness, zone, consent state, and error state. Missing evidence is never treated as open time.
   • A required participant with no connection receives a consent-safe availability path.
   • An optional decline changes scoring without changing required quorum.
   • Private event titles and message bodies stay outside routine evaluation output.
4. 4

   Test conversation and time-zone ambiguity

   Feed replies such as ‘Tuesday after lunch,’ ‘Friday morning,’ a date without a year, and a daylight-saving transition. Grade whether the agent asks the minimum clarification, preserves the original response, and creates normalized windows only after date and zone are authoritative. Run reminder timing in each participant’s local zone.
   • The agent never guesses a time zone from an email domain or phone number.
   • A correction supersedes old evidence while preserving the audit trail.
   • Silence from a required person remains a wait or exception, not consent.
5. 5

   Inject stale state, duplicate delivery, and worker restarts

   Change a required calendar after proposals are generated. Deliver the same participant reply twice. Restart the worker after outreach, after approval, and during booking. Repeat REST commands and MCP tool calls with the same command identity. The system should resume from durable meeting state, rebuild stale proposals, and suppress duplicate side effects.
   • Outreach, approval, and booking use separate idempotency keys.
   • A duplicate webhook or reply does not create another state transition.
   • A restarted worker does not need the original model transcript to recover.
6. 6

   Test consent, approval, and human reclaim

   Deny calendar access, revoke it after a read, opt a participant out of reminders, expire organizer approval, and add a sensitive participant after approval. Grade whether calendar reads, participant contact, slot approval, event creation, update, cancellation, and rescheduling are treated as distinct permissions. The agent must stop when authority ends.
   • Possessing an address does not grant permission to contact it.
   • Approval binds to the current people, proposal, organizer calendar, and revision.
   • A human can pause, revise, reclaim, or end the workflow without hidden work continuing.
7. 7

   Break the booking commit on purpose

   Return a timeout after the provider creates the event, omit one required attendee, reject one invitation, and move a calendar between final check and write. The agent must reconcile the organizer calendar before retrying, repair an allowed partial failure, and return one canonical provider event identifier. Blind retry is a failing grade.
   • An uncertain write enters reconciliation rather than immediate replay.
   • The final read-back verifies time, zone, organizer, attendees, and invitation state.
   • Repeated booking commands return the original result rather than a second event.
8. 8

   Score completion, recovery, and operating evidence

   Score each fixture on intent fidelity, required attendance, cross-company reach, optional-sync fallback, time-zone correctness, consent, duplicate suppression, recovery, observability, and terminal proof. Record active processing separately from participant wait. Compare engineering and support work needed to pass the suite, not only first-call latency.
   • Every failure has a stable reason code and the meeting timeline needed to reproduce it.
   • Expected participant wait is not reported as infrastructure failure.
   • A product passes only the fixtures that finish or stop inside declared policy.

Compare evaluation depth by the work it proves

A useful test does more than confirm an endpoint. It proves whether the product owns the hard meeting across people, providers, elapsed time, and failure.