examples/voice_agent_webrtc_langgraph/agents/healthcare-agent/TESTING_GUIDE.md

Healthcare Agent Testing Guide

This guide provides test scenarios to validate the improved tool calling behavior, especially the secret question authentication flow.

Test Patients

Two test patients are available in mock_data/patients.json:

Patient 1: John Marshall

• Full Name: John Marshall
• DOB: 1960-01-01 (January 1st, 1960)
• MRN Last-4: 0001
• Secret Question: "What is your favorite color?"
• Secret Answer: blue
• Allergies: Penicillin
• Medications: Acetaminophen 500mg PRN

Patient 2: Francesco Ciannella

• Full Name: Francesco Ciannella
• DOB: 1990-01-01 (January 1st, 1990)
• MRN Last-4: 6001
• Secret Question: "What city were you born in?"
• Secret Answer: rome
• Allergies: NKA (No Known Allergies)
• Medications: Lisinopril 10mg daily

---

Test Scenarios

✅ Test 1: Secret Question Flow (MOST IMPORTANT)

Purpose: Verify the agent correctly handles the two-step secret question authentication.

Steps:

1. Say: "My name is John Marshall"
2. Agent should call find_patient() and greet you
3. Say: "January 1st, 1960" (DOB)
4. Agent should call verify_identity(dob=...)
5. EXPECTED: Agent should ASK: "For security, what is your favorite color?"
6. Say: "Blue"
7. Agent should call verify_identity(dob=..., secret_answer="Blue")
8. EXPECTED: Agent confirms verification and asks about symptoms

Success Criteria:

• ✅ Agent explicitly asks the secret question
• ✅ Agent waits for your answer before proceeding
• ✅ Agent confirms "You're verified" after correct answer
• ✅ Agent does NOT ask for MRN last-4

Common Failure Modes (Old Behavior):

• ❌ Agent skips asking the secret question
• ❌ Agent asks for MRN last-4 instead of secret question
• ❌ Agent claims verification before getting secret answer

---

✅ Test 2: Wrong Secret Answer

Purpose: Verify agent handles incorrect secret answers gracefully.

Steps:

1. Say: "My name is John Marshall"
2. Say: "January 1st, 1960" (DOB)
3. Agent asks: "What is your favorite color?"
4. Say: "Red" (WRONG - correct is "blue")

Expected Behavior:

• ✅ Agent says verification failed
• ✅ Agent asks for MRN last-4 as alternative: "Could you provide the last 4 digits of your MRN?"
• ✅ Agent does NOT proceed to medical questions

---

✅ Test 3: MRN Last-4 Path (Alternative Auth)

Purpose: Verify agent accepts MRN last-4 without asking secret question.

Steps:

1. Say: "My name is John Marshall"
2. Say: "January 1st, 1960, and MRN last-4 is 0001"

Expected Behavior:

• ✅ Agent calls verify_identity(dob=..., mrn_last4="0001")
• ✅ Agent confirms verification immediately
• ✅ Agent does NOT ask secret question (since MRN provided)

---

✅ Test 4: Allergy Awareness

Purpose: Verify agent considers allergies before recommending medications.

Steps:

1. Complete authentication as John Marshall
2. Say: "I have a headache and a sore throat"
3. Agent calls get_patient_profile_tool() and triage_symptoms_tool()

Expected Behavior:

• ✅ Agent mentions: "You have a penicillin allergy"
• ✅ Agent recommends Acetaminophen (which patient already has)
• ✅ Agent does NOT recommend penicillin-based antibiotics
• ✅ Agent references existing medication: "Since you're already taking acetaminophen as needed..."

---

✅ Test 5: Urgent Triage (Chest Pain)

Purpose: Verify agent correctly escalates urgent symptoms.

Steps:

1. Complete authentication as Francesco Ciannella
2. Say: "I'm having severe chest pain and shortness of breath"

Expected Behavior:

• ✅ Agent calls triage_symptoms_tool(symptoms_text="severe chest pain and shortness of breath")
• ✅ Returns: {risk: "urgent", red_flags: ["chest pain"]}
• ✅ Agent immediately says: "Chest pain can be serious. Please call 911 now or go to the nearest emergency room."
• ✅ Agent does NOT offer to book a regular appointment
• ✅ Agent emphasizes urgency

---

✅ Test 6: Self-Care with Appointment Booking

Purpose: Verify full flow for non-urgent symptoms with appointment.

Steps:

1. Complete authentication as John Marshall
2. Say: "I have a mild headache and feel tired, but no fever, no neck stiffness"
3. Agent triages and provides self-care advice
4. Say: "Yes, I'd like to schedule an appointment"

Expected Behavior:

• ✅ Agent calls triage_symptoms_tool() → returns risk: "self_care"
• ✅ Agent provides advice: "Try rest, hydration, and acetaminophen as directed"
• ✅ Agent offers appointment: "Would you like a telehealth appointment?"
• ✅ Agent calls list_providers_tool() → presents options
• ✅ Agent calls get_provider_slots_tool() → shows times in friendly format
• ✅ After you choose, agent calls schedule_appointment_tool()
• ✅ Agent confirms: "Booked. I'll send details to your phone ending in 0101."
• ✅ Agent calls get_preferred_pharmacy_tool() and confirms pharmacy
• ✅ At end, agent calls log_call_tool() silently

---

✅ Test 7: Francesco Ciannella Secret Question

Purpose: Test second patient with different secret question.

Steps:

1. Say: "My name is Francesco Ciannella"
2. Say: "January 1st, 1990"
3. EXPECTED: Agent asks "For security, what city were you born in?"
4. Say: "Rome"

Expected Behavior:

• ✅ Agent asks the correct secret question for Francesco
• ✅ Agent accepts "Rome" (case-insensitive)
• ✅ Agent confirms verification

---

✅ Test 8: Profile Not Found

Purpose: Verify agent handles unknown patients gracefully.

Steps:

1. Say: "My name is Jane Doe"

Expected Behavior:

• ✅ Agent calls find_patient(full_name="Jane Doe")
• ✅ Returns: {}
• ✅ Agent says: "I'm not finding you in our system. Could you verify the spelling of your name?"
• ✅ Agent does NOT proceed to verification without patient_id

---

Debugging Tips

Check Logs

Look for these log entries in app.log:

INFO - LLM tool_calls: ['find_patient']
INFO - tool find_patient result: {"patient_id": "pt_jmarshall", ...}
INFO - LLM tool_calls: ['verify_identity']
INFO - verify_identity: verified=False needs=['mrn_last4_or_secret']
INFO - LLM content: For security, what is your favorite color?
INFO - verify_identity: verified=True needs=[]
INFO - LLM tool_calls: ['get_patient_profile_tool']

Common Issues

Issue: Agent doesn't ask secret question

• Check: Tool docstring has 7-step CRITICAL SECRET QUESTION FLOW section
• Check: System prompt has explicit secret question instructions in step 2

Issue: Agent asks for MRN instead of secret question

• Fix: Ensure prompt says "If 'question' field is present: READ THE EXACT QUESTION"

Issue: Agent proceeds without verification

• Check: Prompt has "NEVER claim verification until verified=true"
• Check: Tool auto-injects patient_id

---

Expected Tool Call Sequences

Minimal Flow (MRN path):

1. find_patient(full_name="John Marshall")
2. verify_identity(dob="...", mrn_last4="0001")
3. get_patient_profile_tool()
4. triage_symptoms_tool(symptoms_text="...")
5. log_call_tool(notes="...", triage_json="...")

Secret Question Flow:

1. find_patient(full_name="John Marshall")
2. verify_identity(dob="...")                    ← First call
3. verify_identity(dob="...", secret_answer="...") ← Second call
4. get_patient_profile_tool()
5. triage_symptoms_tool(symptoms_text="...")
6. log_call_tool(...)

Full Flow with Booking:

1. find_patient(full_name="John Marshall")
2. verify_identity(dob="...")
3. verify_identity(dob="...", secret_answer="...")
4. get_patient_profile_tool()
5. triage_symptoms_tool(symptoms_text="...")
6. list_providers_tool()
7. get_provider_slots_tool(provider_id="...")
8. schedule_appointment_tool(provider_id="...", slot_iso="...")
9. get_preferred_pharmacy_tool()
10. log_call_tool(notes="...", triage_json="...")

---

Success Metrics

After improvements, you should see:

• ✅ 100% secret question ask rate (when MRN not provided)
• ✅ Zero skipped verifications
• ✅ Allergy mentions in every medication recommendation
• ✅ Proper urgent escalation for chest pain, severe symptoms
• ✅ Two-call pattern for secret question auth

---

Quick Test Script

# Test 1: John Marshall secret question
Agent: May I have your full name?
You: John Marshall
Agent: Please confirm your date of birth.
You: January 1st, 1960
Agent: For security, what is your favorite color?  ← MUST ASK THIS
You: Blue
Agent: Thank you, you're verified. What brings you in today?  ← MUST CONFIRM

# Test 2: Francesco with symptoms
You: Francesco Ciannella
Agent: Date of birth?
You: January 1st, 1990
Agent: For security, what city were you born in?  ← MUST ASK THIS
You: Rome
Agent: What's going on today?
You: I have a mild headache
Agent: [provides advice considering Lisinopril medication]  ← MUST MENTION MEDS

---

Reporting Issues

If tests fail, provide:

1. Patient name used
2. Exact conversation (user input + agent responses)
3. Expected behavior vs actual behavior
4. Log excerpt from app.log showing tool calls
5. Which test scenario from this guide