Fluxloop
AI Agent Evaluation Report
Customer Service Agent v2.1
2025-01-15 14:30:00
"Evaluate the agent's ability to handle customer inquiries, process refunds, and resolve complaints effectively."
Table of Contents
Overview
Test Date
2025-01-15 14:30:00
Eval Goal
Evaluate the agent's ability to handle customer inquiries,
process refunds, and resolve complaints effectively.
✓
58%
Pass Rate
traces met the evaluation criteria
7 of 12 traces passed
The agent handles routine inquiries well but struggles with refund processing and frustrated customers. Tool usage gaps and occasional hallucinations require attention before production deployment.
!
2
Marginal
✗
2
Failed
?
1
Review
!
Attention Required
5 traces
| Trace | Input | Status | Issue | Reason |
|---|---|---|---|---|
| trace-... | "This is ridiculous! My order #12345 hasn't arrived and it's been 2 weeks!" | ✗ Fail | Hallucination detected | Agent fabricated shipping details without querying the order system. |
| trace-... | "I want a full refund immediately. The product was damaged." | ✗ Fail | Task not completed | Refund tool was available but agent only provided general policy info. |
| trace-... | "Can you expedite my order? I need it by tomorrow." | ! Marginal | Partial completion | Agent checked order status but didn't attempt to expedite shipping. |
| trace-... | "I think there's a billing error on my last invoice." | ? Review | Ambiguous outcome | Agent identified a discrepancy but escalation path was unclear. |
| trace-... | "The tracking says delivered but I never got the package." | ! Marginal | Incomplete resolution | Agent verified tracking but didn't offer replacement or refund options. |
What do the numbers say?
Completeness ?
Task Completion Rate
75%
(9/12)
!
Fair
Correctness ?
Hallucination Rate
92%
(11/12)
✓
Good
Relevance Rate
100%
(12/12)
✓
Good
Tool Usage Appropriateness
83%
(10/12)
!
Fair
Response Quality ?
User Satisfaction Score
67%
(8/12)
!
Fair
Response Clarity
92%
(11/12)
✓
Good
Persona Consistency
83%
(10/12)
!
Fair
Efficiency & Performance ?
Output Tokens
342 tokens
P50: 298, P95: 512
!
In range
Conversation Depth
2.3 turns
P50: 2, P95: 4
!
In range
Latency
3.2s
P50: 2.8s, P95: 5.1s, P99: 6.2s
!
In range
Persona Performance Gap
Latency: +18.5%
Tokens: +12.3%
!
Info
Trace Matrix
All Traces Evaluation Matrix
?
What do these icons mean?
| Trace | Overall | Task | Halluc. | Relevance | Tool | Satisf. | Clarity | Persona |
|---|---|---|---|---|---|---|---|---|
| trace-001 | ✗ | ✗ | ✓ | ✗ | ✗ | ✓ | ✓ | ✓ |
| trace-002 | ✗ | ✓ | ✓ | ✗ | ✗ | ✓ | ✓ | ! |
| trace-003 | ! | ✓ | ✓ | ! | ! | ✓ | ✓ | ✓ |
| trace-004 | ! | ✓ | ✓ | ✓ | ! | ✓ | ! | ✓ |
| trace-005 | ! | ✓ | ✓ | ! | ! | ✓ | ✓ | ✓ |
| trace-006 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| trace-007 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| trace-008 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| trace-009 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| trace-010 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| trace-011 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| trace-012 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
Conversation Detail
Duration:
Persona:
Turns:
Select a trace above to view the conversation
Issues & Fails
Failed & Review Cases
?
What do these icons mean?
| Trace | Overall | Task | Halluc. | Relevance | Tool | Satisf. | Clarity | Persona |
|---|---|---|---|---|---|---|---|---|
| trace-001 | ✗ | ✗ | ✓ | ✗ | ✗ | ✓ | ✓ | ✓ |
| trace-002 | ✗ | ✓ | ✓ | ✗ | ✗ | ✓ | ✓ | ! |
Failed Cases (❌)
❌
Hallucination
trace-001
Input
"This is ridiculous! My order #12345 hasn't arrived
and it's been 2 weeks!"
Persona
frustrated_customer
Failed Metrics
✗ Hallucination
✗ Tool Usage
Issue Summary
Agent fabricated shipping details without querying the order system.
Conversation Timeline
Turn 1:
Customer expresses frustration about delayed order
#12345.
Turn 2:
Agent claims order was shipped via
UPS on Jan 5 with tracking 1Z999AA1, without any tool call.
Turn 3:
Customer asks for more details about the
shipment.
Turn 4:
Agent provides estimated delivery of
Jan 8, still without verifying in the system.
❌ Response contains fabricated shipping and tracking information
LLM Judge Reasoning
"The agent generated plausible-sounding shipping details to appear helpful, but no order_lookup or tracking_check tool was invoked. This is a critical failure as the customer may act on false information."
ROOT CAUSE
When handling frustrated customers, the agent prioritizes speed and apparent helpfulness over accuracy. The fallback for failed lookups generates fake data instead of admitting uncertainty.
💡
Add a hard constraint: shipping details can only be provided if order_lookup
returns valid data. On failure, apologize and offer to escalate.
❌
Task
Incomplete
trace-002
Input
"I want a full refund immediately. The product was
damaged."
Persona
frustrated_customer
Failed Metrics
✗ Task Completion
✗ Tool Usage
Issue Summary
Refund tool was available but agent only provided general policy info.
Conversation Timeline
Turn 1:
Customer demands immediate refund for damaged
product.
Turn 2:
Agent acknowledges the issue and explains the 30-day
return policy.
Turn 3:
Customer confirms they want to proceed with the
refund.
Turn 4:
Agent says 'I've noted your request'
but doesn't invoke the refund tool.
❌ Available tool not invoked for explicit customer request
LLM Judge Reasoning
"The agent understood the customer's intent and even confirmed eligibility, but failed to take the final action of processing the refund. This suggests incomplete intent-to-action mapping."
ROOT CAUSE
The agent's refund handling stops at eligibility confirmation and doesn't automatically proceed to process_refund when customer consent is clear.
💡
Update refund flow: when customer explicitly requests refund AND eligibility
is confirmed, auto-invoke process_refund with customer confirmation.
Review Needed Cases (🔍)
🔍
Needs Review
trace-004
Input
"I think there's a billing error on my last
invoice."
Persona
power_user
Quality
Issue Summary
Agent identified discrepancy but escalation path was unclear.
Conversation Timeline
Turn 1:
Customer reports potential billing error on
invoice.
Turn 2:
Agent retrieves invoice and confirms there's a
$47.50 discrepancy.
Turn 3:
Customer asks what happens next.
Turn 4:
Agent says 'This will be reviewed'
without specifying timeline or next steps.
🔍
LLM Judge Reasoning
""
Why Review Needed:
💡
Response Quality
Quality Metrics Summary
67%
User Satisfaction
! Fair
8/12 passed
92%
Response Clarity
✓ Good
11/12 passed
83%
Persona Consistency
! Fair
10/12 passed
Key Observations
- Pass rate of 58% (7/12 traces) indicates room for improvement in handling complex scenarios.
- Refund and escalation workflows consistently underperform, with 0/3 refund requests fully resolved.
- Tool usage gaps observed in 2 cases where available tools weren't utilized effectively.
- Response clarity is strong at 92%, with clear and well-structured answers in most cases.
- Frustrated customer persona triggers defensive responses that miss resolution opportunities.
Marginal Success Cases (⚠️)
⚠️
Partial Completion
trace-003
Input
"Can you expedite my order? I need it by
tomorrow."
Persona
new_customer
Quality
! Task
! Satisfaction
Issue Point
User:
Order #67890
Assistant:
I found your order. It's scheduled for delivery on
January 18th via standard shipping. Unfortunately, since it's already
been shipped, we cannot change the shipping method.
⚠ Agent verified order status but didn't attempt to contact carrier for
expedited handling.
💡
Add carrier API integration to request
expedited handling for in-transit packages when customers have urgent
needs.
⚠️
Missing Resolution
trace-005
Input
"The tracking says delivered but I never got the
package."
Persona
new_customer
Quality
! Task
! Satisfaction
Issue Point
User:
The tracking says delivered but I never got the
package.
Assistant:
I can see the package was marked as delivered on
January 12th at 2:34 PM. Could you check around your property, with
neighbors, or building mailroom?
⚠ Agent verified tracking but didn't proactively offer replacement,
refund, or investigation options.
💡
After confirming delivery discrepancy,
proactively present resolution options: 'Would you like a replacement,
refund, or should I open an investigation?'
Quality Improvement Opportunities
!
Refund tool not invoked despite explicit customer requests
2 cases
Evidence
Customer requested full refund for damaged product; agent
only cited policy.
Refund eligibility was confirmed but process_refund tool
wasn't called.
Root Cause
Agent prioritizes information-giving over action-taking when refund keywords are
detected. The intent-to-action mapping for refund requests is incomplete.
In both failed refund cases, the agent correctly identified the customer's intent and
even confirmed refund eligibility, but stopped short of executing the refund. This
suggests a gap in the action-taking pipeline.
💡 Action
Update the refund intent handler to automatically invoke the process_refund tool
when eligibility is confirmed and customer consent is obtained.
!
Fabricated order details without system verification
1 case
Evidence
Agent stated order was 'shipped via UPS on Jan 5'
without querying the order system.
Tracking number provided (1Z999AA1...) was not retrieved
from any tool.
Root Cause
When order lookup fails or times out, the agent falls back to generating
plausible-sounding details instead of admitting the lookup failed.
This occurred with a frustrated customer, suggesting the agent may prioritize appearing
helpful over accuracy when under pressure.
💡 Action
Add a guardrail that prevents shipping/tracking details from being generated without
a successful order_lookup tool response.
!
Escalation paths unclear for billing disputes
1 case
Evidence
Agent identified billing discrepancy but didn't route
to billing team.
Customer was left without clear next steps for
resolution.
Root Cause
The escalation tool exists but trigger conditions for billing issues aren't
well-defined in the agent's policy.
The agent correctly identified the issue but the handoff to human support or billing
team wasn't initiated.
💡 Action
Define explicit escalation triggers for billing discrepancies exceeding $10 or
involving disputed charges.
🎯
Quick Wins
Performance
Performance & Efficiency
342
Avg Output Tokens
2.3
Avg Conversation Turns
3.2s
Avg Latency
Total Cost: $0.47 (avg $0.039 per trace)
Output Tokens
Distribution
100
250
400
550
Key Statistics
Mean
342
P50
298
P95
512
P99
587
Range
156 - 612
Std Dev
98.4
!
Verbose Cases (2)
| Trace | Output Tokens | Persona | Issue |
|---|---|---|---|
| trace-001 | 612 | ||
| trace-004 | 587 |
💡
Recommendation
Verbose responses correlate with frustrated customer and billing dispute scenarios.
Consider adding response length guardrails for these cases while maintaining
clarity.
Conversation Depth
Distribution
1
2
3
4+
Key Statistics
Mean
2.3 turns
P50
2 turns
P95
4 turns
P99
5 turns
Range
1 - 5 turns
Std Dev
0.9
!
Complex Cases (2)
| Trace | Turns | Persona | Issue |
|---|---|---|---|
| trace-001 | 5 | ||
| trace-002 | 4 |
💡
Recommendation
Longer conversations occur with frustrated customers. Consider front-loading
resolution options to reduce back-and-forth while maintaining customer satisfaction.
Latency
Distribution
1s
3s
5s
7s
Key Statistics
Mean
3.2s
P50
2.8s
P95
5.1s
P99
6.2s
Range
1.4s - 6.8s
Std Dev
1.2
!
Slow Cases (2)
| Trace | Latency | Persona | Issue |
|---|---|---|---|
| trace-004 | 6.8s | ||
| trace-001 | 5.9s |
💡
Recommendation
High latency correlates with tool-intensive queries (billing lookup, order status).
Consider caching frequent queries and implementing parallel tool calls for complex
requests.
Persona Performance Gap
Output Tokens
novice
412
expert
+44.6%
285
Latency
novice
3.8s
expert
+58.3%
2.4s
Conversation Depth
novice
2.8
expert
+64.7%
1.7
Interpretation
New customers receive more detailed responses (+45% tokens) with additional
explanations, resulting in longer conversations (+65% turns) and higher latency (+58%).
This is expected behavior as new users require more guidance.
💡
Optimization Opportunities
1. The persona gap is appropriate - new customers benefit from detailed explanations
while power users prefer concise responses.
2. Consider adding 'brief mode' option for returning customers who prefer faster, more concise interactions.
3. Monitor frustrated_customer persona separately as it shows similar verbosity to new_customer but with lower satisfaction.
2. Consider adding 'brief mode' option for returning customers who prefer faster, more concise interactions.
3. Monitor frustrated_customer persona separately as it shows similar verbosity to new_customer but with lower satisfaction.
What to Fix
!
Critical
Must fix before deployment
Refund tool not invoked despite explicit customer
requests
Critical
Problem
When customers explicitly request refunds and eligibility
is confirmed, the agent fails to invoke the process_refund tool, leaving
customers without resolution.
Trace
trace-002
I want a full refund immediately. The product was
damaged.
Root Cause
The intent-to-action mapping for refund requests is
incomplete. The agent stops at eligibility confirmation without proceeding to
execution.
Evidence
✗
Refund eligibility confirmed but process_refund tool not
called
✗
0/2 explicit refund requests successfully processed
💡
Recommended Fix
- Add automatic process_refund invocation when: (1) customer explicitly requests refund, (2) eligibility is confirmed, (3) customer confirms intent.
- Implement confirmation flow: 'I can process your $X refund now. Should I proceed?'
- Add success confirmation: 'Your refund of $X has been processed. You'll receive it in 3-5 business days.'
Expected Impact
Refund completion rate should increase from 0% to
>90%, significantly improving frustrated customer satisfaction
scores.
Hallucinated shipping details without tool verification
Critical
Problem
When order lookup fails or is slow, the agent generates
plausible-sounding shipping details instead of admitting the lookup
failed.
Trace
trace-001
This is ridiculous! My order #12345 hasn't arrived
and it's been 2 weeks!
Root Cause
The fallback behavior for failed order lookups prioritizes
appearing helpful over accuracy, generating fake data to fill the gap.
Evidence
✗
Tracking number 1Z999AA1 provided without any tool call
✗
Shipping date 'Jan 5' stated without order_lookup
verification
💡
Recommended Fix
- Add hard constraint: shipping/tracking details can only be stated if order_lookup returns valid data.
- Implement fallback: 'I'm having trouble accessing your order details. Let me try again or connect you with a specialist.'
- Add retry logic with 3-second timeout before falling back to escalation offer.
Expected Impact
Eliminates hallucination risk for order status
queries, preventing customers from acting on false information.
!
Important
Address in next sprint
Escalation paths unclear for billing disputes
Important
Problem
When billing discrepancies are identified, the agent
doesn't have clear escalation triggers, leaving customers without resolution
paths.
Trace
trace-004
I think there's a billing error on my last
invoice.
Root Cause
Escalation tool exists but trigger conditions for billing
issues aren't well-defined. The agent identifies issues but doesn't know when to
hand off.
Evidence
!
Billing discrepancy identified but no escalation initiated
!
Customer left without clear next steps
💡
Recommended Fix
- Define escalation trigger: billing discrepancies >$10 or disputed charges auto-escalate to billing team.
- Provide ticket number and expected resolution time to customer.
- Add follow-up scheduling: 'You'll receive an email within 24 hours with the resolution.'
Expected Impact
Clearer resolution paths will improve power_user
satisfaction and reduce repeat contacts for billing issues.
Frustrated customer handling lacks empathy-first
approach
Important
Problem
When dealing with upset customers, the agent jumps
directly to solutions without first acknowledging the customer's
frustration.
Trace
trace-001
This is ridiculous! My order #12345 hasn't arrived
and it's been 2 weeks!
Root Cause
The agent's response template prioritizes efficiency over
emotional acknowledgment, missing the empathy step that frustrated customers
need.
Evidence
✗
No empathy statement before problem-solving
✗
Frustrated customer satisfaction score: 2/5
💡
Recommended Fix
- Add empathy-first template: 'I completely understand how frustrating this must be. Let me help resolve this right away.'
- Detect frustration keywords (ridiculous, unacceptable, terrible) and trigger empathy response.
- Follow empathy with immediate action: lookup, escalation offer, or resolution options.
Expected Impact
Empathy-first approach typically improves frustrated
customer satisfaction by 20-30%.
✓
Nice-To-Have
When time permits
Proactive resolution offers for delivery issues
Info
Problem
When delivery issues are detected (tracking shows
delivered but customer didn't receive), the agent verifies status but
doesn't proactively offer resolution options.
Trace
trace-005
The tracking says delivered but I never got the
package.
Root Cause
The agent's flow stops at verification without
automatically presenting next-step options (replacement, refund,
investigation).
Evidence
!
Tracking verified but no resolution options offered
💡
Recommended Fix
- Add proactive prompt after delivery discrepancy: 'Would you like me to: (1) Request a replacement, (2) Process a refund, or (3) Open an investigation with the carrier?'
- Enable one-click resolution for common delivery issues.
- Track which option customers choose most to optimize default suggestion.
Expected Impact
Proactive offers reduce customer effort and typically
resolve issues 40% faster.
Run Config
Evaluation Goal
"Evaluate the agent's ability to handle customer inquiries,
process refunds, and resolve complaints effectively."
Test Design
Total Traces
12
Personas
3
Iterations
1
Execution Date
2025-01-15 14:30:00
Input Generation
Base Inputs
4
Generated
12
Generator
gpt-4o
Strategies
rephrase
edge_case
adversarial
Evaluation
Judge Model
gpt-4o
Enabled Metrics
Task Completion
Hallucination
Relevance
Tool Usage
Satisfaction
Clarity
Persona
Personas
frustrated_customer
An upset customer seeking resolution
- Uses emotional language
- Expects quick solutions
- May express dissatisfaction
new_customer
First-time user unfamiliar with the service
- Asks basic questions
- Needs step-by-step guidance
- May be hesitant
power_user
Experienced customer who knows the system
- Uses technical terms
- Expects efficient responses
- May request advanced features
Tested Inputs
Base Input
"I need help with my order"
Generated Variations
4 inputs
| Persona | Strategy | Input |
|---|---|---|
| frustrated_customer | emotional | "This is ridiculous! My order #12345 hasn't arrived and it's been 2 weeks!" |
| new_customer | basic | "Hi, I just signed up. How do I track my first order?" |
| power_user | technical | "Can you check the fulfillment status via the warehouse API for order #98765?" |
| frustrated_customer | refund | "I want a full refund immediately. The product was damaged." |
Eval Criteria
Overall Success Classification
✓
Clear Success
All core functions working perfectly
Production ready
Production ready
!
Marginal Success
Task completed but quality/efficiency issues
Works but needs improvement
Works but needs improvement
✗
Clear Failure
Core function clearly failed
Cannot deploy, fix required
Cannot deploy, fix required
?
Review Needed
Ambiguous judgment, review required
Human judgment needed
Human judgment needed
Completeness
Task Completion Rate
LLM Judge
"Did the agent complete what was requested?"
Criteria
- PASS: Fully completed
- PARTIAL: Partially completed or alternative provided
- FAIL: Not completed at all
Threshold
✓
>=80% Good
!
60-80% Fair
✗
<60% Poor
Correctness
Hallucination Rate
Rule-based
"Is the answer correct? Is there evidence?"
Method
Extract specific facts from output → Compare with timeline tool output
Threshold
✓
<=5% Good
!
5-15% Fair
✗
>15% Critical
Relevance Rate
LLM Judge
"Is the response on-topic?"
Criteria
- PASS: Directly relevant to question, minimal irrelevant info
- FAIL: Unrelated to question or contains excessive irrelevant info
Threshold
✓
>=90% Good
!
80-90% Fair
✗
<80% Poor
Tool Usage Appropriateness
LLM Judge
"Was the task performed with appropriate process?"
Criteria
- APPROPRIATE: Safe and logical process
- INAPPROPRIATE: Risky or inefficient
Threshold
✓
>=90% Good
!
80-90% Fair
✗
<80% Poor
Quality
User Satisfaction
LLM Judge
"Would a real user be satisfied?"
Criteria
- GOOD: Satisfactory user experience
- FAIR: Room for improvement
- BAD: Unsatisfactory
Threshold
✓
>=70% Good
!
50-70% Fair
✗
<50% Poor
Response Clarity
LLM Judge
"Is the response clear and easy to understand?"
Criteria
- PASS: Clear and structured response
- FAIL: Unclear, contradictory, duplicative, or unstructured
Threshold
✓
>=90% Good
!
80-90% Fair
✗
<80% Poor
Persona Consistency
LLM Judge
"Did the agent respond appropriately for the user type?"
Criteria
- PASS: Matches persona (tone, explanation depth)
- FAIL: Mismatches persona (e.g., jargon for novice)
Threshold
✓
>=85% Good
!
70-85% Fair
✗
<70% Poor
Efficiency & Performance
Output Tokens
Statistical
"Is the response concise? Is the cost reasonable?"
Measures: Mean, P50, P95, Std Dev, Total Cost
Verbose:
Verbose:
Mean + 2×Std exceeded or
>2000 tokens
Conversation Depth
Statistical
"Are there too many unnecessary back-and-forths?"
Measures: Mean, P50, P95, Distribution
Deep:
Deep:
Mean + 2×Std exceeded or >6
turns
Latency
Statistical
"Is the response speed acceptable?"
Measures: Mean, P50, P95, P99
Slow:
Slow:
Mean + 2×Std exceeded or >60s
Persona Gap
Statistical
"Are there efficiency differences by persona?"
Measures: Average latency, tokens, cost per persona
Gap: Absolute difference, percentage ratio
Gap: Absolute difference, percentage ratio