Mo Agent v1.2 — Evaluation Briefing

MOJOhealth.org Chatbot • Baseline Assessment • 2026-02-13 to 2026-03-12 • 42 Sessions / 139 Exchanges

Health Scorecard

B
2.29 / 3.0
Overall
B
2.10
Behavioral Compliance
C
1.77
Retrieval Quality
B
2.27
Conversation Quality
B
2.31
User Experience
A
2.97
Operational Health

Top Failure Clusters

General Knowledge Fallback — 50 instances (36% of exchanges)

Mo generates plausible medical content from LLM training data instead of constraining to MOJObase. The "I don't know" protocol fires for clearly out-of-scope questions but fails when the topic is health-adjacent and the model can generate a plausible answer.

Fallback
50
Boundary
23
Dead Ends
11
Fabrication
10

Strategic Insight: KB Depth vs. Prompt Constraints

76% of fallback instances occur on topics where MOJObase has thin or no coverage. Enriching the knowledge base addresses the root cause; prompt constraints only catch what slips through.

Failure PatternCountBetter Fix
General-knowledge-fallback50KB depth (38) Prompt (12)
Boundary violations (critical)12Prompt only
Dead-end loops11KB (4) Prompt (7)
Knowledge fabrication10KB depth

Behavioral Compliance Distribution

27%26%15%9%7%6%4%4%2%User Query Topics (142 queries)Biomarkers & Testing (27%)Uncategorized Health (26%)Treatment Context (15%)General MOJO (9%)Lifestyle & Diet (7%)Supplements (6%)Contact & Referral (4%)Side Effects (4%)Follow-up Clinical (2%)

User Query Topic Distribution

40%30%22%9%Current BC Score Distribution (139 exchanges)Score 3 - StrongScore 2 - AdequateScore 1 - WeakScore 0 - Failure

Priority Actions

1 MOJObase Content Audit + Gap Fill

KB Upsert FIX

Expand MOJObase coverage for top 3 query topics: biomarkers/testing (27%), treatment context (15%), lifestyle/diet (7%). Addresses 38 of 50 fallback instances.

TopicQuery ShareAction
Biomarkers & Testing27%Add cancer-type-specific biomarker guides, testing cadence framing, Signatera/Guardant explanations
Treatment Context15%Add "what MOJO can/cannot help with" content, redirect framing to healthcare team
Lifestyle & Diet7%Add MOJO-authored dietary guidance tied to health defense systems

Deploy via: n8n dashboard /io/mo-upsert • Effort: 2-3 hours

2 Hard Behavioral Boundaries

System Prompt FIX

Add Rule #5 to system prompt: NEVER list treatment options, provide dosages, quote survival statistics, or interpret test results clinically. Addresses 12 score-0 BC violations.

Deploy via: Flowise canvas • Effort: 15 min

3 Oscar Sierra + Referral Content

KB Upsert FIX

Upsert contact info, service description, and booking process for Oscar Sierra / Sierra Botanica and Heal Smarter Workshops. Eliminates 4-exchange dead-end loops.

Deploy via: n8n dashboard /io/mo-upsert • Effort: 30 min

4 Escalation Path for Repeated Questions

System Prompt FIX

Add Rule #6: after 2 unanswered repeats, offer website search or contact page instead of identical non-answer. Addresses 7 remaining dead-end loops.

Deploy via: Flowise canvas • Effort: 10 min

5 Supplement Content Re-Framing

KB Re-Upsert FIX

Audit and re-author existing supplement documents with built-in "MOJO's resources discuss..." framing and disclaimers. Agent reproduces framing from retrieval.

Deploy via: n8n dashboard • Effort: 1 hour

#ItemTypeEffortDeploy Via
1MOJObase content audit + gap fillKB2-3 hrsn8n /io/mo-upsert
2Hard behavioral boundariesPrompt15 minFlowise canvas
3Oscar Sierra + referral contentKB30 minn8n /io/mo-upsert
4Escalation pathPrompt10 minFlowise canvas
5Supplement content re-frameKB1 hrn8n dashboard

Response Flow — Enriched KB Pattern

System Prompt v1.2MOJObase (enriched)Mo AgentUserSystem Prompt v1.2MOJObase (enriched)Mo AgentUseralt[KB returns pre-framed content][KB returns nothing]"What supplements help during chemo?"Vector search"MOJO resources discuss: [supplements]...Discuss with your healthcare team..."Reproduces framing from retrieved content + Strategy Builder linkRuleFire "I Don't Know" protocolGentle refusal + offer MOJO resources

Implementation Architecture

RQ ≥ 2.0Fallback < 15%RQ < 2.0Mo Eval Baseline2026-03-12P1: KB Gap FillTop 3 Query TopicsP2: Hard BoundariesSystem PromptP3: Oscar Sierra +Referral ContentP4: Escalation PathSystem PromptP5: SupplementContent Re-Framen8n Upsert PipelineFlowise Template JSONMOJObaseSupabase VectorsFlowise Agent CanvasRe-Eval CyclePASS: MonitorConsider Temp1.0 → 0.7

Monitoring Targets

MetricCurrentTargetMeasured After
General-knowledge-fallback rate36%<15%P1 + P2 deployed
Retrieval Quality avg1.77≥ 2.0P1 + P5 deployed
BC score-0 exchanges120P2 deployed
Dead-end loop instances11≤ 2P3 + P4 deployed
Knowledge fabrication10≤ 3P1 deployed
Multilingual handlingFunctionalMaintainOngoing

Strengths

  • Operational Health (A / 2.97) — Zero crashes, no truncation, consistent uptime across 42 sessions
  • Empathy delivery — Strong empathetic openings in distress scenarios
  • "I don't know" for out-of-scope — Correctly refuses flu, dentist, WiFi product, pricing questions
  • Resource linking — Correct URLs when MOJO resources are cited
  • Multilingual — Korean exchanges handled competently (Session 41)
  • Oscar Sierra / Workshop referrals — Appropriate when invoked