Ask-ZuZu G-Drive Ingestion Pipeline

Action Needed

Amy’s Design Decisions

8 decisions gate the pipeline build. Each has a recommended option. Your input shapes how content flows into Ask-ZuZu.

A. G-Drive Folder Location

Recommended: New shared folder with IN/OUT subfolders

Content producers (Rachael, Vince) drop files into IN/. After processing, files move to OUT/. A new dedicated folder keeps Zuzu ingestion isolated from other HHE content.

B. Google Auth Method

Recommended: Service account (unattended automation)

A service account runs 24/7 without human login. Amy shares the folder with the service account email. No OAuth popups, no token expiration.

C. File Naming Convention

Recommended: {course-slug}_{YYYY-MM-DD}_{type}.ext

Examples: cell-talk_2026-02-15_transcript.pdf, hbmg_2026-01-30_protocol.txt. Structured names let the pipeline extract metadata from the filename before the LLM even reads the content.

D. Priority Transcripts for First Batch

Recommended: Most enrolled courses (Cell Talk, HBMG, TNC/HHC)

These three courses represent >70% of all chunks. Any quality improvement is immediately visible to the most members. Cell Talk alone has 7,821 chunks.

E. Recency Decay Function

Recommended: Step function (simplest, most predictable)

Current semester = full weight (1.0), previous semester = 0.8, older = 0.5. No complex math — easy to explain, easy to tune.

F. What Does “Recency Date” Mean?

Recommended: Publication date (not ingestion date)

When Amy says “recent content,” she means when it was recorded or written — not when Jonathan loaded it. Publication date preserves content authority.

G. Do HHE Protocols Get Superseded?

Recommended: Demote old versions (don’t delete)

When a protocol is updated, reduce the old version’s weight from 1.0 to 0.3. Zuzu still has historical context but strongly prefers current guidance. No knowledge is lost.

H. Polling Frequency

Recommended: Every 15 minutes

Balanced: files appear in Zuzu within 15 minutes of upload, but we’re not burning Google API quota. 5 min is excessive; hourly feels sluggish.

Submit Your Decisions →

Current State

What Exists Today

17,251 chunks across 10 categories — all manually loaded. No automated pipeline. Three tier mismatches being resolved by the Tier Refactor.

Knowledge Base

17,251

Total chunks in ChromaDB

Ingestion

0

Automated pipelines

Tier Model

3-Tier

Member / Student / Certified

Current Chunk Distribution by Category

Loading diagram…

pie title ChromaDB Chunk Distribution (Current)
    "tnc-hhc" : 7821
    "hbmg" : 4406
    "cohort" : 2016
    "autoimmune" : 915
    "brain-nervous" : 773
    "10-steps" : 470
    "gut-health" : 403
    "detox" : 252
    "microscopy" : 101
    "hhp" : 94

Category	Chunks	Current Tier	New Tier	Status
tnc-hhc	7,821	certified	student	MISMATCH FIX
hbmg	4,406	guest	member	OK
cohort	2,016	certified	certified	No config
autoimmune	915	level-1	student	MISMATCH FIX
brain-nervous	773	level-2	student	OK
10-steps	470	level-2	student	MISMATCH FIX
gut-health	403	level-1	student	OK
detox	252	level-2	student	OK
microscopy	101	certified	student	Needs review
hhp	94	certified	certified	OK

Pipeline Architecture

G-Drive Ingestion Pipeline

Content producers drop files into a Google Drive folder. The pipeline polls every 15 minutes, processes uploads through format detection, LLM metadata extraction, semantic chunking, and batched ChromaDB writes.

End-to-End Ingestion Flow

Loading diagram…

sequenceDiagram
    participant HHE as HHE Content
Producers
    participant GD as G-Drive
IN/ Folder
    participant POLL as Polling Service
(15-min cron)
    participant PARSE as Format Detection
+ Parsing
    participant LLM as Claude Sonnet
Metadata Extraction
    participant CHUNK as Semantic
Chunker
    participant CHROMA as ChromaDB
Batched Write

    HHE->>GD: Drop PDF/TXT/DOCX
    Note over GD: Named: {slug}_{date}_{type}.ext

    POLL->>GD: List new files (Drive API v3)
    GD-->>POLL: File list + metadata

    POLL->>PARSE: Download + detect format
    Note over PARSE: PDF: pdfminer.six
TXT: direct read
DOCX: python-docx

    PARSE->>LLM: Raw text + filename
    Note over LLM: Extract: course_slug
topic, source_type
author, date

    LLM->>CHUNK: Structured text + metadata
    Note over CHUNK: Section-boundary aware
Header hierarchy preserved

    CHUNK->>CHROMA: Batched add_documents()
    Note over CHROMA: tier + source_type
+ source_weight
+ recency_date + hash

    CHROMA-->>POLL: Chunk count delta
    Note over POLL: Move IN/ to OUT/
Log: file, chunks, summary

Pipeline Components

Format Detection

Multi-Format Parser

PDF via pdfminer.six, TXT direct read, DOCX via python-docx. Format detected from extension and content. Karen’s future book is DOCX-ready.

LLM Extraction

Claude Sonnet Metadata

Extracts course_slug, topic classification, source type, author, and date. Filename convention provides baseline; LLM validates and enriches.

Semantic Chunking

Section-Boundary Aware

Chunks respect section headers and paragraph boundaries — not fixed-size sliding windows. Header hierarchy and cross-references preserved.

Batched Writes

ChromaDB Optimization

ALWAYS batched add_documents(). Critical: per-document writes take ~50s each (SQLite WAL). 40 docs batched = ~100s vs 33 minutes.

Chunk Metadata Schema

Field	Type	Example	Purpose
`tier`	enum	student	Access control (Member/Student/Certified)
`source_type`	enum	transcript	Weight assignment + display
`source_weight`	float	1.0	Retrieval ranking boost
`recency_date`	date	2026-03-01	Recency decay scoring
`course_slug`	string	cell-talk	Category routing
`author`	string	Rachael	Attribution + weight
`content_hash`	MD5	abc123...	Dedup (no duplicate chunks)

Retrieval Upgrade

Weighted Scoring Model

Current retrieval is pure cosine similarity. The new model adds recency and source authority weighting for more relevant results.

Enhanced Retrieval Pipeline

Loading diagram…

sequenceDiagram
    participant USER as User Query
    participant SEARCH as Semantic Search
(ChromaDB)
    participant RANK as Weighted Ranker
    participant FILTER as Tier Filter
(Member/Student/Certified)
    participant RESP as Response Generator

    USER->>SEARCH: Query text
    SEARCH-->>RANK: Top-N candidates
with cosine scores

    Note over RANK: combined = 0.6 * relevance
+ 0.2 * recency
+ 0.2 * source_weight

    RANK->>RANK: Recency step function:
Current semester = 1.0
Previous semester = 0.8
Older = 0.5

    RANK->>FILTER: Re-ranked candidates
    Note over FILTER: User tier >= chunk tier?
Segment boost applied?

    FILTER->>RESP: Filtered, ranked chunks
    Note over RESP: High confidence: 4K-7K tokens
Low confidence: flag + cite sources

Source Weight Hierarchy

From Jan 30 consultation: pre-recorded, fact-checked content gets highest authority.

Source Type	Weight	Rationale
Published book	1.0	Published, fact-checked
Pre-recorded lectures	1.0	Reviewed, edited content
Official protocols	1.0	HHE-approved guidance
Refined academic notes	0.85	Vince’s processed notes
Live call transcripts	0.6	Raw, unedited recordings

Recency Decay: Step Function

1.0

Current Semester

Full weight

0.8

Previous Semester

Slightly reduced

0.5

Older

Still searchable

Reference Architecture

SecondAct-Interview Template

The HHE pipeline is modeled on the proven SecondAct-Interview ingestion system. Same stack, adapted metadata.

Component	SecondAct-Interview	HHE Equivalent
Input format	DOCX (Q&A interviews)	PDF/TXT/DOCX (transcripts, protocols)
Parser	python-docx	pdfminer.six + python-docx + stdlib
LLM extraction	Claude Sonnet structured	Claude Sonnet metadata + topic
Embedding	text-embedding-3-small (1536-D)	Same
Vector store	ChromaDB PersistentClient (dual)	ChromaDB PersistentClient (single)
Batch writes	Single add_documents() — 17x speedup	Same pattern
Dedup	MD5 content hash	Same
Metadata	source, authority, year, content_type	tier, source_type, source_weight, recency_date

Critical Lesson

Never call add_documents() per-item — ChromaDB SQLite WAL checkpoint takes ~50s per call. 40 docs per-item = 33 minutes; batched = ~100 seconds. This is a 17x performance difference.

Deployment

Service Architecture

New Python service deployed alongside existing HHE infrastructure on the VPS.

Deployment Architecture

Loading diagram…

sequenceDiagram
    participant GD as Google Drive
    participant PM2 as PM2 Process Manager
(VPS)
    participant ING as hhe-ingestion
(Python, 15-min cron)
    participant CHROMA as ChromaDB
(Railway)
    participant SUPA as Supabase
(VPS :54322)
    participant PROXY as HHE-Kajabi-proxy-chat
(Railway :3100)

    Note over PM2: Existing infrastructure
    PM2->>ING: Trigger every 15 min
    ING->>GD: Poll IN/ folder
    GD-->>ING: New files

    ING->>ING: Parse + Extract + Chunk

    ING->>CHROMA: Batched writes
(3-tier metadata)
    Note over CHROMA: member/student/certified

    ING->>GD: Move IN/ to OUT/

    Note over PROXY: Existing service (unchanged)
    PROXY->>CHROMA: queryWithAccess()
    PROXY->>SUPA: v_user_tier_calculation

Selected Option

PM2 on VPS (Cron)

Simple, familiar tooling. Co-located with Supabase. Python process managed by PM2 with 15-minute polling interval.

Future Upgrade

G-Drive Webhook

Real-time event-driven ingestion. Requires public webhook endpoint. Consider after pipeline is stable and content volume warrants instant processing.

Execution Timeline

Build Phases & Dependencies

Pipeline build depends on Amy’s decisions and the Tier Refactor completion. Estimated timeline after all blockers clear.

Execution Timeline

Loading diagram…

gantt
    title HHE Ingestion Pipeline — Execution Timeline
    dateFormat YYYY-MM-DD
    axisFormat %b %d

    section Blockers
    Amy decisions (8 items)           :crit, amy, 2026-03-01, 5d
    Tier Refactor completion          :crit, tier, 2026-03-01, 7d
    G-Drive folder + credentials      :crit, gdrive, after amy, 3d

    section Pipeline Build
    Python service scaffold           :scaffold, after tier, 2d
    G-Drive watcher + polling         :watcher, after gdrive, 3d
    Format parsers (PDF/TXT/DOCX)     :parsers, after scaffold, 3d
    LLM metadata extraction           :llm, after parsers, 2d
    Semantic chunker                  :chunker, after parsers, 2d
    ChromaDB batched writer           :writer, after chunker, 2d

    section Integration
    End-to-end pipeline test          :test, after writer, 2d
    First batch ingestion             :batch, after test, 2d
    Retrieval scoring upgrade         :retrieval, after batch, 3d
    Verification + QA                 :qa, after retrieval, 2d

    section Deferred
    Karen book ingestion              :milestone, after qa, 0d
    TypeSense migration planning      :milestone, after qa, 0d

Phase Transition Gates

•G1: Pipeline Operational — First batch processes successfully, verified chunk count
•G2: HHE Product List — Amy provides categorized product list (enables Protocol Database)
•G3: Tier Refactor Complete — 3-tier model live in all systems (prerequisite for new content)
•G4: Protocol DB MVP — Public database live with 20+ protocols (enables Practitioner Platform)