Digital Jay Pipeline Architecture

Architecture

System Architecture

Digital Jay operates as a 4-layer stack spanning 2 repositories, 2 databases, 3 LLM models, a knowledge graph, and an embedding service. Every chat request traverses all four layers.

Layer 1: Frontend Widget

Vanilla JS bundle (secondact-home.js) + on-demand chat (secondact-chat.js). EventSource SSE for streaming. Hosted on Kajabi via proxy.

Layer 2: Proxy

Express/Node.js 20+ (SecondAct-Kajabi repo). Builds user context, fetches conversation history, forwards SSE, persists chat messages. Prisma ORM.

Layer 3: V2 Backend

FastAPI + LangGraph + Python 3.11 (SecondActSociety repo). 7-node state machine with hybrid RAG, intent classification, and parallel response generation.

Layer 4: External Services

PostgreSQL + pgvector (RAG), Neo4j (knowledge graph), Claude Sonnet 4.5 (response), Claude Haiku 4.5 (suggestions), OpenAI Embeddings (text-embedding-3-small).

Two Separate Databases — V2 DB (crossover:14004) holds the RAG knowledge base. Kajabi DB (shinkansen:17053) holds leads, surveys, chat sessions, and messages. They are in different Railway projects. Never confuse them.

End-to-End Request Flow

Infrastructure

Component	Technology	Location
V2 Backend	FastAPI + LangGraph + Python 3.11	`society-v2-production.up.railway.app`
Proxy	Express 4.18 + Node.js 20+	`secondact-proxy-production.up.railway.app`
V2 Database	PostgreSQL + pgvector (HNSW)	`crossover.proxy.rlwy.net:14004`
Proxy Database	PostgreSQL + Prisma ORM	`shinkansen.proxy.rlwy.net:17053`
Knowledge Graph	Neo4j (async driver)	Aura
Embeddings	OpenAI `text-embedding-3-small` (1536-D)	OpenAI API
Response LLM	Claude Sonnet 4.5	Anthropic API
Suggestions LLM	Claude Haiku 4.5	Anthropic API

Pipeline

LangGraph Execution Pipeline

Every request flows through a 7-node state machine with conditional routing. The intent node determines whether the user needs clarification or retrieval. Response and suggestions run in parallel after context assembly.

LangGraph State Machine Topology

Node-by-Node Execution

Node 1: Intent Classification Bottleneck ~1.5s

Model: Claude Sonnet 4.5, temp=0.3 • File: agent/nodes/intent.py
Extracts last message, builds 3-turn history summary (100 chars each), preserves acronyms via regex (\b[A-Z]{2,}\b), detects longevity keywords. LLM classifies into 7 intent types with confidence level and search keywords. Sequential bottleneck — blocks all downstream nodes.

Node 2: Routing Gate <1ms

File: agent/routing.py
Requires ALL THREE: ambiguity_detected=true AND confidence="low" AND no injected_context. The proxy's user context (survey/lead data) effectively prevents most clarification triggers. Clarification is terminal — zero streaming tokens.

Node 3: Retrieval — Two-Phase Hybrid RAG ~2.5s

File: agent/nodes/retrieval.py
Launches 6+ parallel tasks: primary vector+keyword (24 candidates each), supplemental vector+keyword (12 each), Neo4j graph search (12, 2s timeout), persona search (3), user profile fetch. RRF fusion produces 12 final results (8 primary + 4 supplemental). Fallback broadening if <3 results. See Retrieval Deep Dive.

Node 4: Context Assembly <100ms

File: agent/nodes/context.py
Merges injected_context (proxy user data) + user_profile + RAG results + graph concepts into structured system prompt. Sections: <user_context>, Book Content, Supporting Context, Related Principles, Voice & Style, Related Concepts. Content-type tags: FRAMEWORK, EXERCISE, EXAMPLE, STORY, CONCEPT, etc.

Node 5: Response Generation Parallel ~3-5s

Model: Claude Sonnet 4.5, temp=0.3, max_tokens=4096 • File: agent/nodes/response.py
Gets journey persona from model_id, builds system prompt with deterministic voice anchors (MD5 hash of query), injects assembled_context. Streams AIMessageChunk tokens. Runs in parallel with suggestions.

Node 6: Suggestions Generation Parallel ~1-2s

Model: Claude Haiku 4.5, temp=0.7, max_tokens=512 • File: agent/nodes/suggestions.py
Extracts concepts from Neo4j graph, maps model_id to journey stage. Tries Haiku-enriched suggestions with manuscript snippets, falls back to journey-specific templates. Generates 3-4 {label, question} pairs. Conditionally appends book CTA after 2+ messages. Hidden behind response streaming — adds 0s to user-perceived latency.

Node 7: Journey Update Fire & Forget

File: agent/nodes/journey.py
UPSERT to user_profiles table mapping model_id to journey name. Non-blocking — errors logged but never raised. Returns empty dict (no state updates).

Retrieval

Retrieval Deep Dive

The retrieval system uses two-phase authority-weighted hybrid search with Reciprocal Rank Fusion. Manuscript content (primary authority) gets 8 of 12 final slots. Supplemental sources (podcasts, articles, tweets, columns) get 4 slots. Intent-aware filters tune what each search retrieves.

Two-Phase Hybrid RAG Pipeline

Intent-Aware Filtering

Intent	Authority	Content Type	Effect
`action`	primary	exercise, framework_walkthrough	Prescriptive book exercises only
`search`	primary, supporting	—	Broad access, minimal filtering
`exploration`	primary, supporting, illustrative	concept_introduction, narrative	Descriptive content across sources
`comparison`	primary, supporting	—	Cross-source evidence
`reflection`	primary, supporting	reflection, narrative	Contemplative content
`troubleshooting`	— (no filters, full corpus access)		Maximum coverage
`clarification`	— (no filters, full corpus access)		Maximum coverage

RRF Fusion Formula

score = vector_weight / (k + rank_v) + (1 - vector_weight) / (k + rank_k)
Default: vector_weight=0.5 (equal), k=60 (standard). Confidence boost: 0.7 + 0.3 * confidence_score. Dedup by first 50 chars of content.

Knowledge Base Composition

Source Distribution (~2,095 Chunks)

Source	Count	Authority	Notes
Tweets	755	illustrative	Twitter/X content
Podcasts	386	supplementary	Podcast transcripts
Manuscript (baseline)	341	primary	IDs 1-341, no source_type, has chapter
Columns	304	supporting	Substack (AI Tuesday, Spotlight, general)
Manuscript (enhanced)	255	primary	Rich metadata: content_type, frameworks
Articles	54	supporting	Blog articles

Gap: ~400+ chunks missing from interviews (200-400), press kit (50-100), and pull quotes (100+). Target is 2,500+. Ingestion pipeline and processors are ready in scripts/processors/.

Search Components

Vector Search (pgvector)

HNSW index (m=16, ef=64). Cosine distance via <=> operator. OpenAI text-embedding-3-small (1536-D). Filtered by authority/content_type/source_type via JSONB.

Keyword Search (FTS)

to_tsvector + plainto_tsquery + ts_rank. Always runs ILIKE fallback for 2+ char acronyms (e.g., BOSS, PRODS). Acronym rank: 0.8.

Graph Search (Neo4j)

3-hop variable-length paths. Keyword match on node names. Returns concepts + relationships sorted by depth. 2.0s timeout with graceful degradation.

Persona Search (Neo4j)

Always retrieves 3 VoicePattern/PersonalityTrait nodes regardless of query. Tagged authority=personality for context routing. 2.0s timeout.

Data Flow

SSE Streaming Flow

The V2 backend uses LangGraph's dual stream mode (["messages", "updates"]) to capture both response tokens and metadata events in a single pass. The proxy selectively forwards tokens to the widget while capturing intent and suggestions for persistence.

SSE Event Lifecycle: V2 → Proxy → Widget

SSE Event Types

V2 Emits	Proxy Action	Widget Receives	Data Shape
`chat.completion.chunk`	Forward content	`event:message`	`{type:"content", content, fullText}`
`chat.completion.intent`	Capture only	Nothing	`{primary_intent, confidence, keywords}`
`chat.completion.suggestions`	Capture only	Nothing	`[{label, question, url?, type?}]`
`[DONE]`	Build + send done	`event:done`	`{type:"complete", fullText, suggestions}`

Alternative Paths

Greeting Flow

__greeting__ sentinel routes to /api/chat/greeting. Haiku + 3 manuscript vectors, max_tokens=150, ~1-2s. Generates personalized welcome + conversation starters. Not persisted.

Clarification Path

Terminal. Requires ambiguity + low confidence + no injected context. Generates 2-3 clarifying questions. Zero streaming tokens. User must respond to continue.

Anonymous Path

No userEmail = no context, no history, no persistence, no greeting. Generic system prompt, single-turn conversation, static welcome message.

Performance

Latency Analysis

The critical path from user input to first token is ~5.8s, dominated by three sequential stages: intent classification (1.5s), retrieval (2.5s), and response TTFB (1.5s). The proxy layer adds only 50-100ms.

End-to-End Latency Waterfall

Latency Distribution by Component

Top Latency Contributors

Rank	Component	Typical	% of Total	Notes
1	Response LLM (Sonnet)	3-5s	35-45%	Streaming mitigates perceived wait
2	Retrieval (Hybrid RAG)	2-3s	25-30%	6+ parallel tasks, Neo4j dominates
3	Intent LLM (Sonnet)	1-2s	15-20%	Sequential bottleneck
4	Neo4j graph search	0.5-2s	5-15%	High variance, 2s timeout cap
5	OpenAI embedding	100-300ms	2-3%	Single API call per request

Key Insight: Intent classification is the #1 optimization target because it is both slow (1.5s) AND sequential (blocks everything downstream). Response LLM is slower overall (3-5s) but streaming mitigates perceived wait — the user sees tokens arrive after ~5.8s, not after ~9.5s.

Critical Path

TTFB Critical Path: Intent LLM (1.5s) → Retrieval (2.5s) → Context (<0.1s) → Response TTFB (1.5s) = ~5.6s minimum TTFB

Hidden Parallel: Suggestions (Haiku, 1-2s) runs parallel to Response — adds 0s. Journey update is fire-and-forget — adds 0s. Within retrieval, 6+ tasks run concurrently, bounded by Neo4j (0.5-2s).

Infrastructure

Data & Connection Architecture

Connection Pool Management

Connection	Type	Init / Warmup	Keepalive
PostgreSQL (V2)	asyncpg.Pool singleton	`SELECT 1` warmup	120s ping
Neo4j	AsyncGraphDatabase.driver singleton	`verify_connectivity()`	120s ping
OpenAI Embeddings	OpenAIEmbeddings singleton	Warmup embed	Per-request
Anthropic (Response)	ChatAnthropic per-graph	None	Per-request
Anthropic (Suggestions)	AsyncAnthropic singleton	None	Per-request
Prisma (Kajabi)	PrismaClient + pg adapter	Auto	Managed

Data Capture Points

Trigger	Endpoint	Database	Table
Email gate (any CTA)	`POST /api/v1/leads/register`	Kajabi	`lead_registrations`
Survey completion	`POST /api/survey/submit`	Kajabi	`survey_submissions`
Chat stream ends	`persistStreamingChat()`	Kajabi	`chat_sessions` + `chat_messages`
Page / CTA events	`POST /api/activity/track`	Kajabi	`activity_events`
Journey update	`journey_node()`	V2	`user_profiles`

Document Schema

second_act_documents (V2 Database)

Column	Type	Purpose
`id`	UUID	Primary key
`content`	TEXT	Chunk text
`embedding`	vector(1536)	OpenAI text-embedding-3-small
`chapter`	TEXT	Book chapter reference
`page_number`	INTEGER	Page reference for citations
`metadata`	JSONB	source_type, content_type, authority, section, frameworks, teaching_context, confidence_score

Indexes: HNSW vector (m=16, ef=64), GIN full-text, GIN metadata, B-tree chapter, B-tree page_number.

Roadmap

Optimization Roadmap

Nine optimization proposals (P1-P9) across four phases, targeting TTFB reduction from ~5.8s to ~3.5s and retrieval improvement from ~2.5s to ~1.5s.

4-Phase Optimization Timeline

Optimization Proposals

P1: Intent Classification Caching

High Priority — -1.5s

Current: Every request makes a full Sonnet LLM call for intent (~1.5s).
Proposed: Cache intent results for semantically similar queries. Hash normalized query, check Redis/in-memory cache, TTL 5 minutes.
Expected: 1-1.5s savings for cached hits (est. 30-40% of requests). Risk: Low.

P2: Intent Model Downgrade to Haiku

High Priority — -1.2s

Current: Sonnet 4.5 for intent classification (~1.5s).
Proposed: Haiku 4.5 (~300-500ms). Intent is a structured JSON extraction task — doesn't need Sonnet's reasoning.
Validation: Shadow mode for 1 week — log both intents, compare accuracy. Switch if Haiku ≥ 90% agreement. Risk: Medium.

P3: Speculative Retrieval

High Priority — -1.5s

Current: Intent must complete before retrieval starts (sequential).
Proposed: Start retrieval with regex-extracted keywords while intent runs. If intent provides better keywords, merge/re-rank.
Expected: 1-2s overlap. Risk: Medium — needs fallback logic for suboptimal results.

P4-P6: Medium Priority

Medium Priority — 500ms-1s each

P4: Neo4j connection optimization — pooling, reduce 3-hop to 2-hop, cache concepts. -200-800ms.
P5: Embedding caching — LRU with 1000 entries, normalized keys. -200ms per cached hit.
P6: Retrieval phase consolidation — single query with authority weighting. -100-300ms.

P7-P9: UX & Quality

Lower Priority — UX improvements

P7: Forward intent to widget for adaptive UI ("Searching book content...", "Finding exercises...").
P8: Pre-fetch greeting during page load (before user clicks). Eliminates 1-2s greeting latency.
P9: Knowledge base expansion — interviews (200-400), press kit (50-100), pull quotes (100+). Target 2,500+ chunks.

Deferred Items

Item	Activation Gate	Target
Real-time latency dashboard	After P1+P2 implemented	LangSmith integration or custom dashboard
A/B testing framework	After D1 (intent model) validated	Route traffic % through variant pipelines
Multi-session memory	After TTFB < 4s	Cross-session context retrieval
Agentic tool use	Phase 4 roadmap	Schedule actions, send resources, create plans

Reference

Configuration Reference

V2 Backend (`agent/config.py`)

Parameter	Value	Category
`MODEL`	claude-sonnet-4-5	LLM
`TEMPERATURE`	0.3	LLM
`MAX_OUTPUT_TOKENS`	4096	LLM
`MAX_RAG_RESULTS`	12	Retrieval
`VECTOR_WEIGHT`	0.5	Retrieval
`RRF_K`	60	Retrieval
`MAX_GRAPH_RESULTS`	12	Retrieval
`PERSONA_RETRIEVAL_COUNT`	3	Retrieval
`FALLBACK_THRESHOLD`	3	Retrieval
`KEEPALIVE_INTERVAL_SECONDS`	120	Infra
`MAX_SUGGESTIONS`	4	Suggestions
`SUGGESTION_MODEL`	claude-haiku-4-5-20251001	Suggestions
`SUGGESTION_MAX_TOKENS`	512	Suggestions
`SUGGESTION_TEMPERATURE`	0.7	Suggestions

Proxy (`src/config/environment.js`)

Parameter	Value	Category
`STREAM_TIMEOUT`	60000ms	Streaming
`STREAM_KEEPALIVE`	15000ms	Streaming
`RATE_LIMIT_WINDOW`	15 min	Security
`RATE_LIMIT_MAX`	100	Security
`STREAM_LIMIT_MAX`	10	Security
`SESSION_BRIDGE_TTL`	2592000s (30d)	Session

Key Files

V2 Backend (SecondActSociety-v2/)

File	Purpose
`routes.py`	HTTP endpoints, SSE streaming, greeting
`agent/graph_builder.py`	LangGraph topology (7 nodes)
`agent/state.py`	SocietyState TypedDict
`agent/routing.py`	should_clarify() gate
`agent/config.py`	All config parameters
`agent/nodes/intent.py`	Intent classification (Sonnet)
`agent/nodes/retrieval.py`	Two-phase hybrid RAG
`agent/nodes/context.py`	Context assembly
`agent/nodes/response.py`	Response generation (Sonnet)
`agent/nodes/suggestions.py`	Suggestions (Haiku)
`shared/retrieval/vector.py`	pgvector search
`shared/retrieval/keyword.py`	PostgreSQL FTS
`shared/retrieval/fusion.py`	RRF fusion
`shared/retrieval/graph.py`	Neo4j graph search

Proxy (SecondAct-Kajabi/)

File	Purpose
`src/routes/chat-stream.js`	Main streaming handler
`src/services/chat-context-builder.js`	User context assembly
`src/services/external-chat-service.js`	History + persistence
`src/services/conversation-starters.js`	Greeting starters
`src/services/suggestion-generator.js`	Fallback suggestions
`src/services/session-bridge.js`	Redis cross-system context

System Architecture

Layer 1: Frontend Widget

Layer 2: Proxy

Layer 3: V2 Backend

Layer 4: External Services

Infrastructure

LangGraph Execution Pipeline

Node-by-Node Execution

Node 1: Intent Classification Bottleneck ~1.5s

Node 2: Routing Gate <1ms

Node 3: Retrieval — Two-Phase Hybrid RAG ~2.5s

Node 4: Context Assembly <100ms

Node 5: Response Generation Parallel ~3-5s

Node 6: Suggestions Generation Parallel ~1-2s

Node 7: Journey Update Fire & Forget

Retrieval Deep Dive

Intent-Aware Filtering

RRF Fusion Formula

Knowledge Base Composition

Search Components

Vector Search (pgvector)

Keyword Search (FTS)

Graph Search (Neo4j)

Persona Search (Neo4j)

SSE Streaming Flow

SSE Event Types

Alternative Paths

Greeting Flow

Clarification Path

Anonymous Path

Latency Analysis

Top Latency Contributors

Critical Path

Data & Connection Architecture

Connection Pool Management

Data Capture Points

Document Schema

Optimization Roadmap

Optimization Proposals

P1: Intent Classification Caching

P2: Intent Model Downgrade to Haiku

P3: Speculative Retrieval

P4-P6: Medium Priority

P7-P9: UX & Quality

Deferred Items

Configuration Reference

V2 Backend (agent/config.py)

Proxy (src/config/environment.js)

Key Files

V2 Backend (`agent/config.py`)

Proxy (`src/config/environment.js`)