SecondActSociety — RAG Pipeline V2 Architecture

Pipeline Overview

7 nodes, parallel fan-out after context assembly. Wall-clock time equals the slowest single task in the parallel segment, not the sum of all tasks.

Node Inventory

Node	File	Purpose	Model
Intent	`intent_node.py`	Classify user query intent and extract routing metadata	Claude Haiku 4.5
Clarification Gate	`clarification_gate.py`	Evaluate confidence; request clarification if ambiguous	Rule-based
Retrieval	`retrieval_node.py`	Two-phase hybrid search across vector + keyword indexes	OpenAI Embedding
Context	`context_node.py`	Format and assemble retrieved content for downstream nodes	—
Response	`response_node.py`	Generate the primary AI response grounded in context	Claude Sonnet 4.5
Suggestions	`suggestions_node.py`	Produce follow-up questions from graph concepts and stage	Claude Haiku 4.5
Journey	`journey_node.py`	Persist interaction to Journey DB; merge response + suggestions	—

Retrieval Architecture

Two-phase retrieval with authority-based source separation, reciprocal rank fusion, and an automatic fallback when result count drops below threshold.

Configuration

Max RAG Results

Phase 1 contributes up to 8, Phase 2 contributes up to 4.

Vector Weight

0.5

Equal weighting between vector similarity and keyword relevance in RRF fusion.

RRF K Constant

Reciprocal Rank Fusion smoothing constant. Standard value balancing top-rank dominance.

Graph Chunks

12 + 3

12 graph relationship results plus 3 persona-contextualized chunks.

Fallback Threshold

< 3

If combined Phase 1 + Phase 2 results fall below 3, unfiltered broad_match adds up to 4 more.

Concurrent Tasks

6–7

Embedding generation fans out to all retrieval channels simultaneously.

Authority Taxonomy

Authority Level	Phase	Sources	Description
primary	Phase 1	Jay Samit's published works, official SAS content	Highest-trust content. Direct from the author or organization. Always prioritized in response grounding.
supplementary	Phase 2	Curated third-party articles, interviews, case studies	Vetted supporting material that extends primary content with additional context and examples.
illustrative	Phase 2	Industry examples, analogies, supporting narratives	Used to make concepts tangible. Never presented as primary source material.
supporting	Phase 2	General knowledge, background context	Fills gaps when primary and supplementary sources lack coverage. Lowest retrieval priority.

Output Split

Retrieval produces two distinct result sets passed downstream independently. They are never merged before reaching the Context and Suggestions nodes.

# Retrieval output structure
{
  "rag_results": [          # Phase 1 + Phase 2 fused results (up to 12)
    {
      "content": "...",
      "authority": "primary",
      "content_type": "book_chapter",
      "rrf_score": 0.847,
      "source": "phase_1"
    }
  ],
  "graph_results": [        # Neo4j relationships + persona chunks (up to 15)
    {
      "entities": [...],
      "relationships": [...],
      "persona_context": "..."
    }
  ]
}

Intent-Driven Retrieval

The Intent node classifies each query into one of seven intent types. Five of these apply two-phase authority filtering; two get full corpus access for broad exploration.

Intent Mapping

Intent	Authority Filter	Content Type Filter	Strategy
concept_explanation	primary, supplementary	book_chapter, framework	Two-Phase Filtered
actionable_advice	primary, supplementary	exercise, worksheet, case_study	Two-Phase Filtered
personal_story	primary	biography, interview, anecdote	Two-Phase Filtered
book_reference	primary	book_chapter, quote	Two-Phase Filtered
case_study	primary, supplementary, illustrative	case_study, example	Two-Phase Filtered
exploration	all	all	Full Corpus Access
general	all	all	Full Corpus Access

Graph → Suggestions Flow

Graph results feed concept extraction, which combines with the user's journey stage to generate contextual follow-up questions. When graph concepts are unavailable, the system falls back to RAG content types for suggestion generation.

How It Works

Step	Input	Output
1. Concept Extraction	Neo4j graph_results (entities + relationships)	List of related concepts, themes, and entity connections
2. Stage Context	Journey DB — user's current progression stage	Stage-appropriate framing (e.g., early exploration vs. deep application)
3. Generation	Concepts + stage context → Claude Haiku 4.5	3–4 follow-up questions tailored to graph neighborhood and user journey
Fallback	RAG content_type metadata (when no graph concepts)	Content-type-driven suggestions (e.g., "explore related case studies")

Latency Profile

Three areas dominate wall-clock time. The parallel fan-out after Context means total latency is bounded by the slowest branch, not the sum.

Embedding Generation

OpenAI embedding API call occurs on every query before any retrieval can begin. This is a serial bottleneck — nothing downstream starts until the embedding returns.

Mitigation: Connection pool warming reduces cold-start penalty. Embedding caching for repeated/similar queries under evaluation.

LLM Response Generation

Claude Sonnet 4.5 response node is the most computationally expensive call. Runs in parallel with Suggestions but is almost always the slower of the two, making it the critical path.

Mitigation: Streaming via SSE delivers partial responses to the client. Context window optimization reduces token count without sacrificing quality.

Neo4j Graph Queries

Graph traversal queries run concurrently with vector/keyword search. Cold-start latency on Railway can spike when the Neo4j instance hasn't been queried recently.

Mitigation: Keep-alive pings via Railway cron. Query result caching for common entity patterns under consideration.

Recent Issues (Resolved)

Both issues surfaced during the V2 rollout and have been resolved in production.

Issue	Root Cause	Resolution	Status
SSE Stream Parsing	Client-side parser dropped partial chunks when the SSE connection delivered multi-line data events. Buffer boundary handling was incomplete.	Implemented proper chunk accumulation with newline-delimited parsing. Added reconnection logic for dropped streams.	Resolved
Clarification Routing	Clarification Gate responses were being passed through the full pipeline (Retrieval → Response) instead of returning directly to the user, causing unnecessary latency and confused responses.	Clarification Gate now returns a terminal response that short-circuits the pipeline. No downstream nodes execute on clarification paths.	Resolved

Decisions Ahead

Five forward-looking decisions where your input shapes the next iteration. Each includes my current recommendation.

Decision 1

What is the top priority for latency optimization?

Recommendation: Connection pool warming

The embedding call is serial and blocks everything downstream. Warming the connection pool on Railway deploy eliminates the cold-start penalty on the first query after idle periods. This is the highest-leverage single change for perceived responsiveness.

Decision 2

What observability approach should we adopt?

Recommendation: Stay with LangSmith only (current state)

LangSmith gives us full trace visibility across all 7 nodes, including token counts, latencies, and retrieval quality metrics. Adding a second observability layer (Datadog, custom dashboards) is premature until we have enough query volume to justify the operational overhead.

Decision 3

What feature enhancement should come next?

Recommendation: Add conversation memory

Currently each query is stateless — the pipeline treats every message independently. Adding a sliding-window conversation memory (last 3–5 turns) would let the system handle follow-up questions naturally, improving the experience for users in deep exploration sessions. The Journey DB already stores interaction history; this extends it into the retrieval and response nodes.

Decision 4

Should we increase graph enrichment depth?

Recommendation: Keep at 3 persona chunks

Current setting of 3 persona-contextualized chunks per query provides enough graph signal for relevant suggestions without inflating context window size. Increasing to 5+ would add latency on the graph query and increase token usage in the Suggestions node. Revisit if user feedback indicates suggestions feel too narrow.

Decision 5

Should the broad_match fallback behavior change?

Recommendation: Keep broad_match fallback as-is

The fallback triggers only when combined Phase 1 + Phase 2 results drop below 3 — a rare edge case for unusual or highly specific queries. Removing it risks returning empty or near-empty results for legitimate questions. The 4-result cap limits noise. No change needed.

Share your priorities on these five decisions.

Provide Your Input →