SecondAct Knowledge Sources V2 V2 — EXECUTION READY

From: Jonathan Gatlit (Technical Lead)  |  Date: March 9, 2026  |  To: Jay Samit, Skip Blankley
4/4 Decisions Confirmed 1,791 Chunks in Production 9 Sources Staged & Ready 3 Items Awaiting Content

Progress Since V1

All client decisions have been captured. Production state has been verified. Email sources have been staged. Ready for execution.

4 of 4 Decisions Confirmed
1,791
Chunks in Production
4
Source Types Ingested
9
Sources Staged
2
New Processors to Build
3
Blocked on Content
V1 → V2 Context Correction: The v1 briefing stated "341 manuscript chunks, old schema, pipeline never run." Production verification (Railway PostgreSQL direct query, 2026-03-09) revealed 1,791 chunks across 4 source types — manuscript chunks have enriched metadata already. The pipeline was partially executed; only interview and press_kit sources remain at 0 chunks.

Confirmed Decisions

All decisions from the v1 input form have been confirmed. These are now implementation facts.

Decision 1 — Resolved
AI Tuesday Columns: Both Knowledge + Distribution
Digital Jay will reference and discuss AI Tuesday column topics in conversations AND columns will be tracked for content distribution cadence. The column processor preserves series metadata (series, issue_date, topic, author).
Decision 2 — Resolved
Pull Quotes: Individual Chunks with Theme Tags
Each of the 100+ pull quotes becomes its own retrievable chunk. Theme metadata auto-tagged (reinvention, career, mindset). Enables: "Give me a quote about reinvention" → targeted retrieval of the best-matching quote.
Decision 3 — Confirmed (context updated)
Manuscript: Validate Existing Metadata, Then Decide
Client chose "re-ingest with enriched metadata." However, production already has 596 manuscript chunks with enriched content_type metadata (exercise, reflection, concept_introduction, biographical_context). Action: validate existing metadata quality first — if sufficient, skip re-ingestion. If source_file field is missing, apply a lighter metadata update.
Decision 4 — Resolved
Email Intake: Manual Staging (Current Batch Done)
Jonathan manually stages email content. The 9 files from the current email batch have already been extracted and staged in SOURCE/2026Mar09-email-sources/. No automation infrastructure needed at current volume. Respondent note: "These designations should not override current validated context."

Production State

Direct query to Railway PostgreSQL (crossover.proxy.rlwy.net:14004, table second_act_documents).

Knowledge Base Composition

Source TypeChunksStatusMetadata Quality
manuscript596IngestedEnriched — content_type (exercise, reflection, concept_introduction, biographical_context, practical_application)
tweet755IngestedBatch import — 2012-2013 tweets
podcast386Ingested4 interview transcripts, topic-segmented
article54IngestedRecent articles collection
interview0StagedGeezer Q&A + persona supplement — never ingested
press_kit0StagedPress Kit + One Sheet — staged from email, never ingested
TOTAL1,791Target: 2,500+ after ingesting staged sources + new processors

Execution Plan

Sequential execution path. No client input needed — all decisions confirmed.

Decisions
4/4 confirmed
WS-1: Ingest Staged
interview sources
Validate
2,500+ chunks

WS-1: Ingest Staged Interview Sources

Run existing interview processor against Geezer Q&A + persona supplement. These files are staged but have 0 chunks in production.

  • interview processor exists (ADW #23)
  • Source files staged in Interview/SOURCE/
  • Install asyncpg in prod venv
  • Run ingestion, validate with --stats

Est: 1-2 hours  |  +200-400 chunks

WS-2: Build column + quotes Processors

Implement 2 new processors per confirmed decisions. column: paragraph-level chunks with series/issue_date metadata. quotes: one-chunk-per-quote with theme tags.

  • Spec finalized (A-3 in priority spec)
  • Decision confirmed: both knowledge + distribution
  • Decision confirmed: individual chunks with themes
  • Implement column processor
  • Implement quotes processor
  • Register in CLI orchestrator

Est: 3-4 hours  |  Depends on WS-1

WS-3: Ingest Email Sources

Ingest 9 files staged in SOURCE/2026Mar09-email-sources/ using existing + new processors.

  • All 9 files extracted from .eml
  • Organized into 4 subdirectories
  • YAML frontmatter on .md files
  • Ingest press assets (press_kit processor)
  • Ingest columns (column processor — WS-2)
  • Ingest pull quotes (quotes processor — WS-2)
  • Ingest knowledge additions (interview processor)

Est: 1-2 hours  |  +300-500 chunks

Execution Pipeline Flow

Total estimated scope: 5-8 hours across 3 workstreams. Target: expand Digital Jay's knowledge base from 1,791 to 2,500+ chunks covering manuscript, tweets, podcasts, articles, interviews, press materials, columns, and pull quotes.

Knowledge Source Registry

Complete inventory with production-verified ingestion state.

#SourceStatusProcessorChunksNotes
In Production (1,791 chunks)
1Manuscript PDFIngestedmanuscript596Enriched metadata (content_type variants)
2TweetsIngestedtweet7552012-2013 batch import
3Podcasts (4)Ingestedpodcast386Interview transcripts
4Articles (4)Ingestedarticle54Recent articles
Staged — Ready for Ingestion (9 sources, 4 subdirectories)
5Press Q&A (Geezer)Stagedinterview0WS-1
6Persona Supplement (4 files)Stagedinterview0WS-1
7Press Kit PDF (417KB)Stagedpress_kit0WS-3
8One Sheet PDF (252KB)Stagedpress_kit0WS-3
9Pull Quotes DOCX (100+)Stagedquotes0WS-2 → WS-3
10Author Q&A PDF (2.2MB)Stagedinterview0WS-3
11AI Tuesday: PomelliStagedcolumn0WS-2 → WS-3
12Spotlight (Valentine's)Stagedcolumn0WS-2 → WS-3
13Knowledge additions (3 files)Stagedinterview0WS-3
Blocked — Content Not Yet Produced
14Personal Q&A (50-70 Qs)BlockedinterviewWaiting on Jay
15Grief / Life TransitionsBlockedinterviewWaiting on Jay
16Resource PagesBlockedarticleWaiting on Skip

Source Status Distribution (Updated)

Blocked Items

Tracked and ready to activate once content is produced. No engineering needed until then.

Personal Q&A (50-70 Questions)

Owner: Jay  |  Processor: interview

Highest-value knowledge source for persona authenticity. When delivered in any format, we ingest immediately.

Grief & Life Transitions

Owner: Jay  |  Processor: interview

Requires both the source material AND approval of sensitivity guardrails.

Resource Pages

Owner: Skip  |  Processor: article/resource

Self-assessment tool, workbook supplement, recommended tools/quizzes.

ADW Handoff

All decisions confirmed. This briefing authorizes creation of the ADW issue for automated execution.

WorkstreamScopeEffortStatus
WS-1: Ingest staged interview sourcesasyncpg install, run interview processor1-2 hoursReady
WS-2: Build column + quotes processors2 new processors, register in CLI3-4 hoursReady
WS-3: Ingest email sources9 files across 4 subdirectories1-2 hoursReady
Next step: Create ADW issue on jgatlit/SecondActSociety with the 3-workstream spec above. The agent executes all workstreams sequentially, expanding Digital Jay from 1,791 → 2,500+ chunks across 8 source types.

End-to-End Pipeline