SecondAct Knowledge Sources V2

The Situation Context

Digital Jay's knowledge base has a fully-built ingestion pipeline that was never activated. Meanwhile, new content has arrived and is sitting unprocessed.

1,791

Chunks in Production

Staged Sources

Staged (Ready)

New Processors Needed

Blocked on Content

Updated (2026-03-09): Production vector store confirmed: 1,791 chunks across 4 source types (596 manuscript, 755 tweet, 386 podcast, 54 article). Manuscript chunks have enriched metadata (content_type: exercise, reflection, concept_introduction, biographical_context). Interview and press_kit sources remain at 0 chunks despite being staged. 9 new sources extracted from email and staged in SOURCE/2026Mar09-email-sources/.

Ready to Execute No Blockers

These workstreams can proceed immediately without waiting on anyone.

WS-A: Activate Pipeline

Run the existing ingest_sources.py --all against production. 5 source types already staged: manuscript, articles (4), podcasts (4), press Q&A, persona supplement.

✓ 6 processors built & merged
✓ CLI orchestrator with --all flag
✓ JSONB metadata schema defined
Install asyncpg in prod venv
Run ingestion, validate with --stats

WS-B: Stage Email Content COMPLETE

9 files extracted from 11 .eml files into SOURCE/2026Mar09-email-sources/ with 4 subdirectories (press-assets, author-qa, columns, knowledge-additions).

✓ Parse .eml MIME attachments
✓ Extract email body text to .md (with YAML frontmatter)
✓ Organize into SOURCE/ directories

WS-C: New Processors

Implement column processor (AI Tuesday, Spotlight editorials) and quotes processor (100+ pull quotes from DOCX).

column: paragraph-level, series metadata
quotes: one-chunk-per-quote, themes
Register in CLI orchestrator

WS-D: Status Tracking

Add pipeline lifecycle tracking to vault. Extends GitHub's binary open/closed with: merged → deployed → executed.

execution_status frontmatter
Knowledge source registry file

Pipeline Architecture How It Works

How content flows from source material to Digital Jay's retrievable knowledge base.

Ingestion Pipeline Flow

Email Content Staging Process

Sources Staged
5 ready

→

Run Pipeline
WS-A

→

Stage Emails
WS-B

→

New Processors
WS-C

→

Validate All
--stats

Knowledge Source Registry 19 Sources Tracked

Complete inventory of all knowledge sources — what's ready, what needs extraction, what's blocked on content production.

#	Source	Status	Processor	Notes
Ingested — In Production (1,791 chunks)
1	Manuscript PDF	Ingested	manuscript	596 chunks — enriched metadata (exercise, reflection, concept_introduction, etc.)
2	Tweets	Ingested	tweet	755 chunks — batch import
3	Podcasts (4)	Ingested	podcast	386 chunks — interview transcripts
4	Articles (4)	Ingested	article	54 chunks — recent articles
Staged — Ready for Ingestion (9 sources)
5	Press Q&A (Geezer)	Staged	interview	0 chunks — staged, never ingested
6	Persona Supplement	Staged	interview	0 chunks — staged, never ingested
7	Press Kit PDF	Staged	press_kit	Extracted from .eml (417KB)
8	One Sheet PDF	Staged	press_kit	Extracted from .eml (252KB)
9	100+ Pull Quotes DOCX	Staged	New: quotes	Extracted from .eml (26KB)
10	Author Q&A PDF (Skip)	Staged	interview	Extracted from .eml (2.2MB)
11	AI Tuesday: Pomelli	Staged	New: column	Extracted, YAML frontmatter added
12	Spotlight (Valentine's)	Staged	New: column	Extracted, YAML frontmatter added
13	Knowledge additions (3)	Staged	interview	Random knowledge, Zoom feedback, Skip's call notes
Blocked — Content Not Yet Produced
13	Personal Q&A (50-70 Qs)	Blocked	interview	Waiting on Jay
14	Grief / Life Transitions	Blocked	interview	Waiting on Jay
15	Resource Pages	Blocked	article	Waiting on Skip
16	Events Calendar	Blocked	TBD	No calendar API

Source Status Distribution

Your Input Needed 4 Decisions

These decisions shape how we process the new content. Your responses unblock the engineering work.

1. AI Tuesday Columns — Knowledge or Content?

✓ Recommended: Both — ingest for Digital Jay's knowledge AND track for content distribution

Should Digital Jay be able to reference and discuss AI Tuesday column topics in conversations? Or are these strictly for newsletter distribution? We recommend ingesting them as knowledge so Jay can say things like "In a recent column, I discussed how Pomelli uses AI for..."

2. Pull Quotes (100+) — Individual or Grouped?

✓ Recommended: Individual chunks — one quote per retrievable unit, with theme metadata

The pull quotes DOCX contains 100+ standalone quotes. We can ingest each as its own chunk (best for targeted retrieval — "give me a quote about reinvention") or group them by theme (fewer chunks but less granular search). If the DOCX is organized by chapter or theme, we'll preserve that in metadata tags.

3. Manuscript Re-ingestion — Replace or Keep Alongside?

✓ Updated: Production has 596 manuscript chunks with enriched metadata already

Production already has 596 manuscript chunks with enriched content_type metadata (exercise, reflection, concept_introduction, biographical_context). The v0 assumption was incorrect. We recommend validating the existing schema quality before deciding on re-ingestion. If metadata is sufficient, no re-ingestion needed.

4. Future Email Content — Auto-intake or Manual?

✓ Recommended: Manual for now — revisit automation at 20+ sources/month

When new content arrives via email, should it automatically flow into the staging pipeline? Current volume (2-3 items/month) doesn't justify automation. Jonathan manually stages content. We'll revisit when volume increases or a regular cadence is established (e.g., weekly AI Tuesday column).

Respond via the input form: Complete decisions here — your responses will be captured automatically and trigger the next phase of work.

Blocked Items Waiting on Content

These are tracked and ready to activate once content is produced. No engineering needed until then.

Personal Q&A (50-70 Questions)

Owner: Jay

The highest-value knowledge source for persona authenticity. Jay answers personal questions about life philosophy, decision-making, values. When delivered in any format (DOCX, email, voice), we ingest immediately.

Grief & Life Transitions

Owner: Jay

Sensitivity-aware content about loss, identity shifts, life transitions. Requires both the source material AND approval of the sensitivity guardrails we've designed for Digital Jay's handling.

Resource Pages

Owner: Skip

Self-assessment tool, workbook supplement, recommended tools/quizzes. Once drafted, content gets ingested into knowledge base AND deployed as Kajabi pages with live URLs for Digital Jay to recommend.

Dependency Chain — What Unblocks What

What's Next ADW Handoff

This briefing feeds into an ADW (Agent DevOps Workflow) issue that executes the engineering work.

Workstream	Scope	Depends On
WS-A: Execute existing pipeline	Activate venv, run --all, validate	Nothing (ready)
WS-B: Stage email content	Extract .eml attachments & body text	Nothing (ready)
WS-C: New processors	Implement column + quotes processors	WS-A (need working pattern)
WS-D: Status tracking	Vault frontmatter + source registry	Nothing (ready)

After your input: We create the ADW issue, the agent executes the remaining workstreams, and Digital Jay's knowledge base expands from 1,791 chunks (4 source types) to a comprehensive multi-source knowledge system adding interviews, press materials, columns, and pull quotes — targeting 2,500+ total chunks.