Admin — how the machine runs

Technical reference for the system itself. Data source: _pipeline\admin-registry.yaml — add entries there. No secrets live in this repo; entries record where access lives, never values.

Architecture

Solid = live; dashed = planned.

Technology stack

Grouped by what each part is for, with a plain-English line on what the technology actually is — the stack as a lesson, not a list.

The data itself

Component	Technology	Status	What this tech is	Role here
Content records	YAML + JSON Schema	live	YAML is a structured text format humans can read directly — labeled fields and lists, like a very tidy outline. JSON Schema is a rulebook that checks each file has the right fields.	One file per session, git-versioned in _CLASSES. The ONLY permanent commitment; every page and index is rebuildable from these.
Course registry	courses.yaml + courseconf.py	live	A single YAML file listing every course (its URL, id, media folder) plus a small Python helper that reads it.	Scripts take --course KEY; no ids or paths hardcoded anywhere.

Getting content in

Component	Technology	Status	What this tech is	Role here
Ingestion adapter	Chrome + Claude-in-Chrome + Circle internal API	live	Circle is the platform Outskill's LMS runs on; its 'internal API' is the data feed the LMS's own pages use. The Claude browser extension rides your logged-in Chrome session to call it — same access you have, automated.	Fetched data leaves the browser as file-drops to Downloads; nothing needs your password stored anywhere.
Scheduler	Cowork task 'catalyst-ingest'	live	Cowork's built-in scheduled tasks — a to-do that runs itself on a timer while Claude Desktop is open.	Daily 21:00 check for new postings + posting-lag tracking.

Processing & archiving

Component	Technology	Status	What this tech is	Role here
Pipeline scripts	Python 3.14 (_pipeline)	live	Plain Python programs — organize / fetch_transcripts / fetch_materials / run_recordings / build_site.	Idempotent: safe to re-run any time; per-course state files remember what's already done.
Video archiver	FFmpeg 8.1.2 (HLS -c copy)	live	FFmpeg is the Swiss-army knife of video tools. HLS is the chunked streaming format sites deliver video in; '-c copy' means FFmpeg stitches the chunks into a normal .mp4 without re-encoding — fast and lossless.	Downloads signed streams, verifies with ffprobe, always runs detached.
Media archive	V:\_CLASSES-media	live	A plain drive folder holding every recording, transcript, and handout.	16 GB and growing. Deliberately NOT in git and NOT OneDrive — too big, no need to sync.

Knowledge extraction

Component	Technology	Status	What this tech is	Role here
Extraction model	Claude Fable 5 (Cowork)	live	The AI doing the reading and writing: full transcript in, structured teaching records out.	~85-90k tokens/session with the enriched schema; corrects ASR garbles; flags point-in-time claims.

Publishing

Component	Technology	Status	What this tech is	Role here
Site generator	build_site.py	live	One Python script that turns the YAML records into every page you're reading — nothing here is hand-edited HTML.	Design locked to the approved Basecamp-5 reference page; 'edit data, not html'.

Search & memory (planned)

Component	Technology	Status	What this tech is	Role here
Search index	SQLite + FTS5 + vector ext	planned	SQLite is a database that lives in a single file, built into nearly everything. FTS5 is its full-text engine (find any phrase, fast). A vector extension adds meaning-based lookup on top.	Will cover full transcripts, not just extracted records. Vector extension choice deliberately open: benchmark sqlite-vec / sqlite-vector / vec1 on the real corpus.
Embeddings	Qwen3-Embedding-4B via Ollama	planned	An embedding model turns text into lists of numbers that capture meaning, so 'similar ideas' can be found even when no words match. Ollama runs such models locally — your data never leaves the machine.	Runs on the RTX 3080 Ti (12 GB).
Local service	Node v24 + shortcut	planned	Node runs JavaScript outside the browser — here, a tiny local web server.	Will serve the search UI and smooth recording seeks; the static pages keep working standalone without it.

Access & authentication

System	How auth works	Notes
Outskill LMS (Circle.so) - platform.outskill.com	Paul's Chrome profile session cookie. No stored credential; if logged out, log back in via Chrome and the pipeline works again.	Internal API requires the authed browser. assets-v2.circle.so files are public hash-URLs (no auth).
Pilot Institute LMS - lms.pilotinstitute.com	Paul's Chrome profile (Logto OIDC). API calls need BOTH `Authorization: Bearer <localStorage.access_token>` and `X-Tenant-Id: er8hu0`. Note localStorage also holds a 43-char `logto_access_token` which is NOT the API token - using it returns 401.	Custom LMS, not Circle. REST at learn-api.pilotinstitute.com; quizzes at quizbank-api.pilotinstitute.com; video on Bunny.net Stream (library 400978). Signed MP4/HLS URLs expire ~1h (vs Circle's ~24h). Lesson pages do NOT freeze the renderer here.
Otter.ai	Paul's Chrome profile session. Fallback source only.	Generic speaker labels; suspected 4h cap; likely dropped once posting-lag pattern is confirmed.
NotebookLM	Google account in Chrome. No API - uploads stay manual drag-and-drop.
Claude (Cowork)	Claude Desktop login. Fable 5 used for extraction; usage drawn from plan's weekly limits.

Configuration registry

Item	Where	Why
Chrome automatic downloads policy	HKCU\Software\Policies\Google\Chrome\AutomaticDownloadsAllowedForUrls = https://platform.outskill.com + https://lms.pilotinstitute.com	Allows the harvest's multi-file Blob drops to Downloads. Set by Paul 2026-08-15 (reg add). lms.pilotinstitute.com added by Paul 2026-08-16 (CL-013, done). STILL PENDING: community.livingwithpixels.com (CL-014). Each unlisted origin fails the same way - the first drop lands, every later one is silently swallowed, and a page reload does NOT clear it. Treat 'add the origin' as step one of onboarding any new platform.
Course registry	_pipeline\courses.yaml (loader: courseconf.py)	One entry per course - baseUrl, courseId/classId, mediaDir, manifest/state filenames, evergreen flag. Every pipeline script takes --course KEY, so adding a course is a registry edit not a code edit. Added 2026-08-16. NOTE manifestDir: courses whose manifest lives in mediaDir (part-107) MUST declare it, or the path resolves to Catalyst's manifest.json.
Python interpreter	py -3.14 (C:\Users\pberg\AppData\Local\Python\pythoncore-3.14-64)	The bare `python` on PATH is 3.13 and has NO pyyaml. Every pipeline script must be run as `py -3.14`.
Scheduled task: catalyst-ingest	Cowork > Scheduled (SKILL.md under Documents\Claude\Scheduled\catalyst-ingest)	Daily 21:00 ingest cycle + posting-lag tracking. Runs only while Claude Desktop is open.
Paths	Library/git: C:\Users\pberg\OneDrive\Documents\_AI\_CLASSES \| Media: V:\_CLASSES-media \| Pipeline: _CLASSES\_pipeline \| Site: _CLASSES\_site	Media deliberately OUTSIDE OneDrive and git (16+ GB, growing).

Technical notes

DLP: the Chrome extension blocks tool-results containing signed URLs/tokens. All token-bearing data must leave the browser as Blob file-drops to Downloads.
CDP evaluate has a 45s timeout: chunk in-page harvest loops. Measured on LWP 2026-08-16: 20 lesson-detail fetches at 80ms pacing = ~20s, a comfortable ceiling. Accumulate in window.__* variables between calls.
Bulk NON-token data can leave the browser without any download permission: render it into the DOM as a <pre> block and read it with the extension's page-text extractor. 83 rows (9.2 KB) came through intact where javascript_tool results truncate at ~1 KB. tsv_to_bundle.py folds that TSV back into the standard meta bundle. Use this for metadata; NEVER for signed URLs - those are what the DLP guard exists to stop, and routing them around it would defeat the guard.
The loopback data channel does NOT work: a page POSTing to http://127.0.0.1 is blocked by Chrome's Local Network Access rules and hangs until it times out (it also wedges the renderer - reload the tab afterwards). Access-Control-Allow-Private-Network on the receiver is necessary but not sufficient; Chrome 138+ additionally wants a user permission grant. bundle_server.py is kept for the day that grant exists.
Circle transcript endpoints can be PUBLIC even when the lesson API is not. On community.livingwithpixels.com /internal_api/* returns 401 to plain urllib but /media_transcripts/<id>.vtt returns 200 - so fetch_transcripts.py pulls all 82 VTTs with no browser at all. Always probe (fetch_transcripts.py --probe) before assuming the browser is required.
Do not size a course from one sample lesson. LWP's 3-minute intro measured 634 MB/hour; Catalyst's long screenshares average 211 MB/hour and Pilot Institute 350. Pilot Institute's single-sample projection came in 44% low. Treat any single-lesson extrapolation as a floor.
Lesson pages with the video player can freeze the tab renderer - always harvest from the space root page.
ffmpeg must be launched detached (Start-Process); children of tool-call shells get reaped mid-download.
Signed HLS URLs expire ~24h; fetch playback_urls immediately before launching run_recordings.py. Re-runs are idempotent.
Circle transcripts are WebVTT at /media_transcripts/<id>.vtt, same-origin - fetch as text, no download needed.
Pilot Institute: `GET /api/v1/class/<id>` returns the whole curriculum tree in one call, but does NOT hydrate `captions` or `contentBlocks` - those need a per-lesson `GET /api/v1/lesson/<id>`. 318 lessons fetched fine in 3 chunks of ~120 under the 45s CDP limit.
Pilot Institute video is Bunny.net direct signed MP4 (`play_720p.mp4`) - no ffmpeg needed, ~1s per lesson. There is no 1080p MP4; 1080p exists only as an HLS variant.
The one-sample trap applies to TIME as well as size: a single Part 107 lesson benchmarked Whisper large-v3 at 12.6x realtime (=> 77 min for the course), but the full run settled at ~7.5x (spread 4.4x-13.0x) => ~2 hours. Dense narration is far slower than slide-heavy segments. Benchmark figures are ceilings; budget 40-60% more.
Transcription: faster-whisper large-v3, float16, VAD on, RTX 3080 Ti. The pip nvidia-cublas / nvidia-cudnn wheels are NOT on the DLL search path by default - pi_transcribe.preload_cuda_libs() adds them. Without it CTranslate2 silently falls back to CPU int8 (~20x slower) and the run still 'succeeds'.
Pilot Institute materials: auth differs PER ENDPOINT FAMILY on the same host. /api/v1/documents/library/download/<guid> and /api/v1/contents/bunny/download/<id> are PUBLIC; /api/v1/contents/images/download/<guid> is 403 without a bearer; /api/v1/contents/transcripts/download/... is 403 even WITH one. Probe each family - do not generalise from one.
Windows WMI misreports the RTX 3080 Ti as 4 GB (32-bit overflow); it is 12 GB.
Outskill posting lag: weekend sessions post Mon/Tue; weekday sessions ~2 days later. Tracked in posting-lag.md.