Skip to content

Roadmap — Recorder + Event-streams-per-topic (context banks)

Source: an internal design session, 2026-08-05 (notes not in this repository).

Two big asmltr initiatives came out of it — a first-class recording app and event streams organized per topic ("context banks") — plus a set of items that belong to the separate downstream phone-agent project (NOT asmltr), and two research asks. This roadmap sorts everything and defines the build order.

A. Streams — event streams per topic (FOUNDATION)

Terminology: these are streams (asmltr streams), not "context banks" — a stream is a per-topic event stream. Recall = retrieve-on-demand via a tool (appends at the prompt tail → cache-safe; never spliced into replayed history). A small always-on working-set summary rides the stable prompt block (re-sent only on change via stableHash inject-once). Freshness is an edge-triggered watermark: a session is told "↑ N new" only when the count of events from other sources rises past what it was last told (its own writes never re-trigger it); the nudge lives in the volatile tail. Search: SQLite FTS5/BM25 now → sqlite-vec + tiny static embeddings + RRF later. Prior-art basis: Letta shared memory blocks (injected working-set) + archival retrieve-on-demand; A2A contextId = a correlation key only (borrow it), MCP resources/tools = the recall surface. See the recall-design briefing on issue #93.

The insight: we don't need a brand-new "context bank" primitive — we need to organize the event stream we already have by topic. Today the collector catches every event from every session into one global stream. Instead, associate sessions with a topic/project so each topic has its own stream a session can register to, recall from on demand (unmodified history, no recompaction, no token-cache invalidation), and drop new material (recordings, files, notes) into.

  • A1. context_banks store + session↔bank association. A bank = {id, name, slug, description, created}; a session can belong to a bank (nullable FK / join). Collector can filter the event stream by bank (union of member sessions + bank-scoped injected assets).
  • A2. Bank-scoped recall. asmltr_bank_recall(bank, query|recent) toolbelt tool + a core endpoint that returns a bank's raw event history on demand — so a session pulls context when it asks, not by carrying it in every turn (protects the prompt cache — the explicit concern in the meeting).
  • A3. Auto-association. A new session that says "I want to work on Ops Hub" resolves/creates the bank and registers to it (prompt + tool). Deterministic slug match first; model-assisted fallback.
  • A4. Turn-based hooks, harness-consistent. A uniform per-turn hook (Claude/codex/gemini) — injected at runtime the way asmltr claude already wraps a session — that self-reports each turn into the bank's stream. (Builds on the existing claude-code hook.py; generalize.)
  • A5. Search. Semantic search over a bank's stream: time-series (source of truth) → export to a vector DB (rolling), tokenize only new content. Start with the existing event store + a lightweight embedding index; keep it optional (RAM-aware — the meeting flagged vector DBs as RAM-hungry).
  • A6. Docs + system-prompt awareness so agents know banks exist and how to use them.

B. Recording app (the star) — Android app + GUI, shared backend

The recorder is fundamentally a context-capture tool. Backend shared between the Android app and the dashboard; the recording component is the DRY piece.

  • B1. Recorder MVP (backend + minimal UI). Record (or upload) audio → chunked transcription (reuse the core STT path proven in the meeting-transcription flow: downsample → segment → /v2/transcribe → stitch) → store recording + transcript as a first-class recording record (in a data silo). Date/time + file. This alone delivers the meeting-capture workflow.
  • B2. Diarization + speaker labeling. Speaker split (diarization), then a smart pass to extract spoken names and assign them to speaker labels → people records, linkable across recordings (search "any recording with the team"). See B11 for voiceprinting (gated on legal).
  • B3. AI enrichment on save. Auto semantic title, ≤500-word description/summary, extracted action items, highlights — generated by the configured reasoning engine right after transcription (like ops-hub does for data structures). Title/description override-able; override locks out AI rewrite.
  • B4. Recordings library UI. List with AI title/desc/date; open → play + transcript with speakers; edit transcript (highlight edits + timestamps, keep original, show confidence); re-transcribe (pick model). Web GUI + app, shared.
  • B5. Save-to-target flow. On finishing a recording: dropdown to pick an active project / event stream (bank) → save; optional checkmark to also fire a prompt into a new session or an existing session ("look at this meeting and send me X on Discord"). So: save-only · save+new-session · save+steer-existing-session.
  • B6. Connector-aware share window. A share action that enumerates the user's wired connectors (email/Discord/Telegram/…) and lets you send the recording+transcript with optional text. Connectors can register their own share target (extension point) on top of plain file download.
  • B7. Media capture as timestamped recording assets. Snap stills (phase 1; images because engines have vision) — and later short video clips — during a recording; stored as assets timestamped relative to the recording, clickable in the transcript with ±15–20s playback. Plus a media gallery / direct file→session upload in the app + GUI so you don't have to forward through Discord.
  • B8. Real-time meeting mode. (1) Highlight→ask: highlight transcript text mid-recording → "send to agent" → background lookup → silent notification or optional voice playback. (2) Live participation: wake-word full participation (reuse the Discord wake-word flow). Per-recording setting: agent active-listening vs background-on-demand. Interactions timestamped + stored as meta.
  • B9. Music features. Detect music vs speech (and route accordingly); tempo / genre / known-song ID; instrument ID; track separation; on-device where a TPU/GPU exists (benchmark device → expose features by capability). Lower priority; fun/creative layer.
  • B10. Precise transcript timestamping — word/segment-level timestamps (needed for B7/B8 anchoring).
  • B11. Voiceprinting — GATED on the legal deep-dive (§E1). Default: voiceprint the owner only; third parties require explicit separate opt-in at record-time; encrypted, per-person, deletable, retention policy. Jurisdiction-aware.

C. Cross-cutting

  • C1. Direct file→session/bank upload in the app + GUI (the dashboard already has an upload button; extend to the app + target a bank/session). Removes the "forward a photo through Discord" workaround.

D. SEPARATE — a downstream phone-agent project — NOT asmltr

the team these as a separate issue ("don't get too hung up"). They belong to the ops-hub / phone-agent project, filed separately there (this roadmap only records them):

  • Real-time transcription API so the admin sees the caller's words before they finish speaking.
  • Admin listen-in button (speaker on, mic muted) while watching a live call.
  • Text-steer Rick mid-call — inject a note into his context from the admin UI.
  • Future: Eve observes calls, built on the same steer framework.
  • A2A: Rick calls MCP but returns an audio stream; investigate whether voice agents can use agent-to-agent protocol like text agents → a path to unify voice + text agents. (Overlaps §E2.)

E. Research

  • E1. Voice-printing legal deep-dive (IN PROGRESS) — BIPA/CUBI/WA, CCPA/CPRA, GDPR Art. 9, wiretap consent, the Google-Photos-facial-scan analogy → a defensible feature design. → docs/research/VOICE-PRINTING-LEGAL.md.
  • E2. A2A protocol investigation — can voice agents (audio-stream models that still call MCP) speak A2A like text agents? Framework that drives phone-agent text-steer today and Eve-observes-calls later.

Build order

  1. E1 voice-printing research (parallel, informs B11).
  2. B1 recorder MVP — fastest tangible value (the meeting-capture workflow), and everything hangs off the recording record.
  3. A1–A3 context-bank foundation — so B5 can target a bank.
  4. B3 AI enrichment, B2 diarization/people, B4 library UI.
  5. B5/B6 save-to-target + share; C1 direct upload.
  6. B7 media assets; B8 real-time meeting mode.
  7. A4/A5 turn hooks + search; B10 precise timestamps.
  8. B9 music; B11 voiceprinting (post-E1); E2 A2A (feeds the separate phone-agent work).