Reflecta

Problem

A voice check-in is only useful if the post-call path is reliable: the webhook may fire twice, a provider may be down, and a safety signal cannot wait on a JSON blob from an LLM.

Cross-session context has to come from prior transcripts, not from the model “remembering.” That means embeddings and retrieval, not a longer prompt.

Constraints

Bolna webhooks are not HMAC-signed in the current vendor docs. Auth is a shared secret on the query string, with an HMAC verifier wired for if they add a header, and an IP allowlist logged (enforced at infra).

Safety detection has to run on the raw transcript, before and independent of LLM analysis, including Hindi/Hinglish Roman-script phrases.

Short or empty transcripts must not call Groq or Hugging Face. A word-count gate returns a default analysis instead.

Architecture

POST /api/bolna/webhook verifies the secret, parses the payload, and upserts the call row ON CONFLICT (id). Transcripts upsert on call_execution_id. If the call is already marked completed, the pipeline is skipped — that is the idempotency guard.

Crisis detection is a regex pass (critical vs elevated) on the transcript. It can override stored risk even if later LLM analysis is bland or missing.

Analysis prefers Groq JSON mode, then Hugging Face, then a heuristic fallback. Embeddings are nomic-embed-text-v1 via Hugging Face, stored in pgvector, with the nomic task prefixes (search_document / search_query) so retrieval of prior calls is not a naive cosine on unprefixed text.

Reliability / eval

Three independent failure domains: ingest (idempotent upsert), safety (deterministic, pre-LLM), analysis (provider waterfall ending in a heuristic, not an exception).

Post-call analysis extracts structured fields when the model succeeds — stress, mood, sleep, work pressure, burnout-risk score, sentiment, key concerns, risk factors — and those fields are nullable. Missing JSON is not filled with invented numbers.

I do not treat field-count as a quality metric. The eval question is whether crisis phrases still fire when Groq is down, and whether a duplicate webhook restates the same call.

Known failure modes

• Regex safety is brittle

  Phrase lists catch direct ideation and a set of Hindi/Hinglish romanisations. Indirect or novel phrasing will miss. That is why the path is an override, not a classifier I pretend is complete.

• Vendor webhook auth is weak

  A query-param secret is replayable if logs leak the URL. HMAC is implemented but unused until Bolna sends a signature header. The real control is infra IP allowlisting, which this app can only log, not enforce by itself on Vercel.

• Heuristic fallback is not analysis

  When Groq and Hugging Face both fail, the default object is a structured empty-ish result, not a fake report. Dashboards that render it as a “score” would be lying. The UI has to treat low analysis_confidence as unusable.

What I would do differently

• I would add a second, still-deterministic, safety pass (e.g. a tiny on-device classifier) without ever letting the LLM be the gate.
• I would store webhook payload hashes so idempotency is content-addressed, not only execution-id based, for the case where Bolna reissues a new id for the same call.