φ(ai) PHI AI
DocumentationDocs
Repository

Streaming data manager

Real-time feeds beside the batch pipeline — admissions and discharges, lab results, claims remittances. Streaming has no clean-run boundary, so each partition keeps its own checkpoint with explicit gap accounting, never sharing the bulk pipeline's watermark. A gap is surfaced loudly rather than interpolated.

10,407
events today across all feeds
6
partitions under checkpoint discipline
1
open gaps — surfaced, not interpolated

Partitions

FeedPartitionCheckpoint Latest offsetLag Events todayLast eventState
ADT admissions/discharges 0 8,842,310 8,842,310 0 1,204 13:58 healthy
ADT admissions/discharges 1 8,790,522 8,790,522 0 1,187 13:58 healthy
ADT admissions/discharges 2 8,811,948 8,811,990 42 1,163 13:51 gap — attention
Offsets 8811949-8811990 unacknowledged after broker restart. Surfaced loudly, never interpolated - an interpolated gap is a silent data-integrity failure that looks like a healthy pipeline.
Claims 835 remittance 0 902,114 902,114 0 214 12:40 healthy
ORU lab results 0 4,410,221 4,410,221 0 3,388 13:59 healthy
ORU lab results 1 4,398,710 4,398,710 0 3,251 13:59 healthy
Why two disciplines. The bulk pipeline has a clean-run boundary and a watermark; a stream has neither — only per-partition offsets and the honesty to account for every one. The gapped partition above holds its checkpoint until the missing offsets are recovered or the loss is formally accepted and recorded; it does not skip ahead.

Source EMR lookup — pull one patient now

Between bulk runs, new patients exist at the source that PHI AI has not seen. Search the source EMR directly and import an individual patient over the per-patient read path — the same path bulk-less vendors force for entire populations.

An individual import runs read → encrypt → store → index for one patient and lands in your session's sandbox: the patient appears in Patients & charts, the import is audited (ingest.patient_imported), and the next reconciliation on the Bulk import manager counts them.