Keeping a live transcript and pen strokes on the same clock
Endeo's value is the join between live speech and Apple Pencil strokes — tap a scribble, hear what was said then. That join is only as good as one shared session clock. Recognizer timestamps are not your timestamps, and restarts are how two capture streams quietly desynchronize over a forty-minute talk.
The feature is a join, not two features
It's tempting to build this as two independent capture pipelines and worry about correlation later. Recording works, transcription works, notes work, ship it, then wire them together in a follow-up. We started down that road and it's a trap, because the correlation is the entire product and it constrains both pipelines in ways you can't retrofit cheaply.
Once you accept that the join is the feature, the design question becomes narrow and answerable: what timestamp do I attach to a stroke, what timestamp do I attach to a transcript segment, and are those two numbers measured against the same origin. Everything else is detail.
The failure mode is unglamorous and very hard to notice in testing. Nothing crashes. The transcript is fine, the notes are fine, and the link is off by eleven seconds in a way that feels wrong to a user and looks correct to a developer scrubbing through a two-minute test recording.
One session clock, and nothing else gets a vote
A study session in Endeo starts a single monotonic clock. Every event either subsystem produces is stamped as an offset in seconds from that origin. Not a wall-clock date, not a per-request timestamp, not a frame index: one number, relative to the moment the session started.
Wall-clock dates are the obvious wrong answer and they're wrong in a way that bites rarely enough to survive testing. Date() moves. Clock sync, a timezone change while crossing a state line, a user adjusting their time settings mid-session. For a duration measured within a single session you want a monotonic source that only ever counts forward, and you want the difference between two of its readings, never its absolute value.
The strokes side is straightforward once that exists. PencilKit hands you strokes as they complete, and we stamp each one with the session offset at completion. Strokes are short, so attaching the offset at the end rather than the start is close enough for a link that resolves to a sentence. The transcript side is where the actual work is.
- 01Session startsOne clock is established
- 02Pen strokesPencilKit captures, stamped against that clock
- 03AudioLive transcription, stamped the same way
- 04LinkA note at minute 14 binds to minute 14 of the talk
- 05SummarizeBoth streams go to the model together
The correspondence is the product. Two separate recordings would be half as useful.
The recognizer's timestamps are not your timestamps
This is the detail that costs people a day. Speech recognition results carry timing information for each recognized segment, and that timing is relative to the beginning of the current recognition request. It is not relative to your session, and if you have started more than one request, it is not even relative to the same origin as the results you got five minutes ago.
So every segment timestamp has to be translated on arrival. When a recognition request begins, record the session offset at that instant and keep it alongside the request. Every segment that request produces gets that base added to its own relative timestamp, and only the result of that addition is ever stored. The raw value from the recognizer never leaves the adapter that owns the request.
Getting this backwards is subtle because it looks perfect at first. Your first recognition request starts at session offset zero, so its relative timestamps are also correct absolute ones. Everything lines up. The bug only appears after the first restart, which in a short test may never happen, and in a forty-minute talk happens constantly.
Restarts are the enemy of a shared clock
Live speech recognition is not designed to run continuously for the length of a sermon. Requests end, from duration limits, from silence, from an audio route change when someone plugs in headphones, from the recognizer simply finishing. A long session is a chain of recognition requests stitched together, not one long one.
That means restart handling isn't an error path, it's the main path, and it needs to be boring. Ours ends the current request, records the session offset for the next one before starting it, and starts it. The gap between the two is real, usually short, and we'd rather have an honest half-second hole in the transcript than a stitched-together stream whose timestamps have quietly drifted by the length of every gap that ever occurred.
The other half is pauses. If a user pauses a session, the session clock has to pause with it, or every stroke and every segment after the pause is offset by however long they stepped away. We keep an accumulated pause duration and subtract it, which is simple to write and easy to forget, because a paused session is the one state nobody thinks to test twice.
What you can build once the clock holds
The payoff is that correlation becomes a range query and stops being a feature. Tapping a note asks for transcript segments overlapping that stroke's offset, widened by a few seconds on each side because people write slightly behind what they hear. Scrubbing the transcript highlights the strokes made during that window. Both directions fall out of the same stored number.
It also makes the session summary honest. The Edge Function that calls Claude to summarize a session sends the transcript with the user's note offsets marked in it, so the summary can weight what the person actually reacted to rather than treating forty minutes of speech as uniformly interesting. That only works because the offsets mean the same thing on both sides.
None of this is difficult code. It's about ten lines of clock management in a place where it's easy to write zero. The reason to be deliberate about it is that a drifting clock produces a product that feels subtly broken while every individual subsystem reports that it is working correctly, and that is the most expensive kind of bug to find late.
Questions
- Why can't each capture stream keep its own timeline?
- Speech engines and pencil input report time on different bases, with gaps and restarts. Without one session clock everything else maps into, "what was said when I wrote this" becomes an accumulating guess.
- What breaks transcript ↔ ink alignment in practice?
- Using the recognizer's segment timestamps as absolute truth, and restarting recognition without rebasing onto the session clock. Small skew early becomes unusable join error by the end of a long session.
- What product features depend on a correct shared clock?
- Tap-to-hear a note, scrubbing a transcript beside ink, and any later search that claims temporal context. Those are one feature — the join — not two independent exports.
Sources
- Apple Speech framework
- Apple Pencil / PencilKit
- Endeo case study
- Apple — CMClock / media timing concepts — Shared clocks are a media problem as much as a UI one.