Skip to content

iMessage memory agent

ECHO

A second brain that lives in iMessage — turning conversations into commitments, deadlines and decisions, and watching threads on your behalf.

PrototypeAIFull Stack

Hackathon prototype. Runs on a seeded demo history; needs a Photon Spectrum project and an LLM key to run live.

Source(opens in a new tab)
Period
Sep 2026
Status
Prototype
Role
Agent design, Backend
Context
Built for the Photon iMessage Agent Challenge.

The problem

Why this needed to exist

Commitments, deadlines and decisions get made in chat and then disappear into scrollback. Remembering them is left to the person least able to: the busy one.

Who it’s for

  • People who run their lives and work through messaging

The solution

How it works

One database, one extraction pipeline, many surfaces. Every inbound message runs through tool-use extraction into commitments, events, decisions, open questions and opportunities — each with an explainable urgency or confidence score. Briefs, a timeline, a knowledge graph and a weekly review are all queries over the same rows.

Implemented

What’s actually built

  • "Catch me up"

    An animated orb appears instantly, then morphs in place into a live brief card using Spectrum’s edit() on an already-sent card.

  • Cross-thread watches

    "When James replies, summarise his answer and tell me what to do" registers a watch on a group thread; his reply triggers a proactive DM to you elsewhere.

  • Screenshot to event

    Send a screenshot of a flyer; ECHO reads it with a vision model and confirms through a real iMessage poll before adding the event.

  • Three autonomy levels that change behaviour

    Suggest only, ask before acting, or act automatically — the screenshot and cleanup flows branch on each.

  • Explainable scoring

    The extraction schema requires a reason alongside every urgency, importance and confidence score, and every surface shows them together.

  • Privacy centre

    Pause, forget a conversation, export, or delete everything — and an activity log tags every autonomous action with the agent that took it.

My role

What I did

Design and implementation of the agent pipeline, data model, commands, dashboard and the orb interaction.

Responsibilities

  • Agent design
  • Backend
  • Dashboard
  • Interaction design (the ECHO orb)

Stack (verified from source)

  • Bun
  • Hono
  • SQLite (bun:sqlite)
  • Claude / Gemini (provider-neutral)
  • Photon Spectrum
  • TypeScript

Decisions

Technical and design choices

  1. 01

    SQLite over hosted Postgres

    Zero external services means nothing can be down on demo day; the dashboard reuses the same process and database.

  2. 02

    Search without embeddings

    A multi-turn tool-use retrieval loop over keyword searches. At this data scale, vectors would be ceremony — and it is called keyword search, not "semantic".

Trade-offs

Challenges along the way

  1. 01

    Its own number, by design

    ECHO only sees threads it is actually in. That is the privacy model, and it is what makes "when James replies" real rather than surveillance.

  2. 02

    Provider differences

    Gemini’s function schema rejects JSON Schema unions and numeric bounds, so tool schemas are converted properly rather than cast and hoped.

Lessons

What I took away

  • A deliberate 900ms pause before the brief is a presentation choice — and it should be documented as one, not passed off as latency.

What’s next

Planned — not built

Planned work

  • Semantic search
  • Time-based follow-ups
  • Auth on the dashboard
  • Writing events to a real calendar

Current limits, stated plainly

  • The dashboard has no auth and should not be exposed publicly.
  • Demo history is seeded directly into the database.