Blog
Plugging Pylon, Fireflies and Attio into a reducer
Reducers read snapshots, never APIs. Every source publishes the same thing: a schema, record IDs, provenance and a version. Refresh is a new snapshot.
In short
- Reducers read snapshots, never APIs.
- Every source publishes the same thing: schema, record IDs, provenance, a version.
- Refresh is a new snapshot. Old runs keep theirs.
A reducer never calls Pylon. It reads a snapshot that a Pylon connector published. That one boundary is what lets the same question run over support tickets on Monday and call transcripts on Tuesday without touching the workflow.
One contract for every source
Import records
Raw evidence
Structured fields
Source references
Check the contract
Schema
Coverage
Provenance
Versioned snapshots
Run A → snapshot 1
Run B → snapshot 2
Every adapter, a connector or a file parser, emits the same things: raw evidence, candidate structured records and provenance back to the source object. Common validation checks the schema and the coverage, then publishes an immutable snapshot. Queries and workflows read snapshots. The proof of the abstraction is that a dataset from an uploaded CSV and one from a connector go through the same compiler, the same workers, the same result tables and the same lineage API.
What a snapshot promises
- A schema you reviewed, with a bounded sample, before publication.
- Stable record IDs, local to the snapshot and independent of display names or the source’s primary keys.
- Provenance per record: the row and header, the JSON pointer, or the source object and its evidence.
- Complete coverage, or an explicit count of omissions and errors published with the snapshot. A partial snapshot is never presented as complete.
- A version. Two identical rows remain two rows. There is no deduplication in this release; dedupe before import if you need it.
Connected today
| Source | Records |
|---|---|
| Pylon | Support tickets and conversations |
| Fireflies | Call transcripts |
| Attio | People, companies and notes |
| Files | CSV, JSON, JSONL, TXT and Markdown. Defaults: 10,000 records or 20 MB per file. CSV cells stay strings, so an ID like 001 keeps its zeros. |
More connectors are coming. Each one has to prove pagination and resume, malformed records, missing fields, schema drift, cancellation, expired permissions and interrupted publication before it ships.
Refresh is a new snapshot
A run pins the snapshot it read. A refresh is a new ingestion with its own configuration and trace, and it publishes only after validation. Old runs keep their pinned versions. A schedule declares one of two behaviours: use the latest snapshot already published at the run’s start boundary, or run a named refresh and wait for its snapshot before freezing the plan. Refresh and “latest” are never raced against each other.
Binding a question to a source
A reusable question names the semantic slot it needs, such as content. A query binds tickets.messages or calls.transcript to that slot. The types have to match, and so does the meaning: a matching string type does not make a question written for articles the right question for a transcript. That review is yours; the compiler checks the shape.
What connectors do not promise
- An atomic snapshot of the remote system. Each adapter records what its platform actually supports.
- Stable pagination or complete search coverage. The collection specification and every page checkpoint are stored, so you can see what was fetched.
- Exact replay of a fetch. Refetching from a changing system is a new collection. Reprocessing captured evidence with a new parser is a new derived snapshot.
Next: in the app, open Data, upload a new version of a dataset you already ran, and open the old run. Its snapshot ID has not moved.