Data imports and synchronization

The job

A partner sends a nightly catalog or account file. Your application must validate it, import accepted records, report rejected rows, and make the resulting data available to downstream systems. Completion means the intended version is published with a visible record of what succeeded and what needs correction.

A related synchronization flow starts when a record changes in one service. It updates the connected systems and tracks their acknowledgments. Both jobs need stable identities and recovery when only part of the work finishes.

The flow

Record the file’s import identifier and version, then validate its structure. Split accepted records into batches. Process each batch through durable steps, using stable record keys for writes to outside systems. Emit explicit batch results and rejected-row events for an operational view.

After the application’s quality checks pass, publish the new version or update the relevant read model. A synchronization workflow similarly consumes a source change, applies mapped updates, and records which destinations have accepted it. The application defines mapping, conflict resolution, and publication policy.

The difficult moment

A worker stops after several batches complete. Their durable step results remain saved, so recovery continues with unfinished work. A batch that was interrupted can run again. Outside writes therefore need idempotent upserts, operation keys, or destination-specific reconciliation.

Duplicate and out-of-order changes need domain rules. Carry a source version or other ordering signal when the source provides one, and check it before overwriting destination state. A durable workflow can preserve progress while still applying an incorrect update if its application conflict rules are wrong.

Ironflow’s role

step.map() and step.parallel() fan out independent batches. A branch is not a recorded step by itself: wrap each batch in its branch’s own uniquely named step.run() so a finished batch is saved and a retry skips it. Events connect downstream updates. Projections can derive progress, rejected-record queues, and read views from the input facts your application records.

Pure managed projection reducers derive state; external projection handlers perform side effects and need their own idempotency. Rebuilding a view reprocesses the retained events. It does not recreate source data that you never recorded. Keep the original file or sufficient record facts when future rebuilds need them.

Implementation sketch

This is an illustrative flow:

1. Validate an import request and retain its source/version identity.
2. Reject invalid rows and form stable batches.
3. Process batches in durable steps with destination idempotency.
4. Record results and resolve failures or conflicting updates.
5. Apply publication checks and expose the completed version.
6. Derive progress views; rebuild them from retained recorded inputs when needed.

Starting point

• Step primitives demonstrates parallel/map work and durable steps inside branches.
• Projections explains managed reducers, external handlers, checkpoints, and rebuilds.
• Subscriptions covers event delivery connections you can use to connect a synchronization consumer.