{ "site": "DATANALYSIS", "id": "DATA-REPRISE-20261110", "locale": "en", "status": "release_ready", "version": "1.0.0", "robots": "index, follow", "preparedAt": "2026-11-10", "updatedAt": "2026-11-10", "publishedAt": "2026-11-10", "title": "Resuming a data flow without multiplying its effects", "description": "Prepare the recovery of an interrupted data flow: request identity, protected writes, external effects, lost-confirmation and concurrency tests, with a recovery checklist.", "lede": "A job stops. The database has recorded the batch of orders, but the confirmation never reached the job. On restart, should the write be replayed? Without a suitable mechanism, that retry can create a second copy of every order.", "intro": [ "The problem is not only getting the job running again. You have to determine what was actually done, what remains uncertain and what can be repeated without changing the expected result. The scenario presented here is fictional; the checks proposed must be adapted to the architecture of the flow." ], "sections": [ { "id": "response-vs-operation", "heading": "Tell a failed response from a failed operation", "blocks": [ { "p": "When a request times out, the client knows it has not received a usable response. It does not necessarily know whether the receiving system carried out the operation. Retrying systematically is therefore not always free of consequences." }, { "p": "AWS engineering guidance describes this ambiguity and the value of idempotent APIs: the same call can be retried without multiplying the intended effects. It also sets out the limits related to the identity of the request, how long it remains valid and changes of intent. [1]" }, { "p": "For our batch of orders, we need to be able to recognise that a new attempt corresponds to the same operation, and not to a new, legitimate order." } ] }, { "id": "business-intent-identity", "heading": "Give the business intent an identity", "blocks": [ { "p": "A stable key can identify the batch, the order or the operation to perform, depending on the chosen scope. It must stay the same across attempts of the same request. Conversely, a new business request must be able to have a different identity, even if some values are equal." }, { "p": "Comparing only the content of a request is not always enough: two genuinely different orders can contain the same items and the same quantities. The choice of key should therefore be discussed with the business owner and the owner of the source application." }, { "p": "You also need to define what happens when a known key comes back with a different amount or recipient. The process must not confuse a correction, a calling error and an identical repetition. The idempotency documentation of an API, Stripe for example, should be consulted for its exact rules rather than assuming universal behaviour. [2]" } ] }, { "id": "protect-the-write", "heading": "Protect the write, not just check beforehand", "blocks": [ { "p": "A check “does this order already exist?” followed by an insert can let a duplicate through if two attempts run at the same time. Protection must cover the moment the effect is recorded." }, { "p": "Depending on the system, this can rely on a uniqueness constraint, a transaction or an idempotency mechanism provided by the destination API. The choice depends on the engine, the data model and the effects to protect. An application-level check does not automatically replace an atomic write guarantee. Uniqueness constraints are among PostgreSQL’s documented mechanisms; their scope must match the chosen business identity. [3]" }, { "p": "The recovery response must also be defined. If the request had already succeeded, the client should be able to retrieve a usable reference rather than remain in an undetermined state." } ] }, { "id": "external-effects", "heading": "Examine every external effect", "blocks": [ { "p": "No duplicate in the orders table does not guarantee no duplicate notification, invoice or write in another system. Each effect must be identified and tested within its own scope." }, { "p": "An architecture that coordinates several systems may have to handle intermediate states, delayed attempts and corrections. A local protection should not be presented as a global “exactly once” processing guarantee. The recovery method must state what it guarantees and which cases remain for a person to examine." } ] }, { "id": "test-uncertainty", "heading": "Test uncertainty before the first incident", "blocks": [ { "p": "Useful acceptance testing is not limited to the happy path. The accompanying checklist suggests testing the following situations with fictional data and expected results defined in advance." }, { "table": { "caption": "Test situations and expected checks", "head": [ "Test situation", "What to check" ], "rows": [ [ "Confirmation lost after the write", "Replaying the same request creates no additional effect; the initial result can be retrieved." ], [ "Two concurrent attempts", "The protection withstands concurrency, not just two successive calls." ], [ "Same key, modified content", "The agreed behaviour is explicit: rejection, conflict or correction workflow." ], [ "Late replay", "The limits on key retention and on replaying history are known." ], [ "One effect succeeds, another fails", "The partial state is visible and recovery does not blindly re-execute effects already carried out." ], [ "Uncertain destination result", "An owner and a reconciliation method are identified before any new attempt." ] ] } }, { "p": "A passing test establishes the behaviour observed under its conditions. It does not remove the need to monitor drift, document versions and review the scope when the flow changes." } ] }, { "id": "recovery-as-service", "heading": "Prepare recovery as part of the service", "blocks": [ { "p": "For a specific flow, the team must be able to name the business request, its key, the system that is authoritative on its state, how long a replay is supported and who owns exceptions. It must also know which outputs to suspend when the result is uncertain." }, { "p": "At DATANALYSIS, integration work means connecting these rules to the writes, the checks and the uses that depend on them. The goal is a verifiable recovery, suited to the flow, rather than a “retry” button whose consequences remain unknown." } ] } ], "body_markdown": "## Tell a failed response from a failed operation\n\nWhen a request times out, the client knows it has not received a usable response. It does not necessarily know whether the receiving system carried out the operation. Retrying systematically is therefore not always free of consequences.\n\nAWS engineering guidance describes this ambiguity and the value of idempotent APIs: the same call can be retried without multiplying the intended effects. It also sets out the limits related to the identity of the request, how long it remains valid and changes of intent. [1]\n\nFor our batch of orders, we need to be able to recognise that a new attempt corresponds to the same operation, and not to a new, legitimate order.\n\n## Give the business intent an identity\n\nA stable key can identify the batch, the order or the operation to perform, depending on the chosen scope. It must stay the same across attempts of the same request. Conversely, a new business request must be able to have a different identity, even if some values are equal.\n\nComparing only the content of a request is not always enough: two genuinely different orders can contain the same items and the same quantities. The choice of key should therefore be discussed with the business owner and the owner of the source application.\n\nYou also need to define what happens when a known key comes back with a different amount or recipient. The process must not confuse a correction, a calling error and an identical repetition. The idempotency documentation of an API, Stripe for example, should be consulted for its exact rules rather than assuming universal behaviour. [2]\n\n## Protect the write, not just check beforehand\n\nA check “does this order already exist?” followed by an insert can let a duplicate through if two attempts run at the same time. Protection must cover the moment the effect is recorded.\n\nDepending on the system, this can rely on a uniqueness constraint, a transaction or an idempotency mechanism provided by the destination API. The choice depends on the engine, the data model and the effects to protect. An application-level check does not automatically replace an atomic write guarantee. Uniqueness constraints are among PostgreSQL’s documented mechanisms; their scope must match the chosen business identity. [3]\n\nThe recovery response must also be defined. If the request had already succeeded, the client should be able to retrieve a usable reference rather than remain in an undetermined state.\n\n## Examine every external effect\n\nNo duplicate in the orders table does not guarantee no duplicate notification, invoice or write in another system. Each effect must be identified and tested within its own scope.\n\nAn architecture that coordinates several systems may have to handle intermediate states, delayed attempts and corrections. A local protection should not be presented as a global “exactly once” processing guarantee. The recovery method must state what it guarantees and which cases remain for a person to examine.\n\n## Test uncertainty before the first incident\n\nUseful acceptance testing is not limited to the happy path. The accompanying checklist suggests testing the following situations with fictional data and expected results defined in advance.\n\n| Test situation | What to check |\n|---|---|\n| Confirmation lost after the write | Replaying the same request creates no additional effect; the initial result can be retrieved. |\n| Two concurrent attempts | The protection withstands concurrency, not just two successive calls. |\n| Same key, modified content | The agreed behaviour is explicit: rejection, conflict or correction workflow. |\n| Late replay | The limits on key retention and on replaying history are known. |\n| One effect succeeds, another fails | The partial state is visible and recovery does not blindly re-execute effects already carried out. |\n| Uncertain destination result | An owner and a reconciliation method are identified before any new attempt. |\n\nA passing test establishes the behaviour observed under its conditions. It does not remove the need to monitor drift, document versions and review the scope when the flow changes.\n\n## Prepare recovery as part of the service\n\nFor a specific flow, the team must be able to name the business request, its key, the system that is authoritative on its state, how long a replay is supported and who owns exceptions. It must also know which outputs to suspend when the result is uncertain.\n\nAt DATANALYSIS, integration work means connecting these rules to the writes, the checks and the uses that depend on them. The goal is a verifiable recovery, suited to the flow, rather than a “retry” button whose consequences remain unknown.", "closing": "Let’s scope the recovery of one specific flow. Bring its source, its destinations and an interruption scenario. We can define the effects to protect, the useful tests and the recovery responsibilities.", "cta": { "label": "Scope the recovery of my data flow", "url": "https://datanalysis.re/contact?lang=en" }, "worksheet": { "label": "Open the checklist (printable HTML)", "url": "https://datanalysis.re/ressources/data-engineering/resuming-data-flow-without-duplicate-effects/checklist", "fields": [ [ "Business intent and key", "Record a batch of orders. Stable batch key, identical across attempts of the same request; a new business request gets another key, even if some values are equal." ], [ "Destination and reference state", "Orders database: it is authoritative on the batch state. Known period during which a replay is supported and the key is retained." ], [ "Writes and effects", "Order insert, notification, invoice, write in another system: each effect identified and tested within its own scope; write protected by a uniqueness constraint, a transaction or the destination API’s idempotency." ], [ "Concurrency and lost-confirmation tests", "Confirmation lost after the write, two concurrent attempts, same key with modified content, late replay, partial effect, uncertain destination result; fictional data and expected results defined in advance." ], [ "Recovery and exceptions", "Initial result retrievable through a usable reference; outputs to suspend if the result is uncertain; reconciliation before any new attempt; same key with different content: agreed rejection, conflict or correction workflow." ], [ "Owner, evidence and review date", "Named owner for exceptions; evidence per effect: request, key, expected and observed result; date of the next review (fictional example: no real date)." ] ], "pdf": "https://datanalysis.re/downloads/data-reprise-checklist.en.pdf" }, "pdf": { "article": "https://datanalysis.re/downloads/data-reprise-article.en.pdf", "worksheet": "https://datanalysis.re/downloads/data-reprise-checklist.en.pdf", "pack": "https://datanalysis.re/downloads/data-reprise-pack-fr-en-de.zip" }, "related_articles": [ { "title": "When data changes meaning: preparing changes across applications", "url": "https://datanalysis.re/ressources/data-engineering/data-changes-meaning-across-applications" }, { "title": "Modernising a data platform while the business keeps working", "url": "https://datanalysis.re/ressources/data-engineering/modernising-data-platform-while-business-runs" } ], "sources": [ { "n": 1, "label": "AWS Builders’ Library | Making retries safe with idempotent APIs", "url": "https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/", "note": "ambiguity of a timed-out request, idempotent APIs, limits related to request identity, validity period and changes of intent" }, { "n": 2, "label": "Stripe | Idempotent requests", "url": "https://docs.stripe.com/api/idempotent_requests", "note": "idempotency rules specific to that API, to be consulted rather than assuming universal behaviour" }, { "n": 3, "label": "PostgreSQL 18 | Constraints", "url": "https://www.postgresql.org/docs/18/ddl-constraints.html", "note": "uniqueness constraints, whose scope must match the chosen business identity" } ], "sources_note": "Primary references consulted on 10 November 2026. The example and the checklist are our own teaching application: they are neither a client case nor a guarantee independent of the system you choose. No Stripe or PostgreSQL connector is claimed here.", "canonical": "https://datanalysis.re/ressources/data-engineering/resuming-data-flow-without-duplicate-effects", "alternates": { "fr": "https://datanalysis.re/ressources/data-engineering/reprendre-flux-sans-multiplier-effets", "en": "https://datanalysis.re/ressources/data-engineering/resuming-data-flow-without-duplicate-effects", "de": "https://datanalysis.re/ressources/data-engineering/datenfluss-wiederaufnehmen-ohne-doppelte-wirkungen" } }