Skip to content
masterclassBuddyTech Buzz5 fresh blogs · 5:00 AM IST

n8n blueprint · Data engineering

Scheduled ETL between systems

A nightly n8n workflow pulls changed records from a paginated source API, transforms and validates them in Code nodes, upserts them into a Postgres warehouse in batches, and posts a run report, with an Error Trigger workflow for failures. It must never lose or duplicate rows: reruns must be idempotent, and the sync cursor may only move forward after a successful load.

Steps
8
Subjective questions
33
Objective questions
33

The problem and the outcome

Problem: Teams copy data between a SaaS system and their warehouse by hand or with fragile scripts. Rows go missing after partial failures, duplicates appear on reruns, and nobody notices when a night's sync silently fails.

Outcome: Each night an incremental, idempotent load runs on its own and leaves a per-run record of rows fetched, upserted and rejected. Failures reach the on-call channel with the execution link, and a rerun repairs the data without manual cleanup.

Architecture

Schedule Triggernightly, IST tz01Watermark tablelast synced cursor01HTTP Requestpagination + retry02Source APIpaged RESTCode nodemap + validate03IF noderow valid?04Rejects tablequarantined rows04Loop Over Itemsfixed batch size05Postgres warehouseupsert on key05Run log tableper-run counts06Run reportCode + Set fields07Slack node#data-ops channel07Error Triggererror workflow08read last cursorupdated_since=cursorGET next pageitems + next cursorraw recordsnormalised rowsinvalid rowsvalid rowsupsert batchbatch countsaggregate totalspost run summaryadvance cursorretries exhaustedfailure alertmark run failed

How it works, step by step

  1. A Schedule Trigger fires nightly, using the timezone set in workflow settings. A Postgres node reads the last successful cursor (an updated_at timestamp or an API cursor) from a watermark table. It subtracts a small overlap window so late-arriving records are not missed.
  2. An HTTP Request node calls the source API with its encrypted credential and uses the node's built-in pagination to follow the next-page cursor or link until the 'complete when' condition is met. A request interval respects rate limits, and Retry On Fail with a wait between tries absorbs transient 429 and 5xx errors.
  3. A Code node in JavaScript flattens the pages into one item per record, maps source fields to warehouse columns, casts types, normalises timestamps to UTC and adds a run_id and a payload hash to each row.
  4. An IF node separates valid rows from rows that fail checks such as a missing primary key or an unparseable date. Rejects go to a quarantine table with the reason and run_id, so one bad record does not fail the whole night.
  5. Loop Over Items splits valid rows into fixed-size batches. A Postgres node upserts each batch with INSERT … ON CONFLICT on the natural key, updating only when the incoming record is newer, so reruns are idempotent.
  6. After each batch, counts of rows fetched, inserted, updated and rejected are written to a run log table keyed by the n8n execution ID.
  7. When all batches succeed, a Code node builds the run report and the Slack node posts it. Only then does a Postgres node advance the watermark to the highest source timestamp actually loaded.
  8. If any node fails after its retries, the workflow's configured error workflow starts from its Error Trigger. It receives the workflow name, execution ID, failing node and error message, marks the run failed in the run log and alerts Slack with the execution link. The watermark stays where it was, so the next run or a manual rerun picks up the same window.

Tech stack

  • n8n self-hosted, queue mode with Redis and workers
  • Schedule Trigger node
  • HTTP Request node with pagination and Retry On Fail
  • Code node (JavaScript)
  • IF and Loop Over Items nodes
  • Postgres node (warehouse, watermark, run log)
  • Slack node for reports and alerts
  • Error Trigger workflow set in workflow settings

Key design decisions

  • Incremental load by watermark with an overlap window rather than full reloads. This keeps nightly volume small, and the upsert makes the overlap harmless.
  • Upsert on the source's natural key with a newer-than check, instead of plain inserts. Retries and reruns then cannot create duplicates or overwrite fresher data with stale data.
  • Advance the watermark as the last step and only on success. A partial failure then never skips records.
  • Quarantine invalid rows instead of failing the run. Data quality issues become visible in the report without blocking the good rows.
  • For large volumes, move the page-fetch and load loop into an Execute Workflow sub-workflow per page range. This bounds memory per execution, and queue-mode workers share the load.

Common mistakes

  • Advancing the cursor to 'now' instead of to the maximum timestamp actually loaded. Records updated during the run are silently skipped.
  • Holding every page in memory in one execution. Large syncs can exhaust the n8n process memory, so batch the work or split it into sub-workflows.
  • Forgetting to set the error workflow in workflow settings. The Error Trigger never fires, and failures only show up in the executions list.
  • Leaving the Schedule Trigger on the server's default timezone. 'Nightly' can drift into business hours or collide with source maintenance windows.
  • Retrying non-idempotent writes or 4xx validation errors. Retry On Fail suits transient errors; permanent errors should fail fast to the error workflow.

What to measure

  • Rows fetched, inserted, updated and rejected per run
  • Run duration and pages fetched per run
  • Data freshness: warehouse max(updated_at) versus source
  • Reject ratio per run and top rejection reasons
  • Failed runs and time from failure alert to successful rerun

Most-asked interview questions

Subjective questions 33

1How do you make this nightly ETL safe to rerun after a failure?Asked almost every time

I make every write idempotent and keep the cursor honest. Rows are upserted with INSERT … ON CONFLICT on the source's natural key, updating only when the incoming updated_at is newer. Loading the same window twice therefore changes nothing. The watermark is read at the start but advanced only as the final step, after all batches succeed, and only to the highest timestamp actually loaded. If the run dies midway, the cursor hasn't moved, so the next scheduled run or a manual rerun from the Error Trigger alert reprocesses the same window safely. I also subtract a small overlap from the cursor to catch late-arriving records. The upsert absorbs the duplicates that overlap creates.

2Walk me through the architecture of this nightly ETL workflow from trigger to report.Asked almost every time

A Schedule Trigger fires nightly in the timezone set in workflow settings. A Postgres node reads the last successful watermark and subtracts a small overlap window. An HTTP Request node pulls changed records using built-in pagination, a request interval for rate limits, and Retry On Fail for transient errors. A JavaScript Code node flattens the pages, maps and casts fields, normalises timestamps to UTC, and adds a run_id and payload hash. An IF node sends invalid rows to a quarantine table. Loop Over Items batches the valid rows, and a Postgres node upserts each batch with ON CONFLICT. Counts go to a run log keyed by execution ID. Only after every batch succeeds do we post the Slack report and advance the watermark. Any failure starts the Error Trigger workflow.

3Why subtract an overlap window from the watermark, and how would you choose its size?Asked almost every time

Source systems often commit records with an updated_at slightly earlier than the moment they become visible to the API. Examples are long transactions, replication lag, or clock differences. If we query strictly after the last watermark, those records fall into a gap and are lost forever. The overlap re-reads a small slice of time we already loaded. That costs nothing for correctness, because the upsert is idempotent and only updates when the incoming record is newer. I would size the window from observed lag: measure how late records actually arrive, then add margin. A wider window is safer but re-fetches more rows each night. If lateness can be unbounded, the overlap alone is not enough, and I would add a periodic reconciliation run.

4How do you handle records that are deleted in the source system?Asked almost every time

An updated_at incremental pull never sees hard deletes, because the row simply stops appearing. The best option is a source that exposes soft deletes, such as an is_deleted flag or a deleted_at timestamp that also bumps updated_at. The upsert then carries that flag into the warehouse. Some APIs offer a deleted-records or events endpoint, and we can pull it with its own cursor. If neither exists, I would schedule a periodic reconciliation. It fetches all primary keys from the source and compares them with the warehouse, then marks the missing ones deleted rather than removing them. That keeps history and makes the operation reversible. I would tell the interviewer this explicitly, because silent drift from deletes is a common gap in incremental ETL.

5This pipeline claims to never lose or duplicate rows. Is that exactly-once delivery?Asked almost every time

Not strictly. Fetching is at-least-once: the overlap window and reruns deliberately re-read records. What we achieve is effectively-once results in the warehouse. Each record lands exactly once, because the load is an idempotent upsert on the natural key with a newer-only condition. No loss comes from never advancing the watermark until the load succeeds, plus the overlap window for late arrivals. The guarantees have limits. Hard deletes are not seen without soft-delete flags or reconciliation. Records arriving later than the overlap window can be missed. A wrong natural key would merge distinct records. Being honest about these limits is usually what the interviewer wants.

6How would you handle errors in this workflow, from a flaky API to a bad record?Very often asked

I sort errors into three kinds. Transient failures, such as timeouts, 429s and 5xx responses, are handled by Retry On Fail on the HTTP Request node with a wait between tries and a request interval in pagination. Data problems, such as missing keys or bad dates, are not failures: an IF node routes those rows to a quarantine table with a reason, and the report counts them. Anything that still fails, such as expired credentials, a schema change or retries exhausted, ends the execution. The error workflow's Error Trigger then receives the execution ID, failing node and message, marks the run failed in the run log and alerts Slack with a link. Because the watermark didn't move, recovery is just a rerun.

7What changes when the data volume grows from thousands to millions of rows a night?Very often asked

Memory and execution time become the limits, so I stop treating the run as one big list. Loop Over Items keeps writes in fixed batches, and I push page fetching and loading into an Execute Workflow sub-workflow that handles one page range at a time. Each execution then holds a bounded amount of data. I run n8n in queue mode with Redis so workers can execute those sub-workflows in parallel, within the source API's rate limits. On the warehouse side I batch upserts, index the conflict key and consider loading into a staging table and merging. If volume keeps growing, I'd ask whether the source offers bulk export or change data capture rather than paging a REST API.

8Why is the watermark set to the highest source timestamp actually loaded rather than the time the run started?Very often asked

The run start time comes from the n8n server clock, while the cursor filters on the source system's timestamps. The two clocks can differ. A record stamped before our start time could also become visible after we fetched its page. Using now() could therefore skip records. The highest updated_at among rows we actually wrote is a fact about data we know is in the warehouse. If the API returned nothing, the watermark stays put. If some rows were quarantined, I would think about whether their timestamps should count. Usually they should, because quarantine holds them with a reason and a run_id. Combined with the overlap window, this keeps the cursor honest and only moving forward.

9Design the watermark table. What columns would you include and why?Very often asked

I would key it by pipeline name, so one table serves several syncs. Its columns would be: pipeline_name as the primary key; cursor_value, the timestamp or opaque API cursor; cursor_tiebreak_id, for composite cursors; last_success_execution_id; last_success_at; and updated_at. The advancing query should be conditional, for example updating only where the new value is greater than the stored one. That way a delayed or duplicate run can never move the cursor backwards. Storing the execution ID links the watermark to the run log and the n8n execution, which helps auditing. Some teams also keep a history table of every watermark change. That makes it easy to see exactly which windows were loaded and to roll back during an incident.

10How would you design the quarantine table and the process for reprocessing rejected rows?Very often asked

The quarantine table stores the raw source payload as JSONB, the source primary key if present, the rejection reason, the run_id, the execution ID, and timestamps for when the row was quarantined and when it was resolved. Storing the raw payload matters, because fixing a mapping bug later should not require re-fetching from the API. To reprocess, a separate manual or scheduled workflow reads unresolved rows. It runs them through the same transform and validation logic, ideally shared as a sub-workflow, and upserts the ones that now pass before marking them resolved. Because the main load is idempotent and only accepts newer records, replaying an old quarantined row cannot overwrite a fresher version already in the warehouse.

11What is the payload hash for, and how would you use it in the upsert?Very often asked

The hash is computed over the business fields after the transform, in a stable order, so identical content always gives the same value. It has two main uses. First, change detection: the ON CONFLICT update can require both a newer timestamp and a different hash. That skips no-op updates, which reduces write load and stops audit columns from churning when the overlap window re-reads unchanged rows. Second, diagnostics: when inserted and updated counts look odd, comparing hashes shows whether content really changed. It also helps reconciliation against the source. Exclude volatile fields like run_id from the hash, or every rerun will look like a change.

12How do you prevent two executions of this workflow from running at the same time, and why does it matter?Very often asked

A manual rerun could overlap with the scheduled run, or a slow night could still be running when someone retries. Overlapping runs duplicate API load and make run logs confusing. In the worst case, a slower run could try to move the watermark after a faster one. I would take a Postgres advisory lock, or claim a lock row for the pipeline, at the start of the workflow, and exit cleanly if it is already held. The lock must be released on both success and failure, so I would either give it a timeout or have the error workflow clear it. The conditional watermark update, which only moves forward, and the newer-only upsert are the safety net: even if two runs overlap, they cannot corrupt data.

13What pagination pitfalls would you watch for when pulling changed records?Very often asked

Offset pagination is the main risk. If records are updated while you page, rows shift between pages, so some get skipped and others duplicated. Cursor-based pagination, or keyset pagination on a stable sort such as updated_at then id, avoids that. Make sure the API sorts results deterministically. Ties on updated_at need a tiebreaker, or a page boundary can split records with the same timestamp. Configure the HTTP Request node's 'complete when' condition carefully, for example when the next cursor is empty or the response has no items. A wrong condition either stops early and loses data or loops forever. I would also set a maximum-requests guard, and log the page count to the run log.

14A batch in the middle of the load fails. What state is the system in, and what happens next?Very often asked

Earlier batches are already committed, because each upsert statement is atomic for its batch. The failed batch either applied fully or not at all, depending on the Postgres node's query batching mode. Running it as a single query or inside a transaction keeps it atomic. The watermark has not moved, because that only happens after all batches succeed. The error workflow marks the run failed in the run log, records the failing node and message, and posts a Slack alert with the execution link. The next scheduled run or a manual rerun fetches the same window again. Already-loaded rows hit ON CONFLICT and are either skipped or updated only if they are newer, so nothing is duplicated and nothing is lost.

15How do you handle many records sharing the same updated_at value at a page or window boundary?Very often asked

If the cursor is only a timestamp and we query with greater-than, records with the same timestamp as the watermark that were not loaded yet get skipped. If we use greater-than-or-equal, we re-read them, which is safe because the upsert is idempotent. The overlap window already makes re-reads normal, so greater-than-or-equal plus the overlap is the simple, safe choice. For very high-volume sources, a composite cursor of (updated_at, id), with the API sorting by both, gives precise resumption without big overlaps. The watermark table can store both values. Whatever you choose, the API's sort order must match the cursor definition, or the boundary logic breaks.

16Should a sudden spike in rejected rows fail the run, even though quarantine exists to prevent that?Very often asked

Quarantine exists so one bad record does not block the night. But if 40 percent of rows fail validation, the likely cause is a systemic problem such as an API change, an auth issue returning error bodies, or a mapping bug, not bad data. Loading the remainder and advancing the watermark would then hide a big data-quality incident. I would add a check after validation: if rejects exceed a configured percentage or count, throw an error. That triggers the error workflow and keeps the watermark in place, so a rerun covers the window after the fix. The threshold should come from normal historical reject rates in the run log, not a guess.

17What security measures would you apply to this workflow?Very often asked

The source API key and Postgres password live in n8n's encrypted credentials and are referenced by nodes, never hard-coded in Code nodes or expressions. The Postgres credential uses a least-privilege role: SELECT and UPDATE on the watermark table, INSERT and UPDATE on the target tables, the run log and quarantine, and no DDL or superuser rights. Use TLS for both the API and the database. Restrict who can edit the workflow and view executions in n8n, because execution data contains real records. Protect the instance's encryption key, and rotate credentials on a schedule. On self-hosted deployments, keep n8n patched and do not expose its editor publicly without authentication.

18Execution logs and Slack reports could expose personal data. How do you manage privacy?Very often asked

n8n can save execution data, and for this workflow that means real customer records sitting in its database. I would configure execution data saving so successful runs are not stored, or are pruned quickly, while failed runs are kept for a limited period for debugging. Access to executions should be limited to the team that owns the pipeline. The Slack report should carry only aggregates: counts, durations, window bounds, and the execution link. It should never include record contents or personal fields. Error messages can echo payloads, so the error workflow should truncate or sanitise them before posting. Quarantine holds raw payloads, so it needs the same access controls and retention policy as the warehouse table it feeds.

19How would you detect a night when the workflow silently did not run at all?Very often asked

The error workflow only fires when an execution fails. If the instance was down, the workflow got deactivated, or the schedule never fired, nothing alerts. I would add a separate dead-man's-switch check. A small workflow, ideally running somewhere independent, queries the run log or watermark table each morning. It alerts if there is no successful run within the expected window or if the watermark has not advanced in too long. An external uptime or heartbeat monitor works too: the main workflow pings it on success, and the monitor alerts when pings stop. Freshness checks on the warehouse tables themselves are also useful for downstream consumers.

20What would your run log table look like, and how is it used?Often asked

I would key it on the n8n execution ID, with pipeline name, run_id, status (running, succeeded or failed), started_at, finished_at, the window start and end, and counts for fetched, inserted, updated and rejected rows. It can also hold the failing node and error message, which the error workflow fills in. Each batch increments the counts, so a mid-run failure still shows how far the run got. The table drives the Slack report and gives operators history to compare nights. It also supports a freshness check: if no successful row appears within the expected window, something is wrong even though no error fired. Keying by execution ID lets you jump straight to the execution in n8n's log.

21How would you tune the request interval and Retry On Fail settings for the source API?Often asked

Start from the API's documented rate limit and set the pagination request interval so we stay comfortably below it, leaving headroom for other consumers of the same key. Retry On Fail should be enabled on the HTTP Request node with a few tries and a wait between them, so a brief 429 or 5xx is absorbed without failing the run. Respect Retry-After headers where the API provides them. Do not retry permanent errors such as 400, 401 or 404 blindly, because they will just fail again and waste time. If 429s still appear, increase the interval or reduce page concurrency rather than adding more retries. Retries also lengthen runtime, so check the run still fits its nightly window.

22How would you estimate how long a nightly run takes?Often asked

I would build a simple model from measured inputs rather than guesses. Fetch time is roughly the number of changed records divided by page size, multiplied by the request interval plus average response time. As an illustration: if 200,000 records change, pages hold 100, and each request plus interval takes about one second, that is about 2,000 requests, or around 33 minutes. Add transform time, which is usually small. Add load time, which is the number of batches multiplied by the measured upsert time per batch. Then add margin for retries and the overlap window. I would compare the total with the nightly maintenance window and with the source's rate limit, and record actual durations in the run log to refine the model.

23Why use a Code node for the transform instead of Set or Edit Fields nodes?Often asked

The transform does several things at once: it flattens nested pages into one item per record, maps many fields, casts types, normalises timestamps to UTC, and computes a hash. In JavaScript that is compact, testable logic in one place. The same work spread across many Set nodes and expressions is harder to review and change. A Code node also lets us use deterministic hashing and consistent date parsing. The trade-offs are that it is less visual for non-developers and that bugs in custom code are our responsibility. For simple renames, Edit Fields is fine. I would keep the Code node focused, avoid side effects, and cover it with test fixtures.

24How do you deal with time zones in this workflow?Often asked

There are two separate concerns. The Schedule Trigger fires according to the timezone in workflow settings, so set it explicitly. That keeps 'nightly' from drifting across daylight-saving changes or server defaults. Data timestamps are a different matter. The Code node parses every source timestamp, including any offset, and normalises it to UTC before loading into timestamptz columns. The watermark is stored in UTC too. Mixing a local-time watermark with UTC data is a classic way to skip or re-read hours of records. Reject unparseable or offset-less dates to quarantine rather than guessing, unless the source documents a fixed zone. The run report can show local times for humans, but the storage and comparison layers stay in UTC.

25The source API adds or renames a field. How should the workflow respond?Often asked

New fields are harmless if the Code node maps fields explicitly instead of passing everything through. They are simply ignored until we decide to add a warehouse column. A renamed or removed field is more dangerous, because the mapping produces nulls. Validation should catch required fields going missing, and the rejects land in quarantine with a clear reason. I would also set a threshold. If the reject rate spikes far above normal, fail the run deliberately so the watermark does not advance and a human looks at it. Keeping the raw payload in quarantine means we can fix the mapping and replay. Schema changes on the warehouse side should go through migrations, not ad hoc edits.

26How would you test this workflow before trusting it in production?Often asked

First, unit-test the transform by pinning sample API responses on the HTTP node and running the Code and IF nodes against fixtures. Those fixtures should cover nulls, bad dates, odd time zones, duplicate keys and nested pages. Then run the whole flow against a staging schema with a non-production API key. Run it twice over the same window and confirm the row counts and hashes are identical, which proves idempotency. Kill it mid-load and rerun it to prove recovery. Note that error workflows do not fire on manual test executions, so test the Error Trigger path with an active scheduled or triggered run that fails on purpose. Finally, reconcile counts against the source for a known window.

27How would you run a historical backfill with this same workflow?Often asked

I would not edit the production watermark by hand. Instead, I would parameterise the window: a manual or sub-workflow entry point accepts a start and end, and the scheduled path reads them from the watermark table. The backfill runs the same fetch, transform, validate and upsert steps. Idempotent upserts mean it can safely overlap with data already loaded. I would chunk a long backfill into smaller date ranges so each execution stays small, and throttle it to respect the API's rate limits. It should also avoid running at the same time as the nightly job. The backfill should log to the run log with its own marker, and it should not advance the nightly watermark unless that is intended.

28When would you move this n8n instance to queue mode?Often asked

In regular mode, one n8n process handles the editor, triggers and executions. A heavy nightly ETL can compete with other workflows for memory and CPU on that process. Queue mode uses Redis as a broker, with separate worker processes running executions, so heavy jobs are isolated and you can scale workers horizontally. I would move when several heavy workflows overlap, when executions queue behind each other, or when a single process becomes a reliability risk. Queue mode does not make one execution faster or lighter on memory. A single huge execution still needs splitting into sub-workflows. It also adds operational parts, Redis and the workers, that need monitoring.

29How could sub-workflows with Execute Workflow improve this design?Often asked

Holding every page and record in a single execution keeps all those items in memory and makes the execution log huge. Instead, the parent workflow could compute the window, then call a sub-workflow per date slice or per group of pages. Each child fetches, transforms and upserts its slice and returns only counts. The parent aggregates the counts, posts the report and advances the watermark only if every child succeeded. Children release memory when they finish, failures are scoped to one slice, and the transform logic can be reused by the quarantine reprocessing and backfill workflows. The trade-off is more moving parts, plus care that a child failure propagates to the parent so the watermark does not move.

30If the Slack post fails, the watermark does not advance. Is that the right design?Often asked

As written, yes: the report and the watermark are in sequence, so a Slack outage fails the run and keeps the cursor in place. That is safe, because the next run re-reads the window idempotently. But it is wasteful, and it creates a false 'failed' signal even though the data loaded fine. A reasonable alternative is to advance the watermark immediately after the last batch succeeds, then post the report with Continue On Fail or a short retry, so a notification failure cannot block progress. The key invariant is that the watermark moves only after a successful load. Notification delivery is secondary. I would discuss this trade-off openly with the interviewer.

31How do you choose between an updated_at timestamp cursor and an opaque API cursor?Often asked

A timestamp cursor is transparent: we can subtract an overlap, inspect it and set it manually for backfills. It depends on the source updating the timestamp reliably and on consistent sorting. An opaque cursor or change token from the API can be more precise and can include deletes, but it cannot be adjusted with an overlap window. It may also expire if we do not use it within a retention period, which forces a full resync. If the API offers a change feed with a cursor, I would use it and store the token only after a successful load. Otherwise I would use updated_at with a tiebreaker. Either way, the watermark table stores whichever value we resume from.

32What would you add to make the warehouse data trustworthy for downstream users beyond 'the job succeeded'?Often asked

Success only means nothing crashed. I would add data-quality checks after the load. These include row-count reconciliation against the source for the window where the API exposes counts, checks that no natural keys are duplicated, null-rate checks on critical columns, and a freshness check on max(updated_at). Persist the results in the run log and include a summary in the Slack report. For important tables, a periodic full-key reconciliation catches deletes and late records that the incremental path misses. Document the table's freshness expectations, for example that it reflects source data up to the last watermark, so analysts know what 'current' means.

33How would you pick the batch size for the Loop Over Items upsert?Sometimes asked

Batch size trades round trips against statement size and lock duration. Very small batches mean many queries and a slow load. Very large batches mean long transactions, more memory in n8n, and bigger rollbacks when one row hits a constraint error. I would start with a moderate size and measure the upsert time per batch in the run log. Then I would adjust while watching Postgres for lock waits and the n8n worker for memory. Also consider the parameter limits on a single statement when many columns are inserted per row. The right number depends on row width and indexes on the target table, so it should be configurable rather than hard-coded.

Objective questions 33

1In n8n, what runs automatically when a production workflow execution fails and you want to alert the team?Asked almost every time
  1. AA Wait node placed after every HTTP Request node
  2. BA workflow starting with an Error Trigger node, set as the error workflow in the failing workflow's settingsCorrect answer
  3. CA second Schedule Trigger that polls the executions list every minute
  4. DThe Retry On Fail setting, which emails the owner after the last retry

Why: n8n starts the configured error workflow, which begins with an Error Trigger, whenever an execution fails. The Error Trigger receives details such as the workflow, execution ID and error. Retry On Fail only retries the node; it does not alert anyone.

2Which node in this workflow follows next-page cursors or links using built-in pagination?Asked almost every time
  1. AWait node
  2. BHTTP Request nodeCorrect answer
  3. CLoop Over Items node
  4. DMerge node

Why: The HTTP Request node has built-in pagination options that follow next-page cursors or links until a 'complete when' condition is met.

3Which Postgres statement makes the batch load idempotent while only applying newer records?Asked almost every time
  1. AINSERT … VALUES with no conflict clause
  2. BDELETE all rows then INSERT
  3. CINSERT … ON CONFLICT (natural_key) DO UPDATE … WHERE excluded.updated_at > target.updated_atCorrect answer
  4. DTRUNCATE then COPY

Why: ON CONFLICT on the natural key turns duplicates into updates, and the WHERE condition prevents older data from overwriting newer rows.

4Why does the workflow subtract a small overlap window from the stored watermark?Asked almost every time
  1. ATo reduce API calls
  2. BTo catch late-arriving records that were committed with earlier timestampsCorrect answer
  3. CTo make the Schedule Trigger fire earlier
  4. DTo skip records already in quarantine

Why: The overlap re-reads a slice of time so records that became visible late are not missed. The idempotent upsert makes the re-read harmless.

5What is the main risk of advancing the watermark before the load finishes?Asked almost every time
  1. AHigher API rate limits
  2. BDuplicate rows in the warehouse
  3. CPermanent data loss if the load then failsCorrect answer
  4. DSlower Slack messages

Why: If the cursor moves first and the load fails, the next run starts after records that were never loaded, so they are skipped.

6What delivery semantics does this pipeline actually provide?Asked almost every time
  1. AAt-least-once fetching made effectively-once by idempotent upsertsCorrect answer
  2. BBest-effort with possible duplicates
  3. CExactly-once fetching from the API
  4. DAt-most-once fetching with no retries

Why: The overlap and reruns re-read data, which is at-least-once. Upserts on the natural key make the stored result effectively-once.

7Why does the load step upsert on the natural key instead of doing plain inserts?Very often asked
  1. AUpserts make reruns and the overlap window idempotent, so no duplicate rows are createdCorrect answer
  2. Bn8n's Postgres node cannot perform plain inserts
  3. CUpserts remove the need for a watermark
  4. DUpserts are always faster than inserts in Postgres

Why: Retries, reruns and the cursor overlap all reprocess some records. An upsert keyed on the natural key turns those repeats into no-ops or updates instead of duplicates.

8When should the workflow advance the stored watermark cursor?Very often asked
  1. AAfter each page is fetched from the API
  2. BAfter all batches are upserted successfully, set to the highest timestamp actually loadedCorrect answer
  3. CInside the Error Trigger workflow when a run fails
  4. DAt the start of the run, before fetching any pages

Why: Moving the cursor only after a fully successful load, and only to what was loaded, means a partial failure never skips records. The next run simply reprocesses the same window.

9Where do you tell n8n which error workflow to run when this ETL workflow fails?Very often asked
  1. AIn the Schedule Trigger's parameters
  2. BIn the HTTP Request node's options
  3. CIn the ETL workflow's settingsCorrect answer
  4. DIn the Postgres credential

Why: The error workflow is selected in the failing workflow's settings. That error workflow starts from an Error Trigger node.

10Which information does the Error Trigger give the error workflow in this design?Very often asked
  1. AThe source API credential in plain text
  2. BOnly a boolean failure flag
  3. CThe full warehouse table contents
  4. DThe workflow name, execution ID, failing node and error messageCorrect answer

Why: The error workflow uses the workflow name, execution ID, last failing node and error message to mark the run failed and alert Slack with the execution link.

11What must exist in Postgres for INSERT … ON CONFLICT (natural_key) to work?Very often asked
  1. AA sequence on natural_key
  2. BA trigger on the table
  3. CA unique constraint or unique index on the conflict columnsCorrect answer
  4. DA foreign key to the watermark table

Why: Postgres needs a unique constraint or unique index matching the conflict target to detect conflicts.

12The watermark is advanced to which value?Very often asked
  1. AThe highest source timestamp actually loadedCorrect answer
  2. BThe time the Slack report was posted
  3. CThe n8n server's current time
  4. DThe time the Schedule Trigger fired

Why: Using the highest loaded source timestamp avoids clock skew between systems and never claims data that was not written.

13Which errors should Retry On Fail on the HTTP Request node mainly absorb here?Very often asked
  1. ATransient 429 and 5xx errorsCorrect answer
  2. BValidation failures in the Code node
  3. C400 Bad Request
  4. D401 Unauthorized

Why: Rate limits and server errors are usually temporary, so waiting and retrying helps. Client errors like 400 or 401 will fail again.

14What does the Loop Over Items node do in this workflow?Very often asked
  1. ASplits valid rows into fixed-size batches for upsertingCorrect answer
  2. BWaits for human approval
  3. CMerges valid and rejected rows
  4. DPaginates the source API

Why: Loop Over Items processes items in batches, so each Postgres upsert handles a fixed-size chunk.

15Why are invalid rows routed to a quarantine table instead of throwing an error?Very often asked
  1. ABecause Postgres cannot store invalid rows
  2. BTo avoid writing a run log
  3. CBecause the IF node cannot throw errors
  4. DSo one bad record does not fail the whole night's loadCorrect answer

Why: Quarantine records the reject with its reason and run_id so valid rows still load and bad ones can be fixed and replayed.

16Why is offset-based pagination risky for pulling changed records?Very often asked
  1. AIt disables rate limiting
  2. BIt always returns empty pages
  3. CRows can shift between pages as data changes, causing skips or duplicatesCorrect answer
  4. DIt cannot be used with credentials

Why: When records are updated during paging, offsets move, so some rows are skipped and others appear twice. Cursor or keyset pagination is stable.

17Batch 7 of 20 fails after retries. Which statement is true?Very often asked
  1. AThe watermark advances to the end of batch 6
  2. BBatches 1–6 stay committed, the watermark is unchanged, and a rerun is safeCorrect answer
  3. CThe rerun will duplicate batches 1–6
  4. DAll previous batches are rolled back automatically

Why: Each batch commits independently and the cursor only moves after all batches succeed. Idempotent upserts make reprocessing batches 1–6 harmless.

18Does an n8n error workflow fire when you run the main workflow manually in the editor and it fails?Very often asked
  1. AOnly in queue mode
  2. BYes, always
  3. CNo, error workflows run for automatic executions, not manual test runsCorrect answer
  4. DOnly if the Error Trigger is pinned

Why: n8n triggers error workflows for failed production (automatic) executions, so testing the alert path needs an active trigger that fails on purpose.

19An incremental pull on updated_at cannot see which type of change without extra work?Very often asked
  1. AType changes in a column
  2. BNew inserts
  3. CField updates
  4. DHard deletes in the sourceCorrect answer

Why: A hard-deleted row simply stops appearing. You need soft-delete flags, a deletes endpoint or periodic key reconciliation.

20What is the best practice for the Postgres credential used by this workflow?Very often asked
  1. AA credential embedded in the Code node
  2. BA superuser so migrations can run
  3. CA least-privilege role limited to the needed tables and operationsCorrect answer
  4. DThe same account used by analysts

Why: Least privilege limits damage from mistakes or a compromised instance. Credentials belong in n8n's encrypted credential store.

21In a Postgres ON CONFLICT DO UPDATE clause, what does EXCLUDED refer to?Often asked
  1. AThe previous version of the watermark
  2. BRows deleted earlier in the run
  3. CThe row that was proposed for insertionCorrect answer
  4. DRows rejected to quarantine

Why: EXCLUDED is the special table holding the incoming row that conflicted, so its values can be compared with or written to the existing row.

22Which setting determines the timezone the nightly Schedule Trigger uses?Often asked
  1. AThe Postgres server timezone
  2. BThe timezone set in workflow settingsCorrect answer
  3. CThe source API's timezone
  4. DUTC always, with no option to change it

Why: Per the design, the Schedule Trigger fires using the timezone configured in the workflow's settings.

23What is a sensible use of the per-row payload hash?Often asked
  1. ASkipping no-op updates when content has not changedCorrect answer
  2. BReplacing the natural key
  3. CAuthenticating to the source API
  4. DEncrypting the row at rest

Why: Comparing hashes in the upsert condition avoids rewriting identical rows that the overlap window re-reads.

24Many records share the same updated_at at the watermark boundary. Which cursor design resumes precisely?Often asked
  1. AA random sample of IDs
  2. BPage number only
  3. CA composite cursor of (updated_at, id) with matching sort orderCorrect answer
  4. DThe n8n execution ID

Why: Adding a unique tiebreaker to the timestamp gives a strict ordering, so resuming neither skips nor endlessly repeats tied records.

25Why does the run log use the n8n execution ID as its key?Often asked
  1. AIt encrypts the run data
  2. BIt is required by the Postgres node
  3. CIt links counts and status directly to the logged n8n execution for debuggingCorrect answer
  4. DIt replaces the natural key

Why: Keying by execution ID lets operators move from a run log row or Slack alert straight to the n8n execution that produced it.

26Why does the Code node normalise all timestamps to UTC?Often asked
  1. APostgres cannot store other zones
  2. BSo comparisons with the watermark are consistent and not distorted by offsets or daylight savingCorrect answer
  3. CTo shorten the payload hash
  4. DBecause Slack requires UTC

Why: Mixing zones causes the watermark comparison to skip or re-read hours of data. UTC gives a single consistent timeline.

27How should the source API key be supplied to the HTTP Request node?Often asked
  1. AHard-coded in a Code node
  2. BThrough an n8n credential stored encrypted and referenced by the nodeCorrect answer
  3. CIn the Slack report
  4. DAs a query parameter in the workflow name

Why: n8n stores credentials encrypted and nodes reference them, which keeps secrets out of workflow JSON and execution logic.

28What does running n8n in queue mode require?Often asked
  1. AKafka
  2. BRedis as a message broker plus worker processesCorrect answer
  3. CA second Postgres warehouse
  4. Dn8n Cloud only

Why: Queue mode uses Redis to distribute executions to worker processes, which scales execution throughput.

29A single execution holding millions of items is running out of memory. Which change helps most directly?Often asked
  1. ASplit the work into date or page slices processed by sub-workflows via Execute WorkflowCorrect answer
  2. BPost more Slack messages
  3. CRemove the IF node
  4. DIncrease Retry On Fail attempts

Why: Sub-workflows process smaller slices and release memory when they finish. Queue mode alone does not shrink a single execution.

30How can you detect a night when the workflow never ran at all?Often asked
  1. ARetry On Fail on the Schedule Trigger
  2. BThe Slack node's delivery receipt
  3. CThe Error Trigger always fires for missed schedules
  4. DA separate freshness or heartbeat check on the run log or watermarkCorrect answer

Why: No execution means no error, so an independent dead-man's-switch check is needed to alert when successful runs stop.

31What is a safe way to stop a manual rerun from overlapping with the scheduled run?Often asked
  1. ATake a Postgres advisory lock or lock row at the start and exit if it is heldCorrect answer
  2. BDelete the watermark before each run
  3. CRun without the error workflow
  4. DDisable the upsert

Why: A lock ensures only one run proceeds. The forward-only watermark update and idempotent upsert remain a safety net.

32What should the nightly Slack run report contain to respect privacy?Often asked
  1. AAggregate counts, window bounds, status and the execution linkCorrect answer
  2. BThe database password for quick fixes
  3. CRaw API responses
  4. DFull records that failed validation

Why: Aggregates give operators what they need without leaking personal data into a chat channel.

33Which languages can the n8n Code node use for the transform step?Sometimes asked
  1. ABash only
  2. BOnly SQL
  3. CJavaScript or PythonCorrect answer
  4. DOnly Go

Why: The Code node supports JavaScript and Python. This design uses JavaScript.

Track your progress through every blueprint in your dashboard.

Start learning