Company docs chatbot with the AI Agent node

I attack it at three layers. First, retrieval quality: sensible chunk sizes with overlap, the same embeddings model for ingest and query, and metadata so the agent sees titles and dates. Second, the prompt and tool setup: the system prompt says to answer only from retrieved chunks, cite the source link, and reply that it does not know when nothing relevant comes back, and the vector store tool description tells the agent to use it for any company question. Third, evaluation: I keep a test set of real employee questions with expected sources and rerun it whenever I change chunking, the model or the prompt. In production I track the share of answers with citations and review thumbs-down executions in the n8n execution log.

There are two workflows. The chat workflow starts at an authenticated Chat Trigger that emits chatInput and a sessionId. An AI Agent node receives the message with a strict grounding system prompt. It has three sub-nodes. Postgres Chat Memory loads recent turns for that session. A vector store in retrieve-as-tool mode returns top-k chunks with title, URL and last-updated metadata. An OpenAI-compatible chat model points at Vercel AI Gateway. The agent replies with cited sources, memory saves the turn, and n8n logs the execution. The ingestion workflow runs on a Schedule Trigger. It fetches changed and deleted documents, chunks and embeds them, and upserts by document ID. An Error Trigger workflow alerts Slack when either workflow fails.

Memory is keyed strictly by session, so the session key is the main control. First, authentication on the Chat Trigger means only staff can reach the chat. Second, I don't trust a client-supplied sessionId on its own. I build the memory key from the authenticated user identity plus the session ID, so a guessed or replayed ID cannot load someone else's turns. Third, Postgres Chat Memory scopes reads and writes to that key, and no shared global key is ever used. Finally, I test it. I open two users in parallel, check that follow-ups stay isolated, and review the memory table to confirm every row has exactly one owner.

Every chunk carries the source document ID in its metadata. When a changed document comes in, the workflow first deletes all existing chunks with that document ID. It then inserts the freshly split and embedded chunks through the vector store node in insert mode. That turns ingestion into an upsert per document. An edit that shortens a document leaves no orphaned old chunks behind, and re-running the same batch produces the same end state. Without the delete step, the old and new versions would both be retrievable, and the agent could cite an outdated rule as current policy.

The two jobs have very different shapes. Chat needs low latency per message, while ingestion is batch work: listing changed files, parsing, chunking, embedding and upserting, which can take minutes and hit rate limits. Keeping them apart means a slow or failing ingestion run never blocks an employee's answer, and I can give ingestion its own retries, Error Trigger alerts and schedule. It also makes incremental updates clean: the scheduled run tracks what changed since last time, deletes old chunks per document ID and inserts new ones. The chat workflow only ever reads from the vector store, so the boundary between them is just the index, which is easy to reason about and test.

I would never rely on the model to hide content it has already been given. At ingestion I tag every chunk with an access group taken from the source system's permissions, or I keep separate indexes for sensitive collections. On the chat side the Chat Trigger has authentication, so I know who the user is, and I map them to their groups. Retrieval then filters by that metadata before chunks reach the agent, so restricted text is never in the prompt. Memory is keyed per user session for the same reason. I also re-sync permissions on the ingestion schedule, because access changes as often as content does, and I audit with test accounts from different groups.

Vectors from different embedding models live in different spaces. They can have different dimensions and, even when dimensions match, different geometry. Similarity search only works when the query vector and the chunk vectors come from the same model. With a mismatch, the search either errors on dimension or quietly returns irrelevant chunks. The quiet case is worse, because the agent then says it doesn't know, or cites the wrong document. So I keep the embeddings model in a single shared setting for both workflows and record the model name in chunk metadata or the index name. That makes a mismatch easy to detect.

I start from how employees ask questions. Most questions target a single rule or paragraph. Chunks need to be large enough to hold a complete policy statement with its context, and small enough that top-k results don't flood the prompt with unrelated text. Overlap keeps a sentence that crosses a boundary retrievable from either side. The recursive splitter tries paragraph breaks first, then sentences, which keeps sections intact. I don't guess final numbers. I build a golden set of real questions, try a few size and overlap settings, and keep the one where the correct chunk shows up in top-k most often without bloating token use.

The scheduled workflow records deletions as well as changes, whether it reads them from Google Drive change data or from the wiki API. For each deleted document ID, it removes every chunk with that ID from the vector store. This matters for both correctness and access. A retired policy must not keep answering questions, and a document pulled for confidentiality reasons must disappear from retrieval. If the source API cannot report deletions, I run a periodic reconciliation. It lists all current IDs, compares them with the IDs in the index, and deletes the difference.

It keeps a cursor, either a last-successful-run timestamp or a change token, in a durable store such as a Postgres table. Each run reads the cursor and asks Google Drive or the wiki API for items modified after it. The cursor only advances after every document in the batch has been processed successfully. If a run fails halfway, the next run picks up the same window again. Because ingestion is delete-then-insert by document ID, reprocessing a document is harmless. I also add a small overlap on the time window so I don't miss edits that land right on the boundary.

If insert fails after delete, that document becomes unanswerable until the next run. Usually that just means a 'don't know' rather than a wrong answer, but it is still a gap. There are three mitigations. First, embed all chunks before deleting anything, so the slow, failure-prone step finishes before the index changes. Second, write new chunks tagged with a version, then delete the older versions, so there is never a moment with zero chunks. Third, keep the cursor unadvanced on failure so the document is retried. Retry-on-fail on the embedding and vector store nodes covers transient errors, and the Error Trigger alerts on persistent ones.

The chat model node sends every request to the gateway using one AI Gateway API key. The gateway is configured with a primary model, for example anthropic/claude-sonnet-4.5, and fallback models from another provider. If the first provider errors, the gateway retries with the next one, so n8n never sees the outage. The risk is that models behave differently. A fallback model might follow the grounding prompt less strictly or format citations differently. I test the system prompt and the golden question set against every fallback model. When all fallbacks are exhausted, the error reaches the Error Trigger workflow, which posts to Slack.

One credential reaches many providers. The n8n side is just an OpenAI-compatible chat model with the gateway base URL and an AI Gateway API key, and switching models is a matter of changing the model name string. No provider keys live in n8n, which shrinks the blast radius if the instance or a workflow export leaks. The gateway also adds provider fallbacks and spend tracking in one place. The trade-offs are an extra dependency in the request path and the need to trust the gateway with prompts, so I check it against the company's data-handling policy.

I run n8n in queue mode. The main instance receives chat requests and puts executions on Redis, and multiple workers process them in parallel, so I scale by adding workers. State is kept out of worker memory. Chat history is in Postgres and embeddings are in the vector store, so any worker can serve any turn. The real bottlenecks are usually model latency and provider rate limits, which the gateway helps manage through fallbacks. Next come vector store query capacity and Postgres connections. Ingestion runs as a separate scheduled workflow, so it doesn't compete with chat during peak hours.

Retry-on-fail is the first line. It is set on the HTTP Request, embeddings and vector store nodes, so short API blips like timeouts or brief rate limits recover without any human noticing. The gateway handles provider-level failures with its own fallbacks. Only when retries and fallbacks are exhausted does the execution fail, and then the Error Trigger workflow posts the execution link to a Slack channel. Someone can open the exact run and inspect it. This layering avoids alert fatigue from transient errors while making sure persistent failures, like a broken wiki credential, get seen quickly.

I build a golden set of real employee questions with known correct source documents. It includes questions the documents cannot answer, where 'I don't know' is the right response. For each question I check three things. Did retrieval return the right document? Is the answer supported by the retrieved text? Is the citation's title and link correct? I rerun the set whenever I change the prompt, chunking, top-k, embeddings model or chat model, including fallback models. In production, I sample logged executions and collect user feedback to find gaps, such as missing documents or poorly chunked sections.

Retrieved chunks are untrusted input, even though they come from internal sources. Anyone who can edit a wiki page could write 'ignore your instructions'. The system prompt says retrieved content is reference material, not instructions, and the agent should only use it as evidence. The agent's only tool is a read-only vector search, so even a successful injection can't send email or change data. I restrict which sources are ingested to curated spaces, keep document owners accountable, and review logged executions when answers look odd. I avoid giving this agent write-capable tools for exactly this reason.

I enable authentication on the Chat Trigger so anonymous users can't reach it, whether they come through the n8n-hosted chat page or the embedded widget. For the widget, I embed it only in an internal, already authenticated page such as the intranet, and restrict which origins can call it. n8n itself sits behind company network controls or SSO where possible. Credentials for Postgres, the vector store and the gateway are stored encrypted in n8n and referenced by nodes, never pasted into Code nodes. I also give the n8n editor role only to the people who maintain the workflow.

The agent can rewrite follow-ups into better queries, call the tool more than once for multi-part questions, and skip retrieval for small talk like 'thanks'. The cost is less predictability. The model might answer without retrieving, and it may make extra model calls, which adds latency and spend. In this design I reduce that risk with a specific tool description and a prompt that requires retrieval for any company question. If logs showed the agent regularly skipping the tool, I would switch to a fixed retrieve-then-answer flow, which gives up flexibility for guaranteed grounding.

The agent decides whether and how to call a tool based mostly on its name and description. A vague description like 'search' can lead the model to answer from its own knowledge instead of retrieving. That is exactly the ungrounded behaviour we must prevent. I write something specific, such as 'Searches the company's internal policies, HR, IT and process documents. Use for any question about company rules or procedures.' Combined with the system prompt rule to answer only from retrieved documents, this makes retrieval the default path. I confirm it by checking execution logs to see that the tool is actually called on policy questions.

Top-k trades recall against noise and cost. If k is too small, the right chunk is missed and the bot says it doesn't know when the answer exists. If k is too large, the prompt grows, latency and spend go up, and loosely related chunks can mislead the model or bury the relevant one. I tune k against a golden set of real employee questions. I measure how often the correct source appears in the results and whether answer quality holds up. I also consider grouping chunks by document so several chunks from one long policy don't crowd out a second relevant source.

Postgres Chat Memory loads the last few turns for that session. When the agent sees 'what about contractors?', it also sees that the previous question was about, say, the leave policy. The agent can then write a self-contained search query, such as 'leave policy for contractors', before calling the vector store tool. Without memory, the retrieval query would be too vague to match anything. I reinforce this in the system prompt by telling the agent to rephrase follow-ups into standalone search queries. After the reply, memory saves the new turn so the next follow-up also has context.

Only the last few turns. Follow-ups rarely refer back further than that, and every extra turn costs tokens on every call, which adds latency and spend through the gateway. Long histories also cause drift. Old topics and earlier answers can bias retrieval or get repeated without being checked against current documents. I choose the window by testing typical multi-turn conversations. I also make sure older turns still sit in Postgres for audit, subject to retention rules, even though they are not loaded into the prompt.

I would state my assumptions rather than quote numbers I don't know. Daily turns equal active employees times questions per employee per day. Peak concurrency is roughly peak turns per second times average turn duration, and turn duration is dominated by model latency plus retrieval. Each turn costs one embedding call, one vector query, and one or more model calls, since the agent may call the tool and then answer. Token cost per turn is the system prompt plus memory turns plus top-k chunks plus the answer. With those variables, I size worker count, gateway spend and vector store throughput, and then validate against real execution logs.

Questions can reveal sensitive things, like health-related leave, grievances or salary concerns, so the logs are personal data. I limit who can view executions in n8n to the small team that maintains the bot. I configure execution data retention and pruning so logs are kept only as long as needed for debugging and review. I apply the same retention thinking to the Postgres memory table. I tell employees in the chat that conversations are logged for quality review. Before using logs for evaluation, I strip or mask identifiers.

Each field has a job. Title and URL let the agent cite sources, so employees can click through and verify the answer instead of trusting the bot. Last-updated lets the agent, and a reviewing human, judge freshness and resolve conflicts between old and new versions. Document ID is the key for upserts and deletions, so ingestion can remove every chunk of a changed or deleted document. Without document ID, I couldn't keep the index consistent. Without title and URL, answers couldn't be verified, which undermines the rule that policy claims must be backed by documents.

The vector store tool returns each chunk together with its title and URL metadata. The system prompt instructs the agent to cite only sources that appeared in the tool results. As a stronger guard, I can add a Code node after the agent that extracts the cited links and checks them against the URLs actually returned in that execution. Any citation not in the retrieved set is removed, or the answer is flagged. I also check citation correctness in the golden-set evaluation, because a wrong link is almost as harmful as a wrong answer.

It should not quietly pick one and present it as policy. The last-updated metadata helps. If one document is clearly a newer version of the other, the agent can prefer it and mention the date. If they genuinely disagree, such as a regional policy versus a global one, the agent should present both with citations and suggest confirming with the owning team, like HR. Persistent conflicts are also a content problem. I log them and report them to document owners so the outdated source gets retired, and the deletion flow then removes it from the index.

I find the execution in the n8n log by time and session, then walk the chain. What did memory load? A stale earlier turn may have steered the query. What query did the agent send to the vector store tool, and which chunks came back? If the right document wasn't retrieved, the problem is ingestion, chunking, top-k or embeddings. If it was retrieved but misread, the problem is the prompt or the model, including whether a gateway fallback model answered. If the source document itself is outdated, it goes to the document owner. The fix then becomes a new golden-set case.

A Postgres-based vector store, such as pgvector, keeps everything in a database we already run for chat memory. That means one backup story, simple metadata filtering, and fewer vendors. A dedicated vector database may offer better performance at larger scale and richer filtering, but it adds another system, another credential and data leaving our infrastructure. I also check delete-by-metadata support, because the upsert design depends on removing all chunks for a document ID. For a company docs corpus, I would usually start with Postgres and move only if measured query latency or scale demanded it.

Vectors from the old and new models can't be mixed, so this is a full re-index, not an incremental change. I create a new index or collection and run a one-off ingestion of the entire corpus into it with the new model, while the chat flow keeps using the old one. Then I run the golden question set against the new index. If it performs better, I switch the chat workflow's vector store and embeddings settings together in one change, so query and index always match. I keep the old index briefly for rollback, then delete it.

Vercel AI Gateway gives spend tracking in one place, so I can see cost per model and spot spikes. The main levers sit in the workflow. Keep the system prompt concise, limit memory to a few turns, tune top-k so chunks don't bloat the prompt, and choose a model that meets quality needs rather than the largest available. Because the agent can make multiple calls, I watch tool-call counts in the logs. I also confirm that fallback models are reasonably priced, since outages can shift traffic to them.

For the chat workflow, I track failed executions and Error Trigger alerts in Slack, response latency, how often the agent calls the vector store tool, and the rate of 'I don't know' answers. A rising rate can signal ingestion gaps. For the gateway, I watch spend and how often fallbacks fire. For ingestion, I check that every scheduled run succeeds and that the cursor advances. I also track counts of documents and chunks upserted and deleted, and how fresh the newest indexed content is. Periodic golden-set runs and user feedback catch quality drift that technical metrics miss.

1. ARedis queue memory
2. BPostgres Chat MemoryCorrect answer
3. CGoogle Sheets memory
4. DWindow Buffer in a Code node

Why: Postgres Chat Memory loads recent turns for the sessionId and saves each new turn after the reply.

1. ATo reduce embedding dimensions
2. BTo stop stale and duplicate chunks from being retrievedCorrect answer
3. CTo reset the chat memory
4. DTo trigger the Error Trigger workflow

Why: Delete-then-insert by document ID makes each update an upsert. The old version can no longer be cited as current policy.

1. ASay it does not knowCorrect answer
2. BAsk the user to rephrase until something is found
3. CReturn the closest chunk as policy
4. DAnswer from general model knowledge

Why: The system prompt requires the agent to answer only from retrieved documents and to say it doesn't know otherwise.

1. ABuild the memory key from the authenticated identity and session, with authentication onCorrect answer
2. BDisable memory for all users
3. CUse one shared memory key for everyone
4. DTrust any sessionId the client sends

Why: Scoping memory to a server-verified identity plus session stops guessed or replayed session IDs from exposing others' history.

1. AThe chat model node reads the vector database directly
2. BThe vector store node is connected to the agent as a tool in retrieve modeCorrect answer
3. CThe vector store must be queried by a Code node before the Chat Trigger
4. DThe memory sub-node stores all document embeddings

Why: Vector store nodes can be attached to the AI Agent as a tool, so the agent decides when to search and receives the matching chunks; memory holds conversation turns, not the document index.

1. Ahttps://api.anthropic.com/v1
2. Bhttp://localhost:5678/v1
3. Chttps://api.openai.com/v1
4. Dhttps://ai-gateway.vercel.sh/v1Correct answer

Why: Vercel AI Gateway exposes an OpenAI-compatible API at https://ai-gateway.vercel.sh/v1, so one AI Gateway key reaches models like anthropic/claude-sonnet-4.5 or openai/gpt-5.

1. AchatInput and sessionIdCorrect answer
2. Bprompt and userToken
3. Cmessage and threadKey
4. Dquery and executionId

Why: The Chat Trigger emits chatInput, the question, and a sessionId. The sessionId is used to scope chat memory.

1. AIn insert mode as a trigger
2. BIn retrieve-as-tool mode as a tool sub-nodeCorrect answer
3. CAs a memory sub-node
4. DThrough an Execute Workflow node only

Why: The vector store node runs in retrieve-as-tool mode with a clear description. The agent calls it like any other tool.

1. AInsertCorrect answer
2. BLoad from memory
3. CRetrieve-as-tool
4. DGet many

Why: Ingestion upserts chunks with the vector store node in insert mode, after deleting old chunks for the same document ID.

1. AAn OpenAI-compatible chat model with a custom base URL and AI Gateway API keyCorrect answer
2. BAn HTTP Request node to each provider
3. CA text classifier node
4. DAn Anthropic node with a provider key

Why: An OpenAI-compatible chat model node pointed at the gateway base URL uses a single AI Gateway API key to reach many providers.

1. ABetter recall across languages
2. BFaster retrieval with the same accuracy
3. CDimension errors or silently irrelevant search resultsCorrect answer
4. DAutomatic re-indexing

Why: Different embedding models produce incompatible vector spaces, so similarity search fails or returns poor matches.

1. AThe chat silently returns an empty answer
2. BAn Error Trigger workflow posts the execution link to SlackCorrect answer
3. Cn8n deletes the vector index
4. DThe workflow switches to Simple Memory

Why: Persistent failures fire the Error Trigger workflow, which posts the execution link to a Slack channel for review.

1. AKafka and a load balancer only
2. BRedis and worker processesCorrect answer
3. CA second vector store
4. DOne n8n instance per user

Why: In queue mode, executions are queued in Redis and processed by multiple workers.

1. ANo provider keys are stored in n8n, and spend is tracked in one placeCorrect answer
2. BIt replaces the vector store
3. CIt guarantees zero latency
4. DIt removes the need for a system prompt

Why: One gateway credential reaches many providers. It adds fallbacks and spend tracking, and keeps provider keys out of n8n.

1. ATo change the embeddings model
2. BTo bill document owners
3. CSo chunks of deleted documents can be removed from the vector storeCorrect answer
4. DTo rebuild chat memory

Why: Without deletion handling, retired or withdrawn documents would keep answering questions.

1. ATreat retrieved chunks as data, not instructions, and keep the agent's tools read-onlyCorrect answer
2. BGive the agent a delete tool so it can remove the page
3. CRaise top-k
4. DTurn off chat authentication

Why: Retrieved content is untrusted input. Prompt rules plus read-only tools limit what an injection can do.

1. ANever; always re-ingest everything
2. BWhenever a chat message arrives
3. CAt the start of each run
4. DOnly after every document in the batch is processed successfullyCorrect answer

Why: Advancing only on success means failed runs reprocess the same window. Idempotent upserts make that safe.

1. ASimple Memory cannot store more than one message
2. BPostgres Chat Memory is required by the Chat Trigger
3. CSimple Memory lives in the worker process, so turns are lost across restarts and different workersCorrect answer
4. DPostgres Chat Memory also stores the document embeddings

Why: In queue mode different executions can run on different workers, so conversation history must be in a shared persistent store keyed by sessionId rather than in process memory.

1. Aanthropic/claude-sonnet-4.5Correct answer
2. Banthropic:claude:sonnet-4.5
3. Cmodels/anthropic-claude-sonnet-4.5
4. Dclaude-sonnet-4.5@anthropic

Why: Gateway model names use the provider/model format, such as anthropic/claude-sonnet-4.5 or openai/gpt-5.

1. AToken Splitter only
2. BRecursive Character Text SplitterCorrect answer
3. CMarkdown Header Splitter only
4. DNo splitter; whole documents are embedded

Why: The Default Data Loader and the Recursive Character Text Splitter produce overlapping chunks with metadata.

1. ATo keep content that crosses a chunk boundary retrievableCorrect answer
2. BTo speed up the Schedule Trigger
3. CTo encrypt chunk text
4. DTo double storage for backups

Why: Overlap stops a sentence or rule split across two chunks from losing its context.

1. AChat Trigger
2. BWebhook
3. CSchedule TriggerCorrect answer
4. DError Trigger

Why: Ingestion runs as a separate workflow started by a Schedule Trigger to keep the index fresh.

1. AA Wait node for approval
2. BMerge node
3. CSwitch node
4. DPer-node retry-on-failCorrect answer

Why: Retry-on-fail on individual nodes absorbs brief errors, so only persistent failures escalate.

1. AHard-coded in Code nodes
2. BStored encrypted and referenced by nodesCorrect answer
3. CPassed in chatInput
4. DSaved in vector store metadata

Why: n8n stores credentials encrypted and nodes reference them, so secrets don't appear in workflow logic.

1. AHTTP RequestCorrect answer
2. BSet/Edit Fields
3. CIF
4. DInformation Extractor

Why: The scheduled workflow lists changed documents from a wiki API with the HTTP Request node, or uses Google Drive.

1. AIt sets the embedding dimension
2. BThe agent relies on it to decide when and how to call the toolCorrect answer
3. CIt is the Postgres table name
4. DIt is shown to users as the answer

Why: The agent chooses tools based on their descriptions. A vague description can lead it to answer without retrieving.

1. ASorting Slack alerts
2. BJudging freshness and resolving conflicts between versionsCorrect answer
3. CEncrypting chunks
4. DChoosing the gateway provider

Why: Last-updated dates help the agent and reviewers prefer current content and spot outdated sources.

1. ARe-embed the whole corpus into a new index and switch query and index togetherCorrect answer
2. BClear only the chat memory
3. CChange only the chat model
4. DNothing; old vectors still work

Why: Vectors from different models are incompatible, so a full re-index is needed before switching queries.

1. AMemory is erased
2. BThe Schedule Trigger stops
3. CEmbeddings stop working
4. DMore tokens, higher cost and loosely related chunks that can mislead the modelCorrect answer

Why: Extra chunks increase prompt size and noise, which can hurt both cost and answer quality.

1. AThe AI Gateway API key
2. BThe full vector index
3. CThe execution linkCorrect answer
4. DThe employee's chat history

Why: Posting the execution link lets engineers open the failed run directly without leaking sensitive data.

1. AReturn a cached answer
2. BRetry the request with another configured provider or modelCorrect answer
3. CSwitch to Simple Memory
4. DRe-run ingestion

Why: The gateway falls back to another provider when the primary errors, keeping the chat available.

1. ARestrict who can view executions and set retention and pruningCorrect answer
2. BDisable all logging permanently
3. CCopy logs into the vector store
4. DMake logs public for transparency

Why: Questions can be sensitive personal data. Limited access and bounded retention protect privacy while still allowing debugging.

1. ALoop Over Items
2. BMerge
3. CChat Trigger
4. DExecute WorkflowCorrect answer

Why: The Execute Workflow node runs another workflow as a sub-workflow.