{
  "name": "Sync website pages from a sitemap to a Supabase vector store with OpenAI",
  "nodes": [
    {
      "parameters": {
        "content": "## Sync website pages from a sitemap to a Supabase vector store with OpenAI\n\nFor anyone building an AI chatbot that answers from their own website. It turns your pages into searchable chunks in a Supabase pgvector table and refreshes them weekly.\n\n### How it works\n1. A manual run or weekly schedule reads your sitemap, or a list of URLs you paste.\n2. Each page is fetched and stripped to readable text. Pages with almost no content are skipped.\n3. Old chunks for that URL are deleted, the text is split into overlapping chunks and OpenAI creates an embedding for each.\n4. Each chunk is written to Supabase, then a summary shows pages and chunks processed.\n\n### Setup\n- [ ] Enable the `vector` extension and create the `documents` table in Supabase.\n- [ ] Add a Supabase API credential (host and service role key) and an OpenAI API credential to the HTTP nodes.\n- [ ] Fill in **Ingestion settings** and run it manually first. Activate it for the weekly refresh.\n\n### Requirements\nn8n, an OpenAI API key and a Supabase project with pgvector.\n\n### Customization tips\nChange chunk size or overlap, the embedding model (match the vector column size) or the weekly schedule. Tune the text-stripping step for unusual sites.",
        "width": 480,
        "height": 1104
      },
      "type": "n8n-nodes-base.stickyNote",
      "typeVersion": 1,
      "position": [
        -560,
        0
      ],
      "id": "b5646623-b0dc-45f0-8379-e37a371ddad9",
      "name": "Template overview"
    },
    {
      "parameters": {
        "content": "## 1. Choose the pages to ingest\nA manual run or the weekly schedule, then a sitemap or your own list of URLs.",
        "width": 1184,
        "height": 480,
        "color": 7
      },
      "type": "n8n-nodes-base.stickyNote",
      "typeVersion": 1,
      "position": [
        0,
        0
      ],
      "id": "a00bacc9-29f6-4a6d-b583-adcd502ac45f",
      "name": "Section 1: Choose the pages to ingest"
    },
    {
      "parameters": {
        "content": "## 2. Fetch each page and clean the text\nLoops over pages, fetches the HTML and strips it to readable text. Low-content pages are skipped.",
        "width": 944,
        "height": 480,
        "color": 7
      },
      "type": "n8n-nodes-base.stickyNote",
      "typeVersion": 1,
      "position": [
        0,
        576
      ],
      "id": "d0921808-1e3e-45bd-b6d1-2732ae729b92",
      "name": "Section 2: Fetch each page and clean the text"
    },
    {
      "parameters": {
        "content": "## 3. Embed and store in Supabase\nDeletes old chunks for the URL, splits the text, creates embeddings and writes the rows.",
        "width": 1184,
        "height": 272,
        "color": 7
      },
      "type": "n8n-nodes-base.stickyNote",
      "typeVersion": 1,
      "position": [
        0,
        1152
      ],
      "id": "1b901945-97e2-447c-8659-844ed2247c5a",
      "name": "Section 3: Embed and store in Supabase"
    },
    {
      "parameters": {},
      "id": "manual-trigger",
      "name": "Click to run setup",
      "type": "n8n-nodes-base.manualTrigger",
      "typeVersion": 1,
      "position": [
        64,
        112
      ]
    },
    {
      "parameters": {
        "rule": {
          "interval": [
            {
              "field": "days",
              "daysInterval": 7
            }
          ]
        }
      },
      "id": "schedule-trigger",
      "name": "Weekly re-ingest",
      "type": "n8n-nodes-base.scheduleTrigger",
      "typeVersion": 1.2,
      "position": [
        64,
        312
      ]
    },
    {
      "parameters": {
        "assignments": {
          "assignments": [
            {
              "id": "cfg1",
              "name": "source_type",
              "value": "sitemap",
              "type": "string"
            },
            {
              "id": "cfg2",
              "name": "sitemap_url",
              "value": "https://example.com/sitemap.xml",
              "type": "string"
            },
            {
              "id": "cfg3",
              "name": "manual_urls",
              "value": "https://example.com/page-one\nhttps://example.com/page-two",
              "type": "string"
            },
            {
              "id": "cfg4",
              "name": "supabase_url",
              "value": "https://YOURPROJECT.supabase.co",
              "type": "string"
            },
            {
              "id": "cfg5",
              "name": "table_name",
              "value": "documents",
              "type": "string"
            },
            {
              "id": "cfg6",
              "name": "chunk_size",
              "value": 500,
              "type": "number"
            },
            {
              "id": "cfg7",
              "name": "chunk_overlap",
              "value": 50,
              "type": "number"
            },
            {
              "id": "cfg8",
              "name": "embedding_model",
              "value": "text-embedding-3-small",
              "type": "string"
            }
          ]
        },
        "options": {}
      },
      "id": "config",
      "name": "Ingestion settings",
      "type": "n8n-nodes-base.set",
      "typeVersion": 3.4,
      "position": [
        304,
        212
      ]
    },
    {
      "parameters": {
        "conditions": {
          "options": {
            "caseSensitive": true,
            "leftValue": "",
            "typeValidation": "loose",
            "version": 2
          },
          "conditions": [
            {
              "id": "src1",
              "leftValue": "={{ $json.source_type }}",
              "rightValue": "sitemap",
              "operator": {
                "type": "string",
                "operation": "equals"
              }
            }
          ],
          "combinator": "and"
        },
        "options": {}
      },
      "id": "if-sitemap",
      "name": "Which source type?",
      "type": "n8n-nodes-base.if",
      "typeVersion": 2.2,
      "position": [
        544,
        212
      ]
    },
    {
      "parameters": {
        "url": "={{ $json.sitemap_url }}",
        "options": {}
      },
      "id": "fetch-sitemap",
      "name": "Fetch sitemap XML",
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 4.2,
      "position": [
        784,
        112
      ]
    },
    {
      "parameters": {
        "jsCode": "// Extract every <loc> entry from the sitemap XML.\nconst xml = $input.first().json.data || $input.first().json.body || $input.first().json;\nconst text = typeof xml === 'string' ? xml : JSON.stringify(xml);\n\nconst matches = [...text.matchAll(/<loc>(.*?)<\\/loc>/g)].map(m => m[1].trim());\n\nif (matches.length === 0) {\n  throw new Error('No <loc> entries found in sitemap. Check sitemap_url is correct and publicly reachable.');\n}\n\nreturn matches.map(url => ({ json: { url } }));"
      },
      "id": "parse-sitemap",
      "name": "Parse sitemap URLs",
      "type": "n8n-nodes-base.code",
      "typeVersion": 2,
      "position": [
        1024,
        112
      ]
    },
    {
      "parameters": {
        "jsCode": "// Split the newline-separated manual_urls field into one item per URL.\nconst cfg = $input.first().json;\nconst urls = String(cfg.manual_urls || '')\n  .split('\\n')\n  .map(u => u.trim())\n  .filter(Boolean);\n\nif (urls.length === 0) {\n  throw new Error('manual_urls is empty. Add at least one URL, or switch source_type to sitemap.');\n}\n\nreturn urls.map(url => ({ json: { url } }));"
      },
      "id": "split-manual-urls",
      "name": "Split manual URL list",
      "type": "n8n-nodes-base.code",
      "typeVersion": 2,
      "position": [
        784,
        312
      ]
    },
    {
      "parameters": {
        "batchSize": 1,
        "options": {}
      },
      "id": "loop-pages",
      "name": "Loop through pages",
      "type": "n8n-nodes-base.splitInBatches",
      "typeVersion": 3,
      "position": [
        64,
        788
      ]
    },
    {
      "parameters": {
        "url": "={{ $json.url }}",
        "options": {
          "response": {
            "response": {
              "responseFormat": "text"
            }
          }
        }
      },
      "id": "fetch-page",
      "name": "Fetch page content",
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 4.2,
      "position": [
        304,
        688
      ]
    },
    {
      "parameters": {
        "jsCode": "// Rough boilerplate strip: drop script/style/nav/header/footer blocks,\n// strip remaining tags, collapse whitespace. Sites with unusual markup\n// (heavy client-side rendering, non-standard nav wrappers) may need this\n// node tuned per site — see the recipe page troubleshooting section.\nconst url = $('Loop through pages').item.json.url;\nlet html = $input.first().json.data || $input.first().json.body || '';\nif (typeof html !== 'string') html = String(html);\n\nhtml = html.replace(/<script[\\s\\S]*?<\\/script>/gi, '');\nhtml = html.replace(/<style[\\s\\S]*?<\\/style>/gi, '');\nhtml = html.replace(/<nav[\\s\\S]*?<\\/nav>/gi, '');\nhtml = html.replace(/<header[\\s\\S]*?<\\/header>/gi, '');\nhtml = html.replace(/<footer[\\s\\S]*?<\\/footer>/gi, '');\n\nconst text = html\n  .replace(/<[^>]+>/g, ' ')\n  .replace(/&nbsp;/g, ' ')\n  .replace(/&amp;/g, '&')\n  .replace(/\\s+/g, ' ')\n  .trim();\n\nif (text.length < 50) {\n  return [{ json: { url, text: '', skip: true, reason: 'stripped content too short — check the strip step against this page\\'s markup' } }];\n}\n\nreturn [{ json: { url, text, skip: false } }];"
      },
      "id": "strip-text",
      "name": "Strip to readable text",
      "type": "n8n-nodes-base.code",
      "typeVersion": 2,
      "position": [
        544,
        688
      ]
    },
    {
      "parameters": {
        "conditions": {
          "options": {
            "caseSensitive": true,
            "leftValue": "",
            "typeValidation": "loose",
            "version": 2
          },
          "conditions": [
            {
              "id": "skip1",
              "leftValue": "={{ $json.skip }}",
              "rightValue": true,
              "operator": {
                "type": "boolean",
                "operation": "equals"
              }
            }
          ],
          "combinator": "and"
        },
        "options": {}
      },
      "id": "skip-check",
      "name": "Skip low-content pages?",
      "type": "n8n-nodes-base.if",
      "typeVersion": 2.2,
      "position": [
        784,
        688
      ]
    },
    {
      "parameters": {
        "method": "DELETE",
        "url": "={{ $('Ingestion settings').first().json.supabase_url }}/rest/v1/{{ $('Ingestion settings').first().json.table_name }}?source_url=eq.{{ encodeURIComponent($json.url) }}",
        "authentication": "predefinedCredentialType",
        "options": {},
        "nodeCredentialType": "supabaseApi"
      },
      "id": "delete-existing-chunks",
      "name": "Delete existing chunks for this URL",
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 4.2,
      "position": [
        64,
        1264
      ]
    },
    {
      "parameters": {
        "jsCode": "// Split the page's text into overlapping chunks.\nconst cfg = $('Ingestion settings').first().json;\nconst page = $('Strip to readable text').first().json;\n\nif (page.skip) {\n  return [{ json: { url: page.url, skip: true } }];\n}\n\nconst chunkSize = Number(cfg.chunk_size) || 500;\nconst overlap = Number(cfg.chunk_overlap) || 50;\n\n// Rough token proxy: ~4 chars per token, so convert chunk_size (tokens) to chars.\nconst charSize = chunkSize * 4;\nconst charOverlap = overlap * 4;\n\nconst text = page.text;\nconst chunks = [];\nlet start = 0;\nlet index = 0;\n\nwhile (start < text.length) {\n  const end = Math.min(start + charSize, text.length);\n  const chunkText = text.slice(start, end).trim();\n  if (chunkText.length > 0) {\n    chunks.push({ json: { url: page.url, chunk_index: index, content: chunkText, skip: false } });\n    index += 1;\n  }\n  if (end === text.length) break;\n  start = end - charOverlap;\n}\n\nif (chunks.length === 0) {\n  return [{ json: { url: page.url, skip: true } }];\n}\n\nreturn chunks;"
      },
      "id": "chunk-text",
      "name": "Chunk text",
      "type": "n8n-nodes-base.code",
      "typeVersion": 2,
      "position": [
        304,
        1264
      ]
    },
    {
      "parameters": {
        "method": "POST",
        "url": "https://api.openai.com/v1/embeddings",
        "authentication": "predefinedCredentialType",
        "sendBody": true,
        "specifyBody": "json",
        "jsonBody": "={{ JSON.stringify({ model: $('Ingestion settings').first().json.embedding_model, input: $json.content }) }}",
        "options": {},
        "nodeCredentialType": "openAiApi"
      },
      "id": "generate-embedding",
      "name": "Generate embedding",
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 4.2,
      "position": [
        544,
        1264
      ]
    },
    {
      "parameters": {
        "jsCode": "// Runs once per chunk so every chunk gets written, not just the first.\n// Merge the chunk metadata with the embedding vector returned by OpenAI.\nconst embedResponse = $input.item.json;\nconst chunk = $('Chunk text').item.json;\n\nconst vector = (embedResponse.data && embedResponse.data[0] && embedResponse.data[0].embedding) || null;\n\nif (!vector) {\n  throw new Error('No embedding returned for chunk ' + chunk.chunk_index + ' of ' + chunk.url + '. Check the OpenAI credential and embedding_model value.');\n}\n\nreturn { json: {\n  source_url: chunk.url,\n  chunk_index: chunk.chunk_index,\n  content: chunk.content,\n  embedding: vector\n} };",
        "mode": "runOnceForEachItem"
      },
      "id": "build-upsert-row",
      "name": "Build upsert row",
      "type": "n8n-nodes-base.code",
      "typeVersion": 2,
      "position": [
        784,
        1264
      ]
    },
    {
      "parameters": {
        "method": "POST",
        "url": "={{ $('Ingestion settings').first().json.supabase_url }}/rest/v1/{{ $('Ingestion settings').first().json.table_name }}",
        "sendHeaders": true,
        "headerParameters": {
          "parameters": [
            {
              "name": "Prefer",
              "value": "return=minimal"
            }
          ]
        },
        "authentication": "predefinedCredentialType",
        "sendBody": true,
        "specifyBody": "json",
        "jsonBody": "={{ JSON.stringify({ source_url: $json.source_url, chunk_index: $json.chunk_index, content: $json.content, embedding: $json.embedding, metadata: { source_url: $json.source_url, chunk_index: $json.chunk_index } }) }}",
        "options": {},
        "nodeCredentialType": "supabaseApi"
      },
      "id": "upsert-chunk",
      "name": "Upsert chunk into Supabase",
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 4.2,
      "position": [
        1024,
        1264
      ]
    },
    {
      "parameters": {
        "jsCode": "// Aggregate results across every loop run, for the summary node.\n// $('Node').all() only returns the latest run, so walk each run index.\nfunction collect(name) {\n  const out = [];\n  for (let r = 0; r < 10000; r++) {\n    let items;\n    try { items = $(name).all(0, r); } catch (e) { break; }\n    if (!items || items.length === 0) break;\n    out.push(...items.map(i => i.json));\n  }\n  return out;\n}\n\nconst pages = collect('Strip to readable text');\nconst chunksWritten = collect('Build upsert row').length;\n\nconst skipped = pages.filter(p => p.skip);\n\nreturn [{ json: {\n  pages_processed: pages.length,\n  pages_skipped: skipped.length,\n  skipped_urls: skipped.map(p => p.url),\n  chunks_written: chunksWritten\n} }];"
      },
      "id": "ingestion-summary",
      "name": "Ingestion summary",
      "type": "n8n-nodes-base.code",
      "typeVersion": 2,
      "position": [
        304,
        888
      ]
    }
  ],
  "pinData": {},
  "connections": {
    "Click to run setup": {
      "main": [
        [
          {
            "node": "Ingestion settings",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Weekly re-ingest": {
      "main": [
        [
          {
            "node": "Ingestion settings",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Ingestion settings": {
      "main": [
        [
          {
            "node": "Which source type?",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Which source type?": {
      "main": [
        [
          {
            "node": "Fetch sitemap XML",
            "type": "main",
            "index": 0
          }
        ],
        [
          {
            "node": "Split manual URL list",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Fetch sitemap XML": {
      "main": [
        [
          {
            "node": "Parse sitemap URLs",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Parse sitemap URLs": {
      "main": [
        [
          {
            "node": "Loop through pages",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Split manual URL list": {
      "main": [
        [
          {
            "node": "Loop through pages",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Loop through pages": {
      "main": [
        [
          {
            "node": "Ingestion summary",
            "type": "main",
            "index": 0
          }
        ],
        [
          {
            "node": "Fetch page content",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Fetch page content": {
      "main": [
        [
          {
            "node": "Strip to readable text",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Strip to readable text": {
      "main": [
        [
          {
            "node": "Skip low-content pages?",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Skip low-content pages?": {
      "main": [
        [
          {
            "node": "Loop through pages",
            "type": "main",
            "index": 0
          }
        ],
        [
          {
            "node": "Delete existing chunks for this URL",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Delete existing chunks for this URL": {
      "main": [
        [
          {
            "node": "Chunk text",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Chunk text": {
      "main": [
        [
          {
            "node": "Generate embedding",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Generate embedding": {
      "main": [
        [
          {
            "node": "Build upsert row",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Build upsert row": {
      "main": [
        [
          {
            "node": "Upsert chunk into Supabase",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Upsert chunk into Supabase": {
      "main": [
        [
          {
            "node": "Loop through pages",
            "type": "main",
            "index": 0
          }
        ]
      ]
    }
  },
  "active": false,
  "settings": {
    "executionOrder": "v1"
  },
  "tags": []
}