Help centre › Enrichment

Crawl: follow links across a site

Crawl starts from each row’s web address, finds the links on that site, and adds every page it discovers as a new row, filled in like a Scrape. Use it to map a site, or to find pages like “team” and “contact”. It needs no key.

Add the column

1. Open a table with a column of website addresses and click Enrichment. Choose Crawl.
2. Under Column with URLs, pick the column that holds the starting page for each row.
3. Set the limits:
   • Max pages per row: 50 by default, from 1 to 500. It counts the starting page too.
   • Max depth: 2 by default, from 0 to 10. A depth of 0 reads only the starting page; 1 also follows its links; and so on.
   • Keyword filter (comma-separated): only follow links whose address or link text contains one of these words, such as “blog, product”.
   • Exclude patterns (comma-separated): skip links whose address path contains these words, such as “login, logout”.
   • Skip translated versions of the same page: on by default.
4. Tick what to collect, using the same options as the Scrape column, and run it.

What happens

The starting row is filled in place. Every other page it finds becomes a new row, marked Crawled. Crawl stays on the same website, treating www. and the bare domain as one site.

It skips pages that would only add duplicates:

• a page whose canonical address is one it already has;
• a page with identical content to one already crawled;
• translated versions of a page;
• links that differ only by tracking parameters or a # fragment.

The log notes these with a “⧉” line, and notes pages that took too long to load.

If it adds nothing

• A starting page with no links on the same site adds no rows. That is normal for a one-page site.
• A Keyword filter that matches nothing blocks every link. Clear it to test.
• “No rows with a valid URL to use as crawl seeds” means the address column is empty.