Websites
Last updated
Last updated
Gleen AI supports crawling and indexing content from publicly accessible websites to answer queries. This includes:
Landing pages
Sitemaps
Product pages
Help articles
Notion docs
Support articles
Developer docs
Zendesk support articles
CSV file links hosted on public cloud
For each website link, there are two different modes of crawling:
Crawl multiple pages: In multi-page crawl, Gleen AI will find the child pages and continue crawling as long as the root path of the child pages is the same as the root path of the parent URL. We crawl up to 5,000 pages per URL. If you have specific needs or require crawling more than 5,000 pages, just message us or reach out to our human customer support. For sitemaps, choose the multi-page crawl as it will also crawl child pages.
Crawl single page: In single-page crawl, we crawl only one page of the given URL.
