The TYPO3 Crawler is an extension which provides possibilities, from both the TYPO3 backend and from CLI that helps you with your cache and e.g. search index.
Cache Warmup and full search index
Having a warm cache is essential for a performant website, this is a task that the Crawler takes over for you. The Crawler takes care of filling the TYPO3 Cache, on page changes. If using the ext:seo it will even crawl pages with respect to SEO priorities. Using indexed_search? no problem the Crawler will fill your search index for you. Cache Warmup and full search index
Want to learn more? See the people involved
Tomas Norre Mikkelsen