Crawl your tenant with SProbot

SProbot automatically indexes and crawls the SharePoint sites and teams on your tenant to make it possible for them to be reported on, assessment by AI, and cleaned up.

When you connect to your tenant, it will automatically start indexing daily, and individual sites will be crawled over a longer period of time.

You can see the status of both processes in the Crawl & Assess screen.

What is the index?

The index provides up-to-date (within 36-72 hours) information on:

• Storage, file count and activity metrics
• Security configurations
• Owners and members
• Guest users

What is the crawl?

The crawl runs over a longer period of time and analyzes individual items to get detailed information about:

• Large files
• Inactive files

You do not need to take any action for either of these processes, they are designed to run automatically without input from you.

Note: Sharing links are not automatically crawled for, they are retrieved on demand for specific sites.

How long does it take for my sites to be crawled?

First crawl

When you start using SProbot for the first time, you'll see detailed information appearing per site as the crawl progresses. Crawling all sites usually takes at least a week or two and may take significantly longer for larger tenants, with the rough guideline being 1 week per million items.

The counter displays the number of sites which have been processed by the current crawl run. When all sites have been crawled or at least a week has passed, a new run will automatically start.

Subsequent crawls

After the first crawl, subsequent runs only process sites which have changed since the last run, and only files within those sites which have been modified. This means that subsequent runs complete significantly faster.

Which sites are crawled first?

The processing sequence is by last activity ascending, so least active sites are crawled first.

How do I see when last a site was crawled?

You can see the last crawl date for a specific workspace in the Cleanup  > Search screen, or within the workspace itself.

If a workspace is currently being crawled, you'll see the orange crawl indicator in the search list.

You'll also see it within the info tiles for reports which are dependent on crawl data.