--- title: The crawl log DOES exist in SharePoint Online — it just isn't where you look for it short-title: The crawl log DOES exist in SharePoint Online summary: Reachable only through CSOM DocumentCrawlLog via /_vti_bin/client.svc/ProcessQuery, gated by a separate Crawl Log Permissions grant that no admin role implies; returns ~52 columns incl. per-pass crawl timestamps, errors, NoIndex and delete state tags: [search, crawl-log, csom, permissions, diagnostics] applies-to: SharePoint Online last-reviewed: 2026-08-07 --- # The crawl log DOES exist in SharePoint Online — it just isn't where you look for it > **Bottom line.** SharePoint Online has no Search Service Application UI and no crawl-time managed property, so it is easy to conclude there is no crawl log. There is one: CSOM `DocumentCrawlLog.GetCrawledUrls`, callable from plain JavaScript via `/_vti_bin/client.svc/ProcessQuery`. It needs a **separate permission grant** that being a tenant admin does *not* give you. > > **Ve zkratce.** SharePoint Online nemá správu vyhledávací služby ani vlastnost s časem procházení, takže se snadno usoudí, že crawl log neexistuje. Existuje: CSOM `DocumentCrawlLog.GetCrawledUrls`, volatelný i z čistého JavaScriptu přes `/_vti_bin/client.svc/ProcessQuery`. Vyžaduje ale **samostatné udělení přístupu**, které rolí správce tenantu nezískáte. ## Symptom You want crawl history for a site — when was this URL last crawled, did it error, is it excluded from the index — and every road looks like a dead end: - Search managed properties `LastCrawlTime`, `IndexedDate` and `CrawlTime` come back **empty**. (And they always "come back": search echoes any property you ask for, so a `null` proves nothing — see [unknown managed properties fail silently](unknown-managed-properties-fail-silently.md).) - The classic search admin pages (`/_layouts/15/searchadmin/searchadministration.aspx`) return an error. - There is no crawl log, crawl health report or crawl schedule anywhere in the SharePoint admin center navigation. So you conclude the crawl log is an on-premises-only feature. **That conclusion is wrong.** ## Cause The crawl log is exposed only through **CSOM**, in an assembly most people never touch: `Microsoft.SharePoint.Client.Search.Administration.DocumentCrawlLog`. There is no REST endpoint and no Graph equivalent, which is why REST-first exploration finds nothing. On top of that, reading it requires a permission grant that is **not implied by any admin role**. Without it every call fails with `UnauthorizedAccessException` — including for a Global Administrator, which makes it look like a broken or non-existent API rather than a missing grant. ## Fix ### 1. Grant crawl log read access Open, as a tenant admin: ``` https://-admin.sharepoint.com/_layouts/15/searchadmin/crawllogreadpermission.aspx ``` ("SharePoint admin center → More features → Search → Crawl Log Permissions".) Add the accounts that need to read the log and confirm. It takes effect immediately — no sign-out needed. ### 2. Call it From PowerShell, PnP wraps it for you: ```powershell Get-PnPSearchCrawlLog -ContentSource Sites -Filter "https://.sharepoint.com/sites/" ``` From a browser or an SPFx web part there is no wrapper — you post the CSOM request yourself. This runs against a **normal site collection**, not the admin site: ```js const web = _spPageContextInfo.webAbsoluteUrl; const digest = (await (await fetch(web + '/_api/contextinfo', { method: 'POST', headers: { Accept: 'application/json;odata=nometadata' } })).json()).FormDigestValue; const from = new Date(Date.now() - 30 * 86400000).toISOString(); const to = new Date().toISOString(); const P = (type, value) => '' + value + ''; // GetCrawledUrls(hotlist, rowLimit, urlFilter, hitCount, contentSourceId, errorLevel, errorId, from, to) const body = '' + '' + '' + '' + P('Boolean', 'false') + P('Int64', 100) + P('String', web) + P('Boolean', 'true') + P('Int32', -1) + P('Int32', -1) + P('Int32', -1) + P('DateTime', from) + P('DateTime', to) + '' + '' + '' + '' + '' + '' + '' + ''; const res = await fetch(web + '/_vti_bin/client.svc/ProcessQuery', { method: 'POST', headers: { 'Content-Type': 'text/xml', 'X-RequestDigest': digest }, body: body }); const parsed = JSON.parse(await res.text()); // CSOM reports failures with HTTP 200 — check ErrorInfo, never the status code if (parsed[0] && parsed[0].ErrorInfo) throw new Error(parsed[0].ErrorInfo.ErrorMessage); const rows = (parsed.filter(x => x && x._ObjectType_ === 'SP.SimpleDataTable')[0] || {}).Rows || []; ``` The two GUIDs are the CSOM type identifiers for `ClientContext` and `DocumentCrawlLog`. If you ever need such an identifier for another CSOM type, read it out of the shipped client library instead of guessing — it is a string literal in the constructor IL: ```powershell $asm = [Reflection.Assembly]::LoadFrom('...\Microsoft.SharePoint.Client.Search.dll') $type = $asm.GetType('Microsoft.SharePoint.Client.Search.Administration.DocumentCrawlLog') $ctor = $type.GetConstructors()[0] $il = $ctor.GetMethodBody().GetILAsByteArray() for ($i = 0; $i -lt $il.Length - 4; $i++) { if ($il[$i] -eq 0x72) { # ldstr $token = [BitConverter]::ToInt32($il, $i + 1) try { $type.Module.ResolveString($token) } catch { } } } ``` ## What you get back An `SP.SimpleDataTable` with roughly 52 columns per row. The ones worth having: | Column | Meaning | |---|---| | `FullUrl` | the crawled URL | | `AccessData` | **what kind of object this is** — `SPWeb`, `SPList`, `SPListItem`, `SPFolder`, `SPView`. Use this, never a guess based on the URL | | `TimeStampUtc` | last crawl of this item | | `TimeStamp_AddModify`, `TimeStamp_SecurityOnly`, `TimeStamp_EnumerateChildren` | separate timestamps for content vs. permissions-only vs. child enumeration passes | | `ErrorCode`, `ErrorDesc`, `ErrorCount`, `ErrorLevel`, `StatusMessage` | why an item failed to crawl | | `NoIndex`, `ExclusionReason` | 🚫 **not usable** — see the traps below; use REST `NoCrawl` instead | | `IsDeleted`, `DeleteReason`, `DeletePending` | removal state | | `SPItemModifiedTime` | content modification time (UTC) — against `TimeStampUtc` this gives indexing lag | | `ChildrenCount`, `ParentDocID` | crawl hierarchy | ## Traps when reading the result - **Values are CSOM-encoded, not plain JSON.** Dates arrive both as `/Date(1785959294170)/` (epoch millis) and as `/Date(2026,7,5,19,48,14,170)/` — where **the month is zero-based**. GUIDs come as `/Guid(...)/`, binary as `/Base64Binary(...)/`. Parse with a regex; `new Date(value)` will not do it. - **Three different time bases, and two of them lie if taken naively.** This is the trap most likely to reach production, because the wrong values look plausible: | Column | Base | How to read it | |---|---|---| | `TimeStampUtc` | epoch millis, **UTC** | the only unambiguous one — use it for "last crawled" | | `TimeStamp`, `TimeStamp_AddModify`, `TimeStamp_SecurityOnly`, `SPItemModifiedTime` | component form, **UTC** | `Date.UTC(y, m, d, …)` — **not** `new Date(y, m, d, …)`, which reads them as browser-local and shifts them by your offset | | `LastRepositoryModifiedTime` | FILETIME in the **backend's local time** | 🚫 do not use — see below | Read as epoch millis, `LastRepositoryModifiedTime` yields the year ~4,250,000. Converted properly from FILETIME it is still **7 hours ahead of reality**: an item whose REST `Modified` is `2026-08-07T06:14:52Z` carries `SPItemModifiedTime` of exactly `06:14:52Z` but a FILETIME of `13:14:52Z`. Seven hours is US Pacific *daylight* time — meaning **the offset becomes eight in winter**. Take the modification time from `SPItemModifiedTime` instead. - **Verify one value against an independent source before building a metric on it.** Indexing lag computed from the wrong column produced 20 usable values out of 400 and **367 rows claiming the crawl happened before the change** — which reads as a property of the tenant ("almost everything is waiting to be re-indexed"), not as a conversion bug. The numbers were nonsense, but *plausible* nonsense. After switching to `SPItemModifiedTime`: 387 usable, zero pending, median 23 minutes. One comparison against REST `Modified` would have caught it immediately. - **HTTP 200 does not mean success.** CSOM reports failures inside `ErrorInfo` with a 200 status — the missing-permission case included. - **Most rows are not "content".** A quiet 30-day window on one small site returned 468 rows: 466 flagged `NoIndex` (mostly `AllItems.aspx` / `DispForm.aspx` form pages, correctly excluded) and 376 flagged `IsDeleted` (transient list items). Filter before showing this to anyone, or a perfectly healthy site will look alarming. - **Do not classify rows by their URL — read `AccessData`.** The obvious heuristic ("`AllItems.aspx` is a view, `DispForm.aspx` is a form, everything else is content") is wrong, because the crawler indexes a list *through* its form pages: to SharePoint, `.../Lists/X/AllItems.aspx` **is the list** and `.../Lists/X/DispForm.aspx?ID=1` **is list item 1**. Measured against `AccessData` on 254 rows, that heuristic agreed on **9**. Everything downstream — counts, per-type breakdowns, rules — inherits the error. - 🚫 **`NoIndex` does not mean "excluded from the index". Do not build anything on it.** The name is a trap. Measured across 2,000 rows and checked against both REST and search: | List | crawl log `NoIndex` | REST `NoCrawl` (the truth) | Search hits | |---|---|---|---| | a deliberately excluded data list | true | **true** | 0 | | `Shared Documents` | true | **false** | 2 | | `SiteAssets` | true | false | 3 | | `TaxonomyHiddenList` | **false** | – | 0 | `NoIndex: true` covers 30 of 32 lists, 20 of 20 root folders, 25 of 31 views, 1,872 of 1,894 items and both site collections — essentially every row, regardless of configuration, and as the last line shows it sometimes points the wrong way. It appears to mean "this row is not itself a document in the index" (containers, deleted items), not "someone excluded it". **The authoritative source is REST `NoCrawl`** on the web (`/_api/web?$select=NoCrawl`) and on the list (`/_api/web/lists?$select=NoCrawl`). Worth stating how this was nearly missed: the first correction only narrowed the count from items to containers, on the theory that items inherit the flag from their list. That reduced 2,251 to 30 and looked solved. It wasn't — the check had only ever looked at lists that genuinely *were* excluded. One ordinary document library, known not to be excluded, was enough to show the field does not discriminate at all. **A sample without a counter-example confirms whatever you already believe.** - **`ExclusionReason` is always 0**, including on rows where `NoIndex` is true — SharePoint Online does not appear to populate it, and no other column carries a reason. ## Meta-lesson Four columns in a row — `LastRepositoryModifiedTime`, the object type, `NoIndex`, `ExclusionReason` — meant something other than their name suggests, and each one moved the resulting number by an order of magnitude. What made them expensive is that every wrong result was **plausible**: "almost everything is waiting to be re-indexed", "most rows are form pages", "a lot of content sits outside the index" could all be true of a real tenant. Nothing about the output invited a second look. So, before a crawl-log column becomes a number in a UI or a threshold in a rule, check its meaning against an **independent** source — REST `Modified` for a timestamp, `AccessData` for a type, REST `NoCrawl` plus a real search query for an exclusion. Two things make such a check worth anything: a **non-empty control** ("searching for that list returned 0" proves nothing until a query that *must* return something runs beside it), and a **counter-example** — at least one object you know is *not* in the state you are testing for. Without the counter-example a field that is always true looks like confirmation. "This API does not exist in SharePoint Online" is a claim about absence, and two quick probes cannot establish it — especially when both probes tested *different* surfaces (search properties, admin pages) than the one it actually lives on (CSOM). Before declaring a SharePoint capability missing, **grep the PnP PowerShell cmdlets**: they are thin wrappers over CSOM and REST, and they routinely expose surfaces that the documentation never mentions. ## See also - [Unknown managed properties fail silently](unknown-managed-properties-fail-silently.md) — why `LastCrawlTime: null` proves nothing - [`NoCrawl` on a library silently blinds your RAG](nocrawl-on-a-library-silently-blinds-your-rag.md) — the `NoIndex` column above, seen from the other end