The crawl log DOES exist in SharePoint Online — it just isn't where you look for it

Symptom

You want crawl history for a site — when was this URL last crawled, did it error, is it excluded from the index — and every road looks like a dead end:

• Search managed properties LastCrawlTime, IndexedDate and CrawlTime come back empty. (And they always "come back": search echoes any property you ask for, so a null proves nothing — see unknown managed properties fail silently.)
• The classic search admin pages (/_layouts/15/searchadmin/searchadministration.aspx) return an error.
• There is no crawl log, crawl health report or crawl schedule anywhere in the SharePoint admin center navigation.

So you conclude the crawl log is an on-premises-only feature. That conclusion is wrong.

Cause

The crawl log is exposed only through CSOM, in an assembly most people never touch: Microsoft.SharePoint.Client.Search.Administration.DocumentCrawlLog. There is no REST endpoint and no Graph equivalent, which is why REST-first exploration finds nothing.

On top of that, reading it requires a permission grant that is not implied by any admin role. Without it every call fails with UnauthorizedAccessException — including for a Global Administrator, which makes it look like a broken or non-existent API rather than a missing grant.

Fix

1. Grant crawl log read access

Open, as a tenant admin:

https://<tenant>-admin.sharepoint.com/_layouts/15/searchadmin/crawllogreadpermission.aspx

("SharePoint admin center → More features → Search → Crawl Log Permissions".) Add the accounts that need to read the log and confirm. It takes effect immediately — no sign-out needed.

2. Call it

From PowerShell, PnP wraps it for you:

Get-PnPSearchCrawlLog -ContentSource Sites -Filter "https://<tenant>.sharepoint.com/sites/<site>"

From a browser or an SPFx web part there is no wrapper — you post the CSOM request yourself. This runs against a normal site collection, not the admin site:

const web = _spPageContextInfo.webAbsoluteUrl;
const digest = (await (await fetch(web + '/_api/contextinfo', {
  method: 'POST', headers: { Accept: 'application/json;odata=nometadata' }
})).json()).FormDigestValue;

const from = new Date(Date.now() - 30 * 86400000).toISOString();
const to = new Date().toISOString();
const P = (type, value) => '<Parameter Type="' + type + '">' + value + '</Parameter>';

// GetCrawledUrls(hotlist, rowLimit, urlFilter, hitCount, contentSourceId, errorLevel, errorId, from, to)
const body =
  '<Request SchemaVersion="15.0.0.0" LibraryVersion="16.0.0.0" ApplicationName="crawl-log"'
  + ' xmlns="http://schemas.microsoft.com/sharepoint/clientquery/2009">'
  + '<Actions>'
  + '<ObjectPath Id="4" ObjectPathId="3" />'
  + '<Method Name="GetCrawledUrls" Id="5" ObjectPathId="3"><Parameters>'
  + P('Boolean', 'false') + P('Int64', 100) + P('String', web) + P('Boolean', 'true')
  + P('Int32', -1) + P('Int32', -1) + P('Int32', -1)
  + P('DateTime', from) + P('DateTime', to)
  + '</Parameters></Method>'
  + '</Actions>'
  + '<ObjectPaths>'
  + '<StaticProperty Id="0" TypeId="{3747adcd-a3c3-41b9-bfab-4a64dd2f1e0a}" Name="Current" />'
  + '<Property Id="2" ParentId="0" Name="Site" />'
  + '<Constructor Id="3" TypeId="{5c5cfd42-0712-4c00-ae49-23b33ba34ecc}">'
  + '<Parameters><Parameter ObjectPathId="2" /></Parameters></Constructor>'
  + '</ObjectPaths></Request>';

const res = await fetch(web + '/_vti_bin/client.svc/ProcessQuery', {
  method: 'POST',
  headers: { 'Content-Type': 'text/xml', 'X-RequestDigest': digest },
  body: body
});
const parsed = JSON.parse(await res.text());

// CSOM reports failures with HTTP 200 — check ErrorInfo, never the status code
if (parsed[0] && parsed[0].ErrorInfo) throw new Error(parsed[0].ErrorInfo.ErrorMessage);

const rows = (parsed.filter(x => x && x._ObjectType_ === 'SP.SimpleDataTable')[0] || {}).Rows || [];

The two GUIDs are the CSOM type identifiers for ClientContext and DocumentCrawlLog. If you ever need such an identifier for another CSOM type, read it out of the shipped client library instead of guessing — it is a string literal in the constructor IL:

$asm = [Reflection.Assembly]::LoadFrom('...\Microsoft.SharePoint.Client.Search.dll')
$type = $asm.GetType('Microsoft.SharePoint.Client.Search.Administration.DocumentCrawlLog')
$ctor = $type.GetConstructors()[0]
$il   = $ctor.GetMethodBody().GetILAsByteArray()
for ($i = 0; $i -lt $il.Length - 4; $i++) {
  if ($il[$i] -eq 0x72) {                       # ldstr
    $token = [BitConverter]::ToInt32($il, $i + 1)
    try { $type.Module.ResolveString($token) } catch { }
  }
}

What you get back

An SP.SimpleDataTable with roughly 52 columns per row. The ones worth having:

Traps when reading the result

• Values are CSOM-encoded, not plain JSON. Dates arrive both as /Date(1785959294170)/ (epoch millis) and as /Date(2026,7,5,19,48,14,170)/ — where the month is zero-based. GUIDs come as /Guid(...)/, binary as /Base64Binary(...)/. Parse with a regex; new Date(value) will not do it.

• Three different time bases, and two of them lie if taken naively. This is the trap most likely to reach production, because the wrong values look plausible:

  Column	Base	How to read it
  TimeStampUtc	epoch millis, UTC	the only unambiguous one — use it for "last crawled"
  TimeStamp, TimeStamp_AddModify, TimeStamp_SecurityOnly, SPItemModifiedTime	component form, UTC	Date.UTC(y, m, d, …) — not new Date(y, m, d, …), which reads them as browser-local and shifts them by your offset
  LastRepositoryModifiedTime	FILETIME in the backend's local time	🚫 do not use — see below

  Read as epoch millis, LastRepositoryModifiedTime yields the year ~4,250,000. Converted properly from FILETIME it is still 7 hours ahead of reality: an item whose REST Modified is 2026-08-07T06:14:52Z carries SPItemModifiedTime of exactly 06:14:52Z but a FILETIME of 13:14:52Z. Seven hours is US Pacific daylight time — meaning the offset becomes eight in winter. Take the modification time from SPItemModifiedTime instead.

• Verify one value against an independent source before building a metric on it. Indexing lag computed from the wrong column produced 20 usable values out of 400 and 367 rows claiming the crawl happened before the change — which reads as a property of the tenant ("almost everything is waiting to be re-indexed"), not as a conversion bug. The numbers were nonsense, but plausible nonsense. After switching to SPItemModifiedTime: 387 usable, zero pending, median 23 minutes. One comparison against REST Modified would have caught it immediately.

• HTTP 200 does not mean success. CSOM reports failures inside ErrorInfo with a 200 status — the missing-permission case included.

• Most rows are not "content". A quiet 30-day window on one small site returned 468 rows: 466 flagged NoIndex (mostly AllItems.aspx / DispForm.aspx form pages, correctly excluded) and 376 flagged IsDeleted (transient list items). Filter before showing this to anyone, or a perfectly healthy site will look alarming.

• Do not classify rows by their URL — read AccessData. The obvious heuristic ("AllItems.aspx is a view, DispForm.aspx is a form, everything else is content") is wrong, because the crawler indexes a list through its form pages: to SharePoint, .../Lists/X/AllItems.aspx is the list and .../Lists/X/DispForm.aspx?ID=1 is list item 1. Measured against AccessData on 254 rows, that heuristic agreed on 9. Everything downstream — counts, per-type breakdowns, rules — inherits the error.

• 🚫 NoIndex does not mean "excluded from the index". Do not build anything on it. The name is a trap. Measured across 2,000 rows and checked against both REST and search:

  List	crawl log NoIndex	REST NoCrawl (the truth)	Search hits
  a deliberately excluded data list	true	true	0
  Shared Documents	true	false	2
  SiteAssets	true	false	3
  TaxonomyHiddenList	false	–	0

  NoIndex: true covers 30 of 32 lists, 20 of 20 root folders, 25 of 31 views, 1,872 of 1,894 items and both site collections — essentially every row, regardless of configuration, and as the last line shows it sometimes points the wrong way. It appears to mean "this row is not itself a document in the index" (containers, deleted items), not "someone excluded it". The authoritative source is REST NoCrawl on the web (/_api/web?$select=NoCrawl) and on the list (/_api/web/lists?$select=NoCrawl).

  Worth stating how this was nearly missed: the first correction only narrowed the count from items to containers, on the theory that items inherit the flag from their list. That reduced 2,251 to 30 and looked solved. It wasn't — the check had only ever looked at lists that genuinely were excluded. One ordinary document library, known not to be excluded, was enough to show the field does not discriminate at all. A sample without a counter-example confirms whatever you already believe.

• ExclusionReason is always 0, including on rows where NoIndex is true — SharePoint Online does not appear to populate it, and no other column carries a reason.

Meta-lesson

Four columns in a row — LastRepositoryModifiedTime, the object type, NoIndex, ExclusionReason — meant something other than their name suggests, and each one moved the resulting number by an order of magnitude. What made them expensive is that every wrong result was plausible: "almost everything is waiting to be re-indexed", "most rows are form pages", "a lot of content sits outside the index" could all be true of a real tenant. Nothing about the output invited a second look.

So, before a crawl-log column becomes a number in a UI or a threshold in a rule, check its meaning against an independent source — REST Modified for a timestamp, AccessData for a type, REST NoCrawl plus a real search query for an exclusion. Two things make such a check worth anything: a non-empty control ("searching for that list returned 0" proves nothing until a query that must return something runs beside it), and a counter-example — at least one object you know is not in the state you are testing for. Without the counter-example a field that is always true looks like confirmation.

"This API does not exist in SharePoint Online" is a claim about absence, and two quick probes cannot establish it — especially when both probes tested different surfaces (search properties, admin pages) than the one it actually lives on (CSOM). Before declaring a SharePoint capability missing, grep the PnP PowerShell cmdlets: they are thin wrappers over CSOM and REST, and they routinely expose surfaces that the documentation never mentions.

See also

• Unknown managed properties fail silently — why LastCrawlTime: null proves nothing
• NoCrawl on a library silently blinds your RAG — the NoIndex column above, seen from the other end