RAG finds nothing that SharePoint Search finds — the reasoning model spent the whole token cap thinking
Reasoning models count their hidden thinking against max_completion_tokens, so a "cheap" helper call with a cap of a few hundred tokens can come back with empty text and HTTP 200. If your RAG pipeline then falls back to searching the user's raw question, KQL ANDs every word ("what", "do", "we", "know"…) and SharePoint returns nothing — while the same topic typed into the SharePoint search box is the first hit.
Reasoning modely počítají skryté přemýšlení do max_completion_tokens, takže „levné“ pomocné volání se stropem pár set tokenů vrátí prázdný text a HTTP 200. Když pak RAG záložně hledá celou otázku uživatele, KQL spojí všechna slova přes AND („co“, „víme“, „o“…) a SharePoint nenajde nic – přestože totéž téma ve vyhledávacím poli SharePointu vyjde jako první.
Symptom
An assistant answers from SharePoint content through a classic pipeline: a small model call turns
the question into 2–3 keyword queries, the queries go to /_api/search/query, the hits become the
context. After switching the helper call to a GPT-5-family deployment:
- Questions about things that clearly exist (an event page, a safety rule) get "I found nothing specific", sometimes followed by a summary of unrelated newest pages.
- The same words typed into SharePoint search return the right page first.
- No errors anywhere. Every call is
200 OK. - It is intermittent — the same question sometimes works.
Cause
Two defects that are harmless alone and fatal together.
1. The cap includes reasoning tokens
Reasoning models accept only max_completion_tokens (max_tokens is not supported), and that limit
is "an upper bound for the number of tokens that can be generated for a completion, including visible
output tokens and reasoning tokens" (REST API reference). A query generator with a long system
prompt (classification rules, KQL rules, examples) can easily think for 300+ tokens before writing
a single word. When the cap is reached first, the call still succeeds and the text is empty — here as
relayed by the backend that wraps the Chat Completions call:
{ "content": "", "usage": { "promptTokens": 1539, "completionTokens": 300 } }completionTokens == maxTokens with empty content is the fingerprint. Measured live against a
gpt-5-mini deployment, same prompt, 5 runs per cell:
| Question | cap 300 | cap 2000 |
|---|---|---|
| short event question | 3 × empty | 0 × empty (max 460 used) |
| short policy question | 5 × empty | 0 × empty (max 584 used) |
It is stochastic: the reasoning length varies run to run, which is why it "sometimes works".
2. The fallback searches the raw question
Empty output is not an exception, so the parser's "nothing usable → search the question itself" branch kicks in. In KQL, free-text terms without an operator are combined with AND, so
(what do we know about the open day event) Path:"https://contoso.sharepoint.com/sites/intranet/*" IsDocument:1requires a document containing every word, including "what", "do", "we" and "know". Nothing does. The pipeline then often degrades further (e.g. "list newest documents in scope"), and the model politely summarizes irrelevant pages.
Fix
Give helper calls reasoning headroom. Cap = expected output + a fixed reserve. A cap of 2,000 (about 1,700 of it reserve) removed the empty answers in the measurement above. The cap is only a ceiling — you are billed for the tokens actually generated, not for the cap.
Treat empty output as a failure, not as "no queries". Log it with the usage numbers; if
completionTokensequals the cap, you hit this gotcha.Make the fallback usable on its own. Never send the raw sentence. Strip stop words and send the remaining terms twice — once ANDed, once ORed:
const STOP = ['what', 'do', 'we', 'know', 'about', 'the', 'is', 'how', 'where', 'who' /* + your languages */]; function fallbackQueries(q: string): string[] { const words = q.replace(/[^0-9A-Za-zÀ-ž\s-]+/g, ' ').split(/\s+/) .filter(w => w && STOP.indexOf(w.toLowerCase()) === -1); if (words.length === 0) return [q]; return words.length === 1 ? words : [words.join(' '), words.join(' OR ')]; }Stripping punctuation also removes stray
(,)and"that could break out of your scope clause (see kql-or-escapes-your-scope-filter).Check every helper call, not just the one that failed. Re-rankers, title generators, "suggest next step" calls and column-fill prompts — in the codebase behind this article they all had caps between 16 and 400 tokens.
⚠️ Watch for coupled limits. If your backend uses
maxTokensfor anything else (rate limiting, billing classification of "internal" vs "user" calls), raising the cap can move calls into a different bucket. Keep the new cap below any such threshold.
How to diagnose quickly
Wrap window.fetch on the page and log two things: the querytext sent to /_api/search/query,
and the JSON returned by the query-generator call. If the KQL contains the user's sentence verbatim,
or the generator returned "" with completionTokens equal to the cap, you are looking at this
gotcha — not at a ranking, permissions or crawl problem.
Related
- graph-search-raw-question-returns-nothing — the same raw-question failure against Microsoft Graph Search (mail, Teams).
References
- Azure OpenAI reasoning models — only
max_completion_tokens,max_tokensunsupported,reasoning_effort - Azure OpenAI REST API reference —
max_completion_tokensincludes reasoning tokens - Keyword Query Language (KQL) syntax reference — implicit AND between free-text terms