If Azure Search is set up as your search provider, you're probably already aware of the limitations that come with it. One common issue you may discover (even well after your initial launch) is the limit on the number of fields Azure Search allows - a maximum of 1,000 fields.In our case, after a major feature implementation and the introduction of a new site into our solution, the following log errors surfaced during the index rebuild process:
Exception: Sitecore.ContentSearch.Azure.Http.Exceptions.AzureSearchServiceRESTCallException Message: {"error":{"code":"","message":"The request is invalid. Details: definition : Invalid index: The index contains 1001 leaf fields (fields of a non-complex type). An index can have at most 1000 leaf fields.\r\n"}}The rebuild would get stuck and not budge. This has been documented.
In our case, the <indexAllFields> property was set to true (which is the default value) causing the index field count to exceed the limit.
For the web and master index, there are two ways to work around this limitation within the Sitecore.ContentSearch.Azure.DefaultIndexConfiguration.config or overriding patch file:
- Set <indexAllFields> to false - then includethe fields which are necessary in the <include hint="list:AddIncludedField"> section.
- Set <indexAllFields> to true and then exclude the fields which are unnecessary in the <exclude hint="list:AddExcludedField"> section.
I like the first option.
We initially attempted to manually create all fields we believed were needed for our custom search services. This turned out to be rather tedious - and left a lot of room for error had we missed a field.
To quickly and easily identify all custom fields to include in the AddIncludedField section, I put together this super simple Sitecore PowerShell Extensions snippet:
Notice the path to the User Defined folder. If you run this to include all custom fields with the query above as-is, you may run into the same 1,000 field limit error. You can simply change the path and run this multiple times (or build a script out of it) to generate sections in bulk for templates that make sense for your specific solution.
The output can be simply copied and pasted into the AddIncludedField section of the Sitecore.ContentSearch.Azure.DefaultIndexConfiguration.config file:
If you need to trim the index further, you should be able to determine your exclusions with a bit more granularity knowing that you haven't missed anything.
Happy Indexing! 🚀
Great post again, Gabe! Very helpful!
ReplyDeleteHere's an update to your script for newer versions of Sitecore that contain the Helix folders (Foundation, Feature and Project) in the Templates section:
Get-ChildItem -Path "master:\sitecore\templates\Foundation\", "master:\sitecore\templates\Feature", "master:\sitecore\templates\Project" -Recurse | Where-Object { $_.TemplateID -eq [Sitecore.TemplateIDs]::TemplateField } `
| ForEach-Object {
$fieldPath = $_.Paths.Path
$fieldId = $_.ID.ToString().ToUpper()
Write-Host "`t`n`t$fieldId"
}
Thanks Gabe ! very helpful and handy for adding custom indexed fields. it was a breeze!
ReplyDelete