Connector Configuration
Configuration for Microsoft SharePoint Online
Please note that certain settings affect the performance of content or principal traversals, see Performance section of the FAQ for details.
Mandatory Settings
In order for the connector to connect to SharePoint Online, some actions are required on the source system as well as on the connector site. The following sections assume that the SharePoint Online related preparations described in SharePoint Online Setup have been done.
Authentication Settings
To establish a successful connection to Microsoft SharePoint Online an Azure application with certificate authentication has to be configured.
| Name | Key | Description |
|---|---|---|
Directory (Tenant) ID |
|
The ID of the Microsoft tenant. It can be found in the 'Overview' tab of the corresponding Azure application registration in the Azure Portal. |
Application (Client) ID |
|
The ID of the application registered in the Azure Portal. It can be found in the 'Overview' tab of the corresponding Azure application registration in the Azure Portal. |
Certificate File |
|
Upload for a certificate file (extension: pfx). See Certificate File Generation for a guide on how to generate a certificate. |
Certificate Password |
|
The password of the certificate file. See Certificate File Generation for a guide on how to generate a certificate. |
Optional Settings
The following sections describe which optional configuration adjustments can be done.
Principal Synchronization Settings
In various scenarios, varying strategies for principal resolution may be of interest. The connector offers support for three distinct modes:
-
Full SharePoint Online and all Azure AD groups and users:-
Fetches and resolves all SharePoint Online site collection specific groups.
-
Fetches and resolves all Entra ID users and groups.
-
-
Full SharePoint Online and referenced Azure AD groups and users-
Fetches and resolves all SharePoint Online site collection specific groups.
-
Resolves only Entra ID groups which were found as members of site collection groups.
-
-
Only SharePoint Online groups-
Fetches and resolves all SharePoint Online site collection specific groups.
-
Entra ID users and groups will not be fetched nor resolved.
-
| Resolution of SharePoint Online as well as Entra ID principals is required for secure search. This is the case, because Microsoft added Entra ID support on top of an already existing permission structure in SharePoint Online. |
| Name | Key | Description |
|---|---|---|
Principal Crawl Algorithm |
|
Defines how and in which scope the principal synchronization should be done.
Possible values are:
We recommend |
Site Discovery Settings
SharePoint Online’s top level content structure are site collections. Each site collection contains sites and lists. In order to define which site collection to traverse, the connector provides two strategies. Either a list of site collections is provided or a request against the Microsoft API will be done, fetching all site collections present in the given tenant. The result of the automatic site discovery can be narrowed down via filtering.
| Name | Key | Description |
|---|---|---|
Auto Site Discovery Enabled |
|
Defines if auto site discovery will be used. To manually state the site
collections to be traversed, unset this flag. Note: In order to use the
automatic site discovery, the Azure application requires the Microsoft Graph
permission |
Site Collections |
|
Upload for a file which contains site collections delimited by line. Only the site collections found in this file will be considered in the traversals. Needs to be configured if auto site discovery is disabled. |
General Filter Settings
The following general filter settings can be configured to restrict the traversals.
| Name | Key | Description |
|---|---|---|
Max File Size |
|
Maximal allowed size for content of documents. If the size is exceeding this value, the content of the document is discarded and ignored. |
Ignore ACL |
|
If this flag is set, the documents will be public. For public search scenarios this makes sense, because fewer requests will be executed. Furthermore, fewer permissions are required for the content traversal. |
Site Collection Filter Settings
Configuration options to restrict the traversal of site collections.
| Name | Key | Description |
|---|---|---|
Enable Site Collection URL Filtering |
|
Enables site collection filtering based on URL. |
Site Collection URL Matching Method |
|
Defines how the include / exclude entries should be matched against the site collection URL. Current supported methods are: EXACT: The site collection is filtered if its URL matches exactly with the entry. PREFIX: The site collection is filtered if its URL starts with the entry. SUFFIX: The site collection is filtered if its URL ends with the entry. SUBSTRING: The site collection is filtered if its URL contains the entry. REGEX: The site collection is filtered if its URL matches the RegEx pattern defined in the entry. |
Site Collection URL Filter Mode |
|
The type of filtering which should be applied. Either "INCLUDE" for including or "EXCLUDE" for excluding matching values. |
Site Collection URL Include List |
|
A list of URL-encoded site collection URLs. The include list entries are matched against site collection URLs. On match, the corresponding site will be traversed. |
Site Collection URL Exclude List |
|
A list of URL-encoded site collection URLs. The exclude list entries are matched against site collection URLs. On match, the corresponding site will not be traversed. |
Enable Geolocation Code Filtering |
|
Enables site collection filtering based on the geolocation code. |
Geolocation Code Filter Mode |
|
The type of filtering which should be applied. Either "INCLUDE" for including or "EXCLUDE" for excluding matching values. |
Geolocation Code Include List |
|
An include list of geolocation codes, against which all discovered site collections are evaluated via exact matching. If no include geolocation codes are configured, all site collections found will be traversed. For a list of available geolocation codes see the Official Microsoft Documentation. This list can not be configured simultaneously with the exclude list. |
Geolocation Code Exclude List |
|
An exclude list of geolocation codes, against which all discovered site collections are evaluated via exact matching. If no exclude geolocation codes are configured, all site collections found will be traversed. For a list of available geolocation codes see the Official Microsoft Documentation. This list can not be configured simultaneously with the include list. |
Site Filter Settings
Configuration options to restrict the traversed content based on site templates.
| Name | Key | Description |
|---|---|---|
Enable Site Template Filtering |
|
Enables site filtering based on templates. |
Site Template Filter Mode |
|
The type of filtering which should be applied. Either "INCLUDE" for including or "EXCLUDE" for excluding matching values. |
Included Site Templates |
|
An include list of site templates, against which all discovered site collections are matched via prefix matching.
If no include site templates are configured, all site collections found will be traversed.
The following site templates may be of interest: |
Excluded Site Templates |
|
An exclude list of site templates, against which all discovered site collections are matched via prefix matching. If no exclude site templates are configured, all site collections found will be traversed. Can not be configured simultaneously with include site templates. See Site Templates for a short guide on how to list all site templates present in your tenant. |
Enable Site URL Filtering |
|
Enables site collection filtering based on URL. |
Site URL Matching Method |
|
Defines how the include / exclude entries should be matched against the site URL. Current supported methods are: EXACT: The site is filtered if its URL matches exactly with the entry. PREFIX: The site is filtered if its URL starts with the entry. SUFFIX: The site is filtered if its URL ends with the entry. SUBSTRING: The site filtered if its URL contains the entry. REGEX: The site is filtered if its URL matches the RegEx pattern defined in the entry. |
Site URL Filter Mode |
|
The type of filtering which should be applied. Either "INCLUDE" for including or "EXCLUDE" for excluding matching values. |
Site URL Include List |
|
A list of URL-encoded site URLs. The include list entries are matched against site URLs. On match, the corresponding site will be traversed. |
Site URL Exclude List |
|
A list of URL-encoded site URLs. The exclude list entries are matched against site URLs. On match, the corresponding site will not be traversed. |
List Filter Settings
Configuration options to restrict the traversed lists.
| Name | Key | Description |
|---|---|---|
Traverse Generic Lists |
|
When disabled, generic lists will not be traversed by the connector. |
Traverse Document Libraries |
|
When disabled, document libraries will not be traversed by the connector. |
Enable List Template ID Filtering |
|
Enables list filtering based on template IDs. For a list of standard list templates see the Microsoft Documentation. |
List Template ID Filter Mode |
|
The type of filtering which should be applied. Either "INCLUDE" for including or "EXCLUDE" for excluding matching values. |
List Template ID Include List |
|
Defines list templates which should be traversed. This list can not be used together with the template ID exclude list. For a list of standard list templates see the Microsoft Documentation. |
List Template ID Exclude List |
|
Defines list templates which should not be traversed. This list can not be used together with the template ID include list. For a list of standard list templates see the Microsoft Documentation. |
Enable List Title Filtering |
|
Enables list filtering based on titles. |
List Title Filter Mode |
|
The type of filtering which should be applied. Either "INCLUDE" for including or "EXCLUDE" for excluding matching values. |
List Title Matching Method |
|
Defines how the include / exclude entries should be matched against the list title. Current supported methods are: EXACT: The list is filtered if its title matches exactly with the entry. PREFIX: The list is filtered if its title starts with the entry. SUFFIX: The list is filtered if its title ends with the entry. SUBSTRING: The list is filtered if its title contains the entry. REGEX: The list is filtered if its title matches the RegEx pattern defined in the entry. |
List Title Include List |
|
Its entries are matched against list titles. On match, the corresponding list will be traversed. |
List Title Exclude List |
|
Its entries are matched against list titles. On match, the corresponding list will not be traversed. |
File Extension Filter Settings
Configuration options to restrict the traversed files based on their extension. Files which are effected by this filter are document library items and generic list item attachments.
| Name | Key | Description |
|---|---|---|
Enable File Extension Filtering |
|
Enables filtering based on file extensions. |
File Extension Filter Mode |
|
The type of filtering which should be applied. Either "INCLUDE" for including or "EXCLUDE" for excluding matching values. |
File Extension Include List |
|
The include list entries are matched against file names. Only if the file name ends with an extension mentioned in this list, the corresponding file will be traversed. |
File Extension Exclude List |
|
The exclude list entries are matched against file names. If the file name ends with an extension mentioned in this list, the corresponding file will not be traversed. |
Transcription Settings
Microsoft SharePoint Online supports transcriptions for audio and video files. By enabling this option the connector will fetch the default transcription for such files and add its metadata and content to the document library item.
(File extensions currently supported by the connector: mp4, avi, mp4v, wmv, mov)
| Name | Key | Description |
|---|---|---|
Enable Transcription Fetching |
|
If enabled the connector will embed the default transcription into document library items. |
Transcription Placement |
|
The transcription content is either placed into the item’s metadata or it will replace the binary content of the item. (Transcription metadata is always added to the library item’s metadata.) |
Metadata Settings
Configuration options to customize the metadata provided by the connector.
| Name | Key | Description |
|---|---|---|
Enable Manual Item Type Definition |
|
Enables manual definition of item types. |
Override Site Item Type |
|
Override the item type for site objects. Default: |
Override List Item Type |
|
Override the item type for list objects. Default: |
Override Document Library File Item Type |
|
Override the item type for document library files. Default: |
Override Document Library Folder Item Type |
|
Override the item type for document library folders. Default: |
Override Generic List Item Item Type |
|
Override the item type for generic list items. Default: |
Override Generic List Folder Item Type |
|
Override the item type for generic list folders. Default: |
Override Attachment Item Type |
|
Override the item type for attachments. Default: |
Connection Settings
In order to adjust the connector on HTTP client level, the following parameters can be adjusted:
| Name | Key | Description |
|---|---|---|
Socket Timeout |
|
Duration after which inactive connections will be terminated. |
Connection Timeout |
|
Maximal duration a client should wait for an initial response from a server. |
Retry Count |
|
Maximal number of times a request should be retried on failure. |
Retry Delay |
|
Fix delay between request retries. |
Max Total Connections |
|
Maximum number of allowed connections overall. |
Max Connections Per Route |
|
Number of concurrent connections per route. |
Connection Time To Live (TTL) |
|
Duration after which unused connections will be discarded. |
Requests Per Second |
|
Maximum number of requests which are active in parallel. |
Rate Limit Requests Per Second |
|
Maximum number of requests which are active in parallel while the API rate is
near its limit. |
User Agent HTTP Header |
|
HTTP header which is sent with every request. It identifies the agent responsible for the request. |
Cache Settings
Configuration options determining values for the connector’s caches.
| Name | Key | Description |
|---|---|---|
Enable Cache Stat Logging |
|
When enabled, internal cache stats (cache hit / miss rate) are logged in a given interval. |
Stat Logging Interval |
|
The interval in which the cache stats should be logged. |
Site Templates
To restrict the number of site collections fetched by auto site discovery,
the connector provides the functionality to define a set of site templates
that should be considered or not. To obtain a list of site templates used in your
SharePoint Online tenant, you can run a PowerShell script provided by Raytion.
You can find it in the scripts folder of the connector.
-
Install the SharePoint Online PowerShell module. See the official documentation from Microsoft.
-
Open a PowerShell console as an administrator and navigate to the script’s folder location.
-
Run the PowerShell script:
.\SiteCollectionsToFile.ps1
-
If the Execution Policy denies execution, it can be bypassed by using the -ExecutionPolicy parameter with the Bypass option:
powershell -ExecutionPolicy Bypass -File .\SiteCollectionsToFile.ps1
-
Enter the URL to the Microsoft SharePoint Online admin center. Format:
https://<domain>-admin.sharepoint.com -
In the appearing prompt login with a tenant administrator account.
-
Two files will be generated: one containing all the site collection URLs, and another listing all the site templates found in the tenant.
ServiceNow AI Search Configuration
Instance Configuration
Configuration options related to specifying the ServiceNow AI Search instance.
| Setting | Description |
|---|---|
ServiceNow Instance Hostname |
Host Name of your ServiceNow Platform e.g. |
AI Search External Content User |
Username of the technical user with at least the role |
AI Search External Content User Password |
Password for the specified technical user. |
User Proxy |
If enabled, the connection to ServiceNow will be established through an HTTP/HTTPS proxy. |
Proxy Endpoint |
Target proxy URL including protocol, host and port. |
Connection Configuration (Optional)
Configuration options for fine-tuning the Http connection parameters.
| Setting | Description |
|---|---|
Requests per second |
The Number of requests sent per second. The default is set to |
Connection Timeout |
Timeout value for establishing a connection to host. Modify the value only if you modified system property |
Max. Number of Connections |
Max. number of allowed connections maintained by the connection manager. Modify the value only if you modified system property |
Payload Configuration (Optional)
Configuration options for fine-tuning AI Search payload constraints. Each batch must fulfill both the size constraint and the count constraint.
| Setting | Description |
|---|---|
Max. Content Batch Payload Size |
Max. batch payload size for ingesting documents. Modify the value only if you modified system property |
Max. Principal Batch Payload Size |
Max. batch payload size for mapping users. Modify the value only if you modified system property |
Max. Content Batch Item Count |
Max. number of items per batch for ingesting documents. |
Max. Principal Batch Item Count |
Max. number of users per batch for mapping users. |
General Configuration
Database Configuration
| Name | Property Key | Description |
|---|---|---|
Configuration Type |
|
Supported are PostgreSQL, MS SQL Server, and JDBC URL configuration. |
PostgreSQL
| Name | Property Key | Description |
|---|---|---|
Host |
|
Domain name or IP address of the database server. |
Port |
|
Specifies the port number PostgreSQL is listening on, default is 5432. |
Database Name |
|
Name of the database. |
Username |
|
Username to authenticate with. The regarding user has to have read and write permissions to the database. |
Password |
|
Password of the configured database user. |
Add Custom Parameter |
|
Enables the configuration of additional parameters. |
MS SQL Server
| Name | Property Key | Description |
|---|---|---|
Host |
|
Domain name or IP address of the database server. Instance to connect to on server can be specified by '‹server_name>|<instance_name>'. |
Port |
|
Specifies the port number MS SQL Server is listening on, default is 1433. |
Database Name |
|
Name of the database. |
Username |
|
Username to authenticate with. The regarding user has to have read and write permissions to the database. |
Password |
|
Password of the configured database user. |
Add Custom Parameter |
|
Enables the configuration of additional parameters. |
JDBC URL
| Name | Property Key | Description |
|---|---|---|
URL |
|
JDBC URL for the target database. Out of the box, the connector will use H2 file database. For productive usage, use PostgreSQL specifying the URL in format: |
Username |
|
Database Username to read and write to database. |
Password |
|
Database Password for the specified user |
Traversal Configuration
| Name | Property Key | Description |
|---|---|---|
Traversal History Length |
|
Max. number of traversals to store in the history. Once the limit is exceeded, the connector will automatically remove oldest entries in the history. (default: 100) |
Include Checksum |
|
If enabled, any changes made to the pipeline e.g. configuration, the subsequent incremental run triggers a refeed of all items. |
Change Processing Interval |
|
Interval between change processing traversals. |
Resume on Start |
|
If enabled, any traversals in paused state are automatically resumed after the connector restart. Otherwise, the traversal remains in paused state. |
Number of Traversal Workers |
|
Number of workers to execute the traversal in parallel. Increasing this value might improve the performance, but will footprint higher memory consumption. It is recommended to keep the default value. (default: 10) |
Traversal Job Poll Interval |
|
Interval between the workers to be triggered to fetch and process the next tasks. (default: 10ms) |
Completion Timeout |
|
If the search engine indexes the items asynchronously, there might be some processing still in-flight during the completion process of a traversal. This value specifies the timeout value until all asynchronous callbacks are expected to return before completing the traversal. (default: 10m) |
Executor Size |
|
The executor size restricts the max. number of concurrent running traversals. |
Queue Size |
|
The queue size restricts the max. number of queued traversals. If the value is exceeded, the connector rejects further traversal requests until the queue size is below the configured size. |
Traversal Jobs
| Name | Property Key | Description |
|---|---|---|
Job Timeout Check Frequency |
|
Configures how often the connector checks for timed out jobs. |
Job Timeout |
|
The duration for which a job can stay idle before it is timed out. |
Job Cache Size |
|
Max. cache size of Jobs waiting for processing in memory. When cache is empty, next batch is fetched. |
Security Configuration
Request Restriction Settings
| Name | Property Key | Description |
|---|---|---|
Accepted Host Domains |
|
A list of domains (+ port) that are allowed as host names in the headers of HTTP requests
to the connector. This means that you can access the connector only via a URL that
employs one of the configured domains. Each entry must have the format
If no domains are configured (the default), then you can use any domain via which the connector host is reachable. |
Principal Aliaser Configuration
Principal Aliasing is applied on user information as part of Content ACL processing during Content Synchronization and Principal processing during Principal Synchronization. It’s purpose is to map external source system user to the corresponding user in search engines domain. You can configure a list of aliasers in the connector which will be applied in sequence and in order on user ACEs and user principals. The Connector supports following custom aliasing mechanism.
Custom Aliaser Disabled
If the Custom Aliaser checkbox is not selected, the connector will process user information on ACE and user principals unchanged to Search Engine. If all relevant users in the source system can be found with the same identifier in the search engine, this setup is sufficient to reflect the same secure search experience in the search engine as defined by the policy in the source system. The connector uses this option as default to process user information.
Custom Aliaser Enabled
If custom aliasing is enable then there are four types of aliaser avaialble:
Simple XML Table Aliaser
Static mapping table which can be uploaded as XML file. The connector uses the uploaded file as lookup table to map a user in the source system to a user in the search engine. Users missing a record in the file will be dropped from the ACE and during Principal Synchronization. This option is only recommended for environment with a manageable amount of users as for each user the corresponding mapping entry needs to be specified in the file.
| Name | Description |
|---|---|
XML Mapping File |
Browse and upload or drag and drop. |
Sample XML mapping file:
<?xml version="1.0" encoding="UTF-8"?>
<storeddata>
<entry keyValue="user1">user1@raytion.com</entry>
<entry keyValue="user2">user2@raytion.com</entry>
<entry keyValue="user3">user3@raytion.com</entry>
</storeddata>
Regex Replacer Aliaser
Regex Replacer Aliaser computes aliases based on a regular expression. Principals that match the regular expression are replaced by the Substitution String.
| Name | Property Key | Description |
|---|---|---|
Pattern |
|
The regular expression to match, this is the part that will be replaced. If braces (…) are used in the pattern then the matched value can be retrieved using $1 |
Substitute String |
|
String to replace the matching part of the find string. Matched value is accessed by employing $1 |
Regex Extractor Aliaser
Regex Extractor Aliaser computes aliases based on a regular expression. Principals that match the regular expression are inserted into the Insert-Into String.
| Name | PropertyKey | Description |
|---|---|---|
Pattern |
|
The regular expression to match, this is the part that will be inserted into the new value. If braces (…) are used in the pattern then the matched value can be retrieved using $$ |
Insert-Into String |
|
String to replace the matching part of the pattern. Matched value is accessed by employing $$ |
LDAP Aliaser
Ldap Aliaser searches for an LDAP entry with the requested name in the input value and returns the specified output attribute.
| Name | Property Key | Description |
|---|---|---|
Host |
|
Fully Qualified Domain Name of an LDAP server |
Port |
|
Port to use for LDAP connection, defaults are 389/636 or (recommended) 3268/3269 for simple/SSL |
AccountDN |
|
AccountDN for bind to LDAP |
Password |
|
Password part of credentials |
Input Field |
|
The Active Directory attribute name for this equality filter |
Search Root DN |
|
Distinguished Name of the subtree which is searched. The smaller the subtree the better the performance but the higher the chance of encountering principals which are not part of this subtree |
Output Field |
|
Attribute that should be returned in result entries |