Indexing Custom Entities

16 min

Note: this guide covers integration of entities with a custom entity type, not the injection of additional entities into existing entity type sync (for example, sending blog posts as KLEVU_CMS alongside CMS Pages).

Beyond the standard Product, Category, and CMS Page enities, the Klevu indexes support sync and return of custom entity types. Ultimately, all records are stored in the same "bucket", with the same core structure and attributes. Behaviour (such as in search) is then modified using filters and actions referencing the entity type - natively KLEVU_PRODUCT; KLEVU_CATEGORY; and KLEVU_CMS.

What this means is that we can create and sync any "type" of entity with the Klevu indexes, providing it has the same basic data points and a unique entity type identifier. Frontend customisations can then handle search and display to allow integration seamlessly into your site.

Implementing in Magento

Version 4.x of the Klevu Magento integration is designed with extensibility in mind - including the ability to inject different entity types into the indexing discovery and sync operations. This is the same extensibility used within the module itself to manage products, categories, and cms so we can follow the same steps to handle other entities.

Note: the creation and management of these entities is the responsibility of the integrator, though may include entities (such as blog posts) from other third party extensions. The key requirement for integration with Klevu's indexing modules is that the entity implements either ExtensibleDataInterface or PageInterface. If integrating a third party entity which does not implement one of these interfaces, you may want need to create a custom version of the class. This is out of the scope of this guide.

A working example module can be found on Github.

Create an Entity Provider

First we need to create a service to provide those custom entities to the indexing module. This class must implement the \Klevu\IndexingApi\Service\Provider\EntityProviderInterface interface, and contain a get method which returns a \Generator yielding arrays of custom entities to support batching.

The class must also contain a getEntitySubtype method, which often will simply contain a string of your choice. This method is primarily used to support indexing of entities which have different subtypes, such as Products (configurable, simple, variants, etc).

An example Provider implementation can be found in the sample extension on Github.

Once you have created the class, you will need to wire it in using dependency injection. You will need to create both a standard and "batched" version of the provider, though this can be achieved by simply creating a virtualType of the concrete class.

    <type name="Vendor\KlevuIndexingCustomEntities\Service\Provider\CustomEntityProvider">
        <arguments>
            <argument name="logger" xsi:type="object">Klevu\Indexing\Logger\Logger</argument>
            <argument name="batchSize" xsi:type="const">Klevu\Indexing\Constants::DEFAULT_INDEXING_BATCH_SIZE</argument>
            <argument name="batchSizeValidator" xsi:type="object">Klevu\Indexing\Validator\BatchSizeValidator</argument>
        </arguments>
    </type>

    <virtualType name="Vendor\KlevuIndexingCustomEntities\Service\Provider\CustomEntityProvider\Batched"
                 type="Vendor\KlevuIndexingCustomEntities\Service\Provider\CustomEntityProvider" />

You will also need to create implementations of the Klevu\Indexing\Service\Provider\EntityProviderProvider class and inject your custom provider(s - if you are syncing multiple custom entity types via the same module). This can be done using a virtualType as follows

    <virtualType name="Vendor\KlevuIndexingCustomEntities\Service\Provider\EntityProviderProvider"
                 type="Klevu\Indexing\Service\Provider\EntityProviderProvider">
        <arguments>
            <argument name="entityProviders" xsi:type="array">
                <item name="vendor_custom_entity"
                      xsi:type="object">Vendor\KlevuIndexingCustomEntities\Service\Provider\CustomEntityProvider</item>
            </argument>
        </arguments>
    </virtualType>

    <virtualType name="Vendor\KlevuIndexingCustomEntities\Service\Provider\EntityProviderProvider\Batched"
                 type="Klevu\Indexing\Service\Provider\EntityProviderProvider">
        <arguments>
            <argument name="entityProviders" xsi:type="array">
                <item name="vendor_custom_entity"
                      xsi:type="object">Vendor\KlevuIndexingCustomEntities\Service\Provider\CustomEntityProvider\Batched</item>
            </argument>
        </arguments>
    </virtualType>

Note: the item name should correspond to the entity type identifier you will use for sync (eg, for Klevu, klevu_product, klevu_category, and klevu_cms)

Incorporate Is Indexable Conditions

As part of entity provision, we also need to tell the core module how to decide if an entity is indexable (think disabled products, or those assigned to different websites).

The Determiner itself can be a virtualType of Klevu\Indexing\Service\Determiner\IsIndexableDeterminer composed of one or more IsIndexable condition classes.

    <virtualType name="Vendor\KlevuIndexingCustomEntities\Service\Determiner\IsIndexableDeterminer"
                 type="Klevu\Indexing\Service\Determiner\IsIndexableDeterminer">
        <arguments>
            <argument name="isIndexableConditions" xsi:type="array">
                <!-- Here you can add conditions covering entity status, published date, etc -->
                <item name="exampleIsIndexableCondition"
                      xsi:type="object">Vendor\KlevuIndexingCustomEntities\Service\Determiner\ExampleIsIndexableCondition</item>
            </argument>
        </arguments>
    </virtualType>

An example IsIndexableCondition can be found in the Github module.

Note: isIndexableConditions are evaluated on an OR basis, meaning if any one returns true then the record is considered indexable. If you need AND conditions, that should be implemented in a single condition class at a code level.

Create a Discovery Provider

Next, we need to configure and inject a discovery provider so the core actions can find and index our entities. Again, this can all be done via di.xml

    <virtualType name="Vendor\KlevuIndexingCustomEntities\Service\Provider\EntityDiscoveryProvider\Batched"
                 type="Klevu\Indexing\Service\Provider\EntityDiscoveryProvider">
        <arguments>
            <argument name="entityType" xsi:type="string">VENDOR_CUSTOM_ENTITY</argument>
            <argument name="entityProviderProvider"
                      xsi:type="object">Vendor\KlevuIndexingCustomEntities\Service\Provider\EntityProviderProvider\Batched</argument>
            <argument name="isIndexableDeterminer"
                      xsi:type="object">Vendor\KlevuIndexingCustomEntities\Service\Determiner\IsIndexableDeterminer</argument>
            <!-- If a custom entity can be excluded at a store level, we need this to be true -->
            <argument name="isCheckIsIndexableAtStoreScope" xsi:type="boolean">false</argument>
        </arguments>
    </virtualType>

    <virtualType name="Vendor\KlevuIndexingCustomEntities\Service\Provider\EntityDiscoveryProvider"
                 type="Klevu\Indexing\Service\Provider\EntityDiscoveryProvider">
        <arguments>
            <argument name="entityType" xsi:type="string">VENDOR_CUSTOM_ENTITY</argument>
            <argument name="entityProviderProvider"
                      xsi:type="object">Vendor\KlevuIndexingCustomEntities\Service\Provider\EntityProviderProvider</argument>
            <argument name="isIndexableDeterminer"
                      xsi:type="object">Vendor\KlevuIndexingCustomEntities\Service\Determiner\IsIndexableDeterminer</argument>
            <!-- If a custom entity can be excluded at a store level, we need this to be true -->
            <argument name="isCheckIsIndexableAtStoreScope" xsi:type="boolean">true</argument>
        </arguments>
    </virtualType>

    <type name="Klevu\Indexing\Service\EntityDiscoveryOrchestratorService">
        <arguments>
            <argument name="discoveryProviders" xsi:type="array">
                <item name="VENDOR_CUSTOM_ENTITY"
                      xsi:type="object">Vendor\KlevuIndexingCustomEntities\Service\Provider\EntityDiscoveryProvider\Batched</item>
            </argument>
        </arguments>
    </type>

Inject Custom Entity Discovery into Core Filters

Other than injecting into the Entity Discovery Orchestrator Service, we also need to add our provider to the indexing filter services.

    <type name="Klevu\Indexing\Service\FilterEntitiesToDeleteService">
        <arguments>
            <argument name="discoveryProviders" xsi:type="array">
                <item name="VENDOR_CUSTOM_ENTITY"
                      xsi:type="object">Vendor\KlevuIndexingCustomEntities\Service\Provider\EntityDiscoveryProvider</item>
            </argument>
        </arguments>
    </type>

    <type name="Klevu\Indexing\Service\FilterEntitiesToSetToIndexableService">
        <arguments>
            <argument name="discoveryProviders" xsi:type="array">
                <item name="VENDOR_CUSTOM_ENTITY"
                      xsi:type="object">Vendor\KlevuIndexingCustomEntities\Service\Provider\EntityDiscoveryProvider</item>
            </argument>
        </arguments>
    </type>

    <type name="Klevu\Indexing\Service\FilterEntitiesToSetToNotIndexableService">
        <arguments>
            <argument name="discoveryProviders" xsi:type="array">
                <item name="VENDOR_CUSTOM_ENTITY"
                      xsi:type="object">Vendor\KlevuIndexingCustomEntities\Service\Provider\EntityDiscoveryProvider</item>
            </argument>
        </arguments>
    </type>

Create an Entity Indexing Record Creator

Once we have our provider wired in, we need a service to convert those entities into Indexing Records via an implementation of the \Klevu\IndexingApi\Service\EntityIndexingRecordCreatorServiceInterface interface.

This class must contain an execute method which receives the record ID, an action (add, update, delete), the entity itself (which is typecast to PageInterface|ExtensibleDataInterface hence the custom entity mentioned at the start of the guide must implement one of these interfaces), and an option parent entity (again typecast).

Note: It's unlikely you will need to implement parent-child relations though, if required, you can reference the record creator in Klevu's IndexingProducts module, where this is used for configurable variants).

An example Creator service can be found in the Github module.

Add Indexing Pipeline Definitions

Next we'll take a slight detour into our pipeline definitions. There are three actions which need to be configured: Add, Update, and Delete. In most cases, however, Add and Update can use the same pipeline definitions as the payload and endpoint remain the same and PATCH support will not be introduced to this module.

By convention, pipeline YAML files live in etc/pipeline. For this module, we will not be nesting beneath the entity identifier (eg etc/pipeline/vendor_custom_entity, though if you are supporting multiple entities in a single module then this is recommended).

Note: We also use imports (again, just a convention) to separate out distinct parts of the pipeline and reduce duplication. For this example, we are not going to the extent of using separate files for individual attributes, but a complex example can be found in the core IndexingProducts module.

As such, we would expect to see YAML files for

Add / Update operations

Delete operations

Entity Transformation (for inclusion in the Add / Update file)

Examples of these files can be found in the sample module on Github.

A basic overview of the actions you will find in those pipelines:

Logging at the start and end of the process

Iteration over the provided batches

Iteration over records within the batch

Registration of some variables to context, to allow access deeper in the pipeline as required

Transformation of the custom entity object int JSON representation suitable for sync

Processing of the payload, including send to Klevu's APIs

Within the transformation stage, we use the same record structure as for any indexing API request. You can see more about this structure in the Data Indexing documentation.

Once you have created these files, you will need to register them with the core Pipeline Configuration Provider via di.xml, which allows our core indexing process to reference them when an action is initiated for your (or all) entities.

    <type name="Klevu\PlatformPipelines\Service\Provider\PipelineConfigurationProvider">
        <arguments>
            <argument name="pipelineConfigurationFilepaths" xsi:type="array">
                <item name="VENDOR_CUSTOM_ENTITY::add" xsi:type="string">Vendor_KlevuIndexingCustomEntities::etc/pipeline/add_update.yml</item>
                <item name="VENDOR_CUSTOM_ENTITY::delete" xsi:type="string">Vendor_KlevuIndexingCustomEntities::etc/pipeline/delete.yml</item>
                <item name="VENDOR_CUSTOM_ENTITY::update" xsi:type="string">Vendor_KlevuIndexingCustomEntities::etc/pipeline/add_update.yml</item>
            </argument>
        </arguments>
    </type>

Configure Indexing

Once you have the creator and pipeline definitions, you can configure the indexing record providers and inject them into the core indexer services. Again, this is all done using di.xml

Configure Indexing Record Providers

    <virtualType name="Vendor\KlevuIndexingCustomEntities\Service\Provider\Sync\EntityIndexingRecordProvider"
                 type="Klevu\Indexing\Service\Provider\Sync\EntityIndexingRecordProvider">
        <arguments>
            <argument name="entityProviderProvider"
                      xsi:type="object">Vendor\KlevuIndexingCustomEntities\Service\Provider\EntityProviderProvider</argument>
            <argument name="indexingRecordCreatorService"
                      xsi:type="object">Vendor\KlevuIndexingCustomEntities\Service\EntityIndexingRecordCreatorService</argument>
            <argument name="entityType" xsi:type="string">VENDOR_CUSTOM_ENTITY</argument>
        </arguments>
    </virtualType>

    <virtualType name="Vendor\KlevuIndexingCustomEntities\Service\Provider\Sync\EntityIndexingRecordProvider\Add"
                 type="Vendor\KlevuIndexingCustomEntities\Service\Provider\Sync\EntityIndexingRecordProvider">
        <arguments>
            <argument name="action" xsi:type="string">Add</argument>
        </arguments>
    </virtualType>

    <virtualType name="Vendor\KlevuIndexingCustomEntities\Service\Provider\Sync\EntityIndexingRecordProvider\Update"
                 type="Vendor\KlevuIndexingCustomEntities\Service\Provider\Sync\EntityIndexingRecordProvider">
        <arguments>
            <argument name="action" xsi:type="string">Update</argument>
        </arguments>
    </virtualType>

    <virtualType name="Vendor\KlevuIndexingCustomEntities\Service\Provider\Sync\EntityIndexingRecordProvider\Delete"
                 type="Vendor\KlevuIndexingCustomEntities\Service\Provider\Sync\EntityIndexingRecordProvider">
        <arguments>
            <argument name="action" xsi:type="string">Delete</argument>
        </arguments>
    </virtualType>

Create entity indexers

    <virtualType name="Vendor\KlevuIndexingCustomEntities\Service\EntityIndexerService\Add"
                 type="Klevu\Indexing\Service\EntityIndexerService">
        <arguments>
            <argument name="entityIndexingRecordProvider"
                      xsi:type="object">Vendor\KlevuIndexingCustomEntities\Service\Provider\Sync\EntityIndexingRecordProvider\Add</argument>
            <argument name="pipelineIdentifier" xsi:type="string">VENDOR_CUSTOM_ENTITY::add</argument>
        </arguments>
    </virtualType>

    <virtualType name="Vendor\KlevuIndexingCustomEntities\Service\EntityIndexerService\Delete"
                 type="Klevu\Indexing\Service\EntityIndexerService">
        <arguments>
            <argument name="entityIndexingRecordProvider"
                      xsi:type="object">Vendor\KlevuIndexingCustomEntities\Service\Provider\Sync\EntityIndexingRecordProvider\Delete</argument>
            <argument name="pipelineIdentifier" xsi:type="string">VENDOR_CUSTOM_ENTITY::delete</argument>
        </arguments>
    </virtualType>

    <virtualType name="Vendor\KlevuIndexingCustomEntities\Service\EntityIndexerService\Update"
                 type="Klevu\Indexing\Service\EntityIndexerService">
        <arguments>
            <argument name="entityIndexingRecordProvider"
                      xsi:type="object">Vendor\KlevuIndexingCustomEntities\Service\Provider\Sync\EntityIndexingRecordProvider\Update</argument>
            <argument name="pipelineIdentifier" xsi:type="string">VENDOR_CUSTOM_ENTITY::update</argument>
        </arguments>
    </virtualType>

Finally, hook up with the Entity Sync Orchestrator

    <type name="Klevu\Indexing\Service\EntitySyncOrchestratorService">
        <arguments>
            <argument name="entityIndexerServices" xsi:type="array">
                <item name="VENDOR_CUSTOM_ENTITY" xsi:type="array">
                    <item name="delete"
                          sortOrder="10"
                          xsi:type="object">Vendor\KlevuIndexingCustomEntities\Service\EntityIndexerService\Delete</item>
                    <item name="update"
                          sortOrder="20"
                          xsi:type="object">Vendor\KlevuIndexingCustomEntities\Service\EntityIndexerService\Update</item>
                    <item name="add"
                          sortOrder="30"
                          xsi:type="object">Vendor\KlevuIndexingCustomEntities\Service\EntityIndexerService\Add</item>
                </item>
            </argument>
        </arguments>
    </type>

Add Configuration to Enable/Disable Sync via Backend

Optionally, you can add the ability to disable sync for your custom entities via Stores > Configuration by configuring a Yes/No field in the backend (standard Magento customisation).

The core Klevu module contains a class to provide scoped configuration easily via di.xml configuration, which we recommend using.

    <virtualType name="Vendor\KlevuIndexingCustomEntities\Service\Provider\SyncEnabledProvider"
                 type="Klevu\Configuration\Service\Provider\ScopeConfigProvider">
        <arguments>
            <argument name="path" xsi:type="string">vendor/klevu_indexing_custom_entities/sync_enabled</argument>
            <argument name="returnType"
                      xsi:type="const">Klevu\Configuration\Service\Provider\ScopeConfigProvider::TYPE_BOOLEAN</argument>
        </arguments>
    </virtualType>

You can then use this in your custom entity provider to prevent return of any entities when sync is disabled.

class CustomEntityProvider implements EntityProviderInterface
{
    // snip
    private readonly ScopeConfigProviderInterface $syncEnabledProvider;

    public function __construct(
        // snip
        ScopeConfigProviderInterface $syncEnabledProvider,
    ) {
        // snip
        $this->syncEnabledProvider = $syncEnabledProvider;
    }

    /**
     * @param StoreInterface|null $store
     * @param int[]|null $entityIds
     *
     * @return \Generator<CustomEntityInterface[]>|null
     */
    public function get(
        ?StoreInterface $store = null,
        ?array $entityIds = [],
    ): ?\Generator {
        // Included to support disabling sync via configuration
        if (!$this->syncEnabledProvider->get()) {
            return null;
        }

    <type name="Vendor\KlevuIndexingCustomEntities\Service\Provider\CustomEntityProvider">
        <arguments>
           <!-- snip -->
            <argument name="syncEnabledProvider"
                      xsi:type="object">Vendor\KlevuIndexingCustomEntities\Service\Provider\SyncEnabledProvider</argument>
        </arguments>
    </type>

Responding to Changes

The implementation as detailed above (and in the example module) covers integrating custom entities into the core discovery and sync process. This means that that the provided klevu_indexing_discover_entities cron, and klevu:indexing:entity-discovery / klevu:indexing:entity-update CLI command will find your records for indexing; and that the klevu_indexing_sync_entities cron and klevu:indexing:entity-sync CLI command will send your records to Klevu.

The examples will not, however, respond to changes made during the course of day-to-day operations, for example record saves.

You will need to implement plugins or observers to listen for these changes and trigger an entity update in response.

These plugins may trigger the \Klevu\IndexingApi\Service\EntityUpdateResponderServiceInterface::execute via an implementation in your module.

A straightforward example of this can be found in the IndexingCms module either in the \Klevu\IndexingCms\Plugin\CmsPageResourceModelPlugin plugin or \Klevu\IndexingCms\Observer\CmsPageDeleteObserver observer. The responder service implementation can be found at \Klevu\IndexingCms\Service\EntityUpdateResponderService

Resources

Example module (Github)

IndexingProducts module (klevu/module-m2-indexing-products)

IndexingCms module (klevu/module-m2-indexing-cms)