Indexing Custom Entities

note this guide covers integration of entities with a custom entity type, not the injection of additional entities into existing entity type sync (for example, sending blog posts as klevu cms alongside cms pages) note this guide covers integration of entities with a custom entity type, not the injection of additional entities into existing entity type sync (for example, sending blog posts as klevu cms alongside cms pages) beyond the standard product, category, and cms page enities, the klevu indexes support sync and return of custom entity types ultimately, all records are stored in the same "bucket", with the same core structure and attributes behaviour (such as in search) is then modified using filters and actions referencing the entity type natively klevu product; klevu category; and klevu cms what this means is that we can create and sync any "type" of entity with the klevu indexes, providing it has the same basic data points and a unique entity type identifier frontend customisations can then handle search and display to allow integration seamlessly into your site implementing in magento version 4 x of the klevu magento integration is designed with extensibility in mind including the ability to inject different entity types into the indexing discovery and sync operations this is the same extensibility used within the module itself to manage products, categories, and cms so we can follow the same steps to handle other entities note the creation and management of these entities is the responsibility of the integrator, though may include entities (such as blog posts) from other third party extensions the key requirement for integration with klevu's indexing modules is that the entity implements either extensibledatainterface or pageinterface if integrating a third party entity which does not implement one of these interfaces, you may want need to create a custom version of the class this is out of the scope of this guide note the creation and management of these entities is the responsibility of the integrator, though may include entities (such as blog posts) from other third party extensions the key requirement for integration with klevu's indexing modules is that the entity implements either extensibledatainterface or pageinterface if integrating a third party entity which does not implement one of these interfaces, you may want need to create a custom version of the class this is out of the scope of this guide a https //github com/klevultd/m2 customisations/blob/main/vendor klevuindexingcustomentities create an entity provider first we need to create a service to provide those custom entities to the indexing module this class must implement the \klevu\indexingapi\service\provider\entityproviderinterface interface, and contain a get method which returns a \generator yielding arrays of custom entities to support batching the class must also contain a getentitysubtype method, which often will simply contain a string of your choice this method is primarily used to support indexing of entities which have different subtypes, such as products (configurable, simple, variants, etc) an https //github com/klevultd/m2 customisations/blob/main/vendor klevuindexingcustomentities/service/provider/customentityprovider php on github once you have created the class, you will need to wire it in using dependency injection you will need to create both a standard and "batched" version of the provider, though this can be achieved by simply creating a virtualtype of the concrete class \<type name="vendor\klevuindexingcustomentities\service\provider\customentityprovider"> \<arguments> \<argument name="logger" xsi\ type="object">klevu\indexing\logger\logger\</argument> \<argument name="batchsize" xsi\ type="const">klevu\indexing\constants default indexing batch size\</argument> \<argument name="batchsizevalidator" xsi\ type="object">klevu\indexing\validator\batchsizevalidator\</argument> \</arguments> \</type> \<virtualtype name="vendor\klevuindexingcustomentities\service\provider\customentityprovider\batched" type="vendor\klevuindexingcustomentities\service\provider\customentityprovider" /> you will also need to create implementations of the klevu\indexing\service\provider\entityproviderprovider class and inject your custom provider(s if you are syncing multiple custom entity types via the same module) this can be done using a virtualtype as follows \<virtualtype name="vendor\klevuindexingcustomentities\service\provider\entityproviderprovider" type="klevu\indexing\service\provider\entityproviderprovider"> \<arguments> \<argument name="entityproviders" xsi\ type="array"> \<item name="vendor custom entity" xsi\ type="object">vendor\klevuindexingcustomentities\service\provider\customentityprovider\</item> \</argument> \</arguments> \</virtualtype> \<virtualtype name="vendor\klevuindexingcustomentities\service\provider\entityproviderprovider\batched" type="klevu\indexing\service\provider\entityproviderprovider"> \<arguments> \<argument name="entityproviders" xsi\ type="array"> \<item name="vendor custom entity" xsi\ type="object">vendor\klevuindexingcustomentities\service\provider\customentityprovider\batched\</item> \</argument> \</arguments> \</virtualtype> note the item name should correspond to the entity type identifier you will use for sync (eg, for klevu, klevu product , klevu category , and klevu cms ) incorporate is indexable conditions as part of entity provision, we also need to tell the core module how to decide if an entity is indexable (think disabled products, or those assigned to different websites) the determiner itself can be a virtualtype of klevu\indexing\service\determiner\isindexabledeterminer composed of one or more isindexable condition classes \<virtualtype name="vendor\klevuindexingcustomentities\service\determiner\isindexabledeterminer" type="klevu\indexing\service\determiner\isindexabledeterminer"> \<arguments> \<argument name="isindexableconditions" xsi\ type="array"> \<! here you can add conditions covering entity status, published date, etc > \<item name="exampleisindexablecondition" xsi\ type="object">vendor\klevuindexingcustomentities\service\determiner\exampleisindexablecondition\</item> \</argument> \</arguments> \</virtualtype> an https //github com/klevultd/m2 customisations/blob/main/vendor klevuindexingcustomentities/service/determiner/exampleisindexablecondition php can be found in the github module note isindexableconditions are evaluated on an or basis, meaning if any one returns true then the record is considered indexable if you need and conditions, that should be implemented in a single condition class at a code level create a discovery provider next, we need to configure and inject a discovery provider so the core actions can find and index our entities again, this can all be done via di xml \<virtualtype name="vendor\klevuindexingcustomentities\service\provider\entitydiscoveryprovider\batched" type="klevu\indexing\service\provider\entitydiscoveryprovider"> \<arguments> \<argument name="entitytype" xsi\ type="string">vendor custom entity\</argument> \<argument name="entityproviderprovider" xsi\ type="object">vendor\klevuindexingcustomentities\service\provider\entityproviderprovider\batched\</argument> \<argument name="isindexabledeterminer" xsi\ type="object">vendor\klevuindexingcustomentities\service\determiner\isindexabledeterminer\</argument> \<! if a custom entity can be excluded at a store level, we need this to be true > \<argument name="ischeckisindexableatstorescope" xsi\ type="boolean">false\</argument> \</arguments> \</virtualtype> \<virtualtype name="vendor\klevuindexingcustomentities\service\provider\entitydiscoveryprovider" type="klevu\indexing\service\provider\entitydiscoveryprovider"> \<arguments> \<argument name="entitytype" xsi\ type="string">vendor custom entity\</argument> \<argument name="entityproviderprovider" xsi\ type="object">vendor\klevuindexingcustomentities\service\provider\entityproviderprovider\</argument> \<argument name="isindexabledeterminer" xsi\ type="object">vendor\klevuindexingcustomentities\service\determiner\isindexabledeterminer\</argument> \<! if a custom entity can be excluded at a store level, we need this to be true > \<argument name="ischeckisindexableatstorescope" xsi\ type="boolean">true\</argument> \</arguments> \</virtualtype> \<type name="klevu\indexing\service\entitydiscoveryorchestratorservice"> \<arguments> \<argument name="discoveryproviders" xsi\ type="array"> \<item name="vendor custom entity" xsi\ type="object">vendor\klevuindexingcustomentities\service\provider\entitydiscoveryprovider\batched\</item> \</argument> \</arguments> \</type> inject custom entity discovery into core filters other than injecting into the entity discovery orchestrator service, we also need to add our provider to the indexing filter services \<type name="klevu\indexing\service\filterentitiestodeleteservice"> \<arguments> \<argument name="discoveryproviders" xsi\ type="array"> \<item name="vendor custom entity" xsi\ type="object">vendor\klevuindexingcustomentities\service\provider\entitydiscoveryprovider\</item> \</argument> \</arguments> \</type> \<type name="klevu\indexing\service\filterentitiestosettoindexableservice"> \<arguments> \<argument name="discoveryproviders" xsi\ type="array"> \<item name="vendor custom entity" xsi\ type="object">vendor\klevuindexingcustomentities\service\provider\entitydiscoveryprovider\</item> \</argument> \</arguments> \</type> \<type name="klevu\indexing\service\filterentitiestosettonotindexableservice"> \<arguments> \<argument name="discoveryproviders" xsi\ type="array"> \<item name="vendor custom entity" xsi\ type="object">vendor\klevuindexingcustomentities\service\provider\entitydiscoveryprovider\</item> \</argument> \</arguments> \</type> create an entity indexing record creator once we have our provider wired in, we need a service to convert those entities into indexing records via an implementation of the \klevu\indexingapi\service\entityindexingrecordcreatorserviceinterface interface this class must contain an execute method which receives the record id, an action (add, update, delete), the entity itself (which is typecast to pageinterface|extensibledatainterface hence the custom entity mentioned at the start of the guide must implement one of these interfaces), and an option parent entity (again typecast) note it's unlikely you will need to implement parent child relations though, if required, you can reference the record creator in klevu's indexingproducts module, where this is used for configurable variants) an https //github com/klevultd/m2 customisations/blob/main/vendor klevuindexingcustomentities/service/entityindexingrecordcreatorservice php can be found in the github module add indexing pipeline definitions next we'll take a slight detour into our pipeline definitions there are three actions which need to be configured add, update, and delete in most cases, however, add and update can use the same pipeline definitions as the payload and endpoint remain the same and patch support will not be introduced to this module by convention, pipeline yaml files live in etc/pipeline for this module, we will not be nesting beneath the entity identifier (eg etc/pipeline/vendor custom entity , though if you are supporting multiple entities in a single module then this is recommended) note we also use imports (again, just a convention) to separate out distinct parts of the pipeline and reduce duplication for this example, we are not going to the extent of using separate files for individual attributes, but a complex example can be found in the core https //github com/klevu/module m2 indexing products/ module as such, we would expect to see yaml files for add / update operations delete operations entity transformation (for inclusion in the add / update file) https //github com/klevultd/m2 customisations/tree/main/vendor klevuindexingcustomentities/etc/pipeline can be found in the sample module on github a basic overview of the actions you will find in those pipelines logging at the start and end of the process iteration over the provided batches iteration over records within the batch registration of some variables to context, to allow access deeper in the pipeline as required transformation of the custom entity object int json representation suitable for sync processing of the payload, including send to klevu's apis within the transformation stage, we use the same record structure as for any indexing api request you can see more about this structure in the https //docs klevu com/indexing apis/overview#record structure once you have created these files, you will need to register them with the core pipeline configuration provider via di xml , which allows our core indexing process to reference them when an action is initiated for your (or all) entities \<type name="klevu\platformpipelines\service\provider\pipelineconfigurationprovider"> \<arguments> \<argument name="pipelineconfigurationfilepaths" xsi\ type="array"> \<item name="vendor custom entity add" xsi\ type="string">vendor klevuindexingcustomentities etc/pipeline/add update yml\</item> \<item name="vendor custom entity delete" xsi\ type="string">vendor klevuindexingcustomentities etc/pipeline/delete yml\</item> \<item name="vendor custom entity update" xsi\ type="string">vendor klevuindexingcustomentities etc/pipeline/add update yml\</item> \</argument> \</arguments> \</type> configure indexing once you have the creator and pipeline definitions, you can configure the indexing record providers and inject them into the core indexer services again, this is all done using di xml configure indexing record providers \<virtualtype name="vendor\klevuindexingcustomentities\service\provider\sync\entityindexingrecordprovider" type="klevu\indexing\service\provider\sync\entityindexingrecordprovider"> \<arguments> \<argument name="entityproviderprovider" xsi\ type="object">vendor\klevuindexingcustomentities\service\provider\entityproviderprovider\</argument> \<argument name="indexingrecordcreatorservice" xsi\ type="object">vendor\klevuindexingcustomentities\service\entityindexingrecordcreatorservice\</argument> \<argument name="entitytype" xsi\ type="string">vendor custom entity\</argument> \</arguments> \</virtualtype> \<virtualtype name="vendor\klevuindexingcustomentities\service\provider\sync\entityindexingrecordprovider\add" type="vendor\klevuindexingcustomentities\service\provider\sync\entityindexingrecordprovider"> \<arguments> \<argument name="action" xsi\ type="string">add\</argument> \</arguments> \</virtualtype> \<virtualtype name="vendor\klevuindexingcustomentities\service\provider\sync\entityindexingrecordprovider\update" type="vendor\klevuindexingcustomentities\service\provider\sync\entityindexingrecordprovider"> \<arguments> \<argument name="action" xsi\ type="string">update\</argument> \</arguments> \</virtualtype> \<virtualtype name="vendor\klevuindexingcustomentities\service\provider\sync\entityindexingrecordprovider\delete" type="vendor\klevuindexingcustomentities\service\provider\sync\entityindexingrecordprovider"> \<arguments> \<argument name="action" xsi\ type="string">delete\</argument> \</arguments> \</virtualtype> create entity indexers \<virtualtype name="vendor\klevuindexingcustomentities\service\entityindexerservice\add" type="klevu\indexing\service\entityindexerservice"> \<arguments> \<argument name="entityindexingrecordprovider" xsi\ type="object">vendor\klevuindexingcustomentities\service\provider\sync\entityindexingrecordprovider\add\</argument> \<argument name="pipelineidentifier" xsi\ type="string">vendor custom entity add\</argument> \</arguments> \</virtualtype> \<virtualtype name="vendor\klevuindexingcustomentities\service\entityindexerservice\delete" type="klevu\indexing\service\entityindexerservice"> \<arguments> \<argument name="entityindexingrecordprovider" xsi\ type="object">vendor\klevuindexingcustomentities\service\provider\sync\entityindexingrecordprovider\delete\</argument> \<argument name="pipelineidentifier" xsi\ type="string">vendor custom entity delete\</argument> \</arguments> \</virtualtype> \<virtualtype name="vendor\klevuindexingcustomentities\service\entityindexerservice\update" type="klevu\indexing\service\entityindexerservice"> \<arguments> \<argument name="entityindexingrecordprovider" xsi\ type="object">vendor\klevuindexingcustomentities\service\provider\sync\entityindexingrecordprovider\update\</argument> \<argument name="pipelineidentifier" xsi\ type="string">vendor custom entity update\</argument> \</arguments> \</virtualtype> finally, hook up with the entity sync orchestrator \<type name="klevu\indexing\service\entitysyncorchestratorservice"> \<arguments> \<argument name="entityindexerservices" xsi\ type="array"> \<item name="vendor custom entity" xsi\ type="array"> \<item name="delete" sortorder="10" xsi\ type="object">vendor\klevuindexingcustomentities\service\entityindexerservice\delete\</item> \<item name="update" sortorder="20" xsi\ type="object">vendor\klevuindexingcustomentities\service\entityindexerservice\update\</item> \<item name="add" sortorder="30" xsi\ type="object">vendor\klevuindexingcustomentities\service\entityindexerservice\add\</item> \</item> \</argument> \</arguments> \</type> add configuration to enable/disable sync via backend optionally, you can add the ability to disable sync for your custom entities via stores > configuration by configuring a yes/no field in the backend (standard magento customisation) the core klevu module contains a class to provide scoped configuration easily via di xml configuration, which we recommend using \<virtualtype name="vendor\klevuindexingcustomentities\service\provider\syncenabledprovider" type="klevu\configuration\service\provider\scopeconfigprovider"> \<arguments> \<argument name="path" xsi\ type="string">vendor/klevu indexing custom entities/sync enabled\</argument> \<argument name="returntype" xsi\ type="const">klevu\configuration\service\provider\scopeconfigprovider type boolean\</argument> \</arguments> \</virtualtype> you can then use this in your custom entity provider to prevent return of any entities when sync is disabled class customentityprovider implements entityproviderinterface { // snip private readonly scopeconfigproviderinterface $syncenabledprovider; public function construct( // snip scopeconfigproviderinterface $syncenabledprovider, ) { // snip $this >syncenabledprovider = $syncenabledprovider; } / @param storeinterface|null $store @param int\[]|null $entityids @return \generator\<customentityinterface\[]>|null / public function get( ?storeinterface $store = null, ?array $entityids = \[], ) ?\generator { // included to support disabling sync via configuration if (!$this >syncenabledprovider >get()) { return null; } \<type name="vendor\klevuindexingcustomentities\service\provider\customentityprovider"> \<arguments> \<! snip > \<argument name="syncenabledprovider" xsi\ type="object">vendor\klevuindexingcustomentities\service\provider\syncenabledprovider\</argument> \</arguments> \</type> responding to changes the implementation as detailed above (and in the example module) covers integrating custom entities into the core discovery and sync process this means that that the provided klevu indexing discover entities cron, and klevu\ indexing\ entity discovery / klevu\ indexing\ entity update cli command will find your records for indexing; and that the klevu indexing sync entities cron and klevu\ indexing\ entity sync cli command will send your records to klevu the examples will not, however, respond to changes made during the course of day to day operations, for example record saves you will need to implement plugins or observers to listen for these changes and trigger an entity update in response these plugins may trigger the \klevu\indexingapi\service\entityupdateresponderserviceinterface execute via an implementation in your module a straightforward example of this can be found in the https //github com/klevu/module m2 indexing cms/ module either in the \klevu\indexingcms\plugin\cmspageresourcemodelplugin plugin or \klevu\indexingcms\observer\cmspagedeleteobserver observer the responder service implementation can be found at \klevu\indexingcms\service\entityupdateresponderservice resources https //github com/klevultd/m2 customisations/blob/main/vendor klevuindexingcustomentities https //github com/klevu/module m2 indexing products/ https //github com/klevu/module m2 indexing cms/