Changing Attribute Data Sent to Klevu Indexes

tl;wr 1\ identify the yaml path(s) to the attribute you wish to modify eg stages iterateindexingrecordsbatch stages iterateindexingrecords stages processproduct stages virtualproduct stages attributes shortdescription 2\ determine the changes required to alter the attribute output eg change the source data from using short description to meta description , or append a string 3\ express the changes as yaml overrides eg, replacing just the specific extraction stage stages generatedata stages default getdefaultlanguagevalue stages getdata stages extract pipeline stage\extract args extraction currentproduct getmetadescription() transformations \ striptags(\["p", "br", "hr", "h1", "h2", "h3", "h4", "h5", "h6", "strong", "em", "ul", "ol", "li", "dl", "dt", "dd", "img", "sub", "sup", "small"], \["script"]) \ escapehtml \ trim or appending additional data stages generatedata stages default getdefaultlanguagevalue stages getdata addstages after validate transform pipeline stage\transform args transformation append(" this is a very special product ") 4\ implement the changes in a local or staging environment eg, add the yaml overrides under the identified attribute path in the var/klevu/indexing/pipeline/product/add update overrides yml file and set off a sync 5\ transfer changes to a dedicated customisation module, including creation of specific overrides files, and registration via di xml eg, stages iterateindexingrecordsbatch stages iterateindexingrecords stages processproduct stages virtualproduct stages attributes shortdescription stages generatedata stages default stages getdefaultlanguagevalue stages getdata stages extract import vendor klevuindexingoverrides etc/pipeline/indexing/product/attributes/short description/extract yml addstages after validate transform import vendor klevuindexingoverrides etc/pipeline/indexing/product/attributes/short description/transform yml di xml \<virtualtype name="klevu\indexingproducts\service\provider\pipelineconfigurationoverridesfilepathsprovider\add"> \<arguments> \<argument name="pipelineconfigurationoverridefilepaths" xsi\ type="array"> \<item name="vendor klevuindexingoverrides indexing product add update" xsi\ type="string">vendor klevuindexingoverrides etc/pipeline/indexing/product/add update overrides yml\</item> \</argument> \</arguments> \</virtualtype> \<virtualtype name="klevu\indexingproducts\service\provider\pipelineconfigurationoverridesfilepathsprovider\update"> \<arguments> \<argument name="pipelineconfigurationoverridefilepaths" xsi\ type="array"> \<item name="vendor klevuindexingoverrides indexing product add update" xsi\ type="string">vendor klevuindexingoverrides etc/pipeline/indexing/product/add update overrides yml\</item> \</argument> \</arguments> \</virtualtype> 6\ deploy the customisation module to production; requeue any affected products; and trigger an entity sync background the klevu magento integration uses a pipeline system where product records are fed in, converted into a json representation, pushed to klevu via api, and the result returned for processing by the application these pipelines are defined using one or more yaml files, compiled into a single set of instructions designed to be human readable and easily customisable you can view a comprehensive list of yaml config files used to generate all pipelines with the following command php bin/magento klevu\ pipelines\ configuration debug list files you can view the full, compiled configuration for any (indexing) pipeline using the following command (where \<pipelineidentifier> is replaced with the actual pipeline, eg klevu product update for the product update pipeline php bin/magento klevu\ indexing\ configuration dump pipeline \<pipelineidentifier> generated pipelines can be thousands of lines long we recommend piping the output into a temporary file in var to reference more easily this output is the most reliable way of identifying the paths required (including for custom attributes) to modify stages via overrides which we will need in this guide generation of all attribute data including extraction and transformation can be found in these pipelines, separated into various stages which can be removed or replaced with a small bespoke module these changes can also be incorporated into existing customisation modules, though we recommend keeping customisations separate to allow easier maintenance, debugging, and removal if required changes can be made directly to the autogenerated var/klevu/ /pipeline/ / overrides yml files (for example var/klevu/indexing/pipeline/product/add update overrides yml file, for product indexing add and update operations) while this is a valid way of quickly testing a change in a non production environment, be aware that any files in var are intentionally transitory and modifications should be moved to a permanent location within a separate extension before pushing to production identifying the yaml configuration path for this guide, we will be assuming you are changing a product attribute category and cms pipelines can be modified in the same manner, though you should use klevu category / klevu cms identifiers instead of klevu product the first step is to identify the full yaml path to the attribute you wish to modify we can do this by referencing the parent add update yml file and its included children, or generate a configuration dump and trace the path through the consolidated file (the more reliable option) when reading a path, we record the name of the stage and then its stages node, which tells the system that there are child pipelines to to be processed for example stages logstart pipeline indexing\stage\log # log that the pipeline has begun args message "start add/update products pipeline" iterateindexingrecordsbatch pipeline pipeline\iterate # loop over all provided batches args continueonexception stages iterateindexingrecords pipeline pipeline\iterate # loop over all provided records args continueonexception stages processindexingrecordstart # set some variables before processing product data this snippet gives us various paths, culminating in stages iterateindexingrecordsbatch stages iterateindexingrecords stages processindexingrecordstart more clearly illustrated as stages \# logstart \# pipeline indexing\stage\log # log that the pipeline has begun \# args \# message "start add/update products pipeline" iterateindexingrecordsbatch \# pipeline pipeline\iterate # loop over all provided batches \# args \# continueonexception stages iterateindexingrecords \# pipeline pipeline\iterate # loop over all provided records \# args \# continueonexception stages processindexingrecordstart # set some variables before processing product data we’re initially looking for the processproduct stage, which will perform different steps depending on the product type being processed at time of writing, this path is stages iterateindexingrecordsbatch stages iterateindexingrecords stages processproduct reference module m2 indexing products/etrc/pipeline/add update yml we will need to record the paths for each of the different product types that this modification will support (eg …stages variantproduct ) within each product, attribute output definitions are defined at stages attributes , with an identifier corresponding to the klevu attribute name (eg …stages shortdescription ) this stage should receive the current product object as its payload, which is passed through the defined child stages, and the output is the generated value which will be sent to klevu’s indexes as a rule, core attributes' definitions can be found in individual yaml files which are included into the main pipeline this allows us to re use the same steps for different product types, as well as giving an easy reference point if you ever need to check how klevu generates an attribute value reference module m2 indexing products/etc/pipeline/product/attributes before making our changes before we make our changes, a quick reminder of the different types of pipeline stage you may wish to use or replace stage\extract extracts a value from the passed payload (or, in many cases, from context such as grabbing an attribute value from the current product object) may contain transformations stage\transform passes the passed payload through one or more transformation functions (each transformation maps to a php class of the same name) stage\validate validates the passed payload against one or more conditions, throwing an exception on failure (which may be caught and handled, or left to break an iteration or the pipeline overall) pipeline\fallback a parent pipeline containing multiple stages intended to be validated and, on failure, fall back to the next stage also, a quick reminder of the override operations we can perform on an existing pipeline stage (this applies to any pipeline stage, not just those generating attribute values, and at any depth allowing us to target specific parts of an attribute generation if required remember, the more specific we can be, the smaller the overall impact of our customisation) add ( addstages ) insert new stages before or after a named target (for example, adding an additional transformation) delete ( removestages ) remove existing stages (for example, removing a case from a fallback pipeline) override straight up replace the targeted stage with a new definition implementing the customisation finally, we get to making a change, which involves the following steps identify the yaml path(s) to the attribute you wish to modify determine what changes we need to make implement and refine those changes in a staging environment transfer the changes to a customisation module and retest in a staging environment before deployment deploy changes, requeue affected products, and trigger an entity sync in our example, we are going to modify the way that the shortdescription attribute is generated for virtual and downloadable products only the changes we want to make are to use the meta description attribute, rather than short description, as our source value append the string “ this is a very special product ” at the end of the generated value if the meta description is not empty apply this change to virtual and downloadable product types identify the yaml path(s) to the attribute you wish to modify stages iterateindexingrecordsbatch stages iterateindexingrecords stages processproduct stages virtualproduct stages attributes shortdescription stages iterateindexingrecordsbatch stages iterateindexingrecords stages processproduct stages downloadableproduct stages attributes shortdescription if you don’t know how we got these paths, go re read the identifying the yaml configuration path section above reference module m2 indexing products/etc/pipeline/product/attributes/short description yml determine the changes required to alter the attribute output change the extraction from currentproduct getshortdescription() to currentproduct getmetadescription() add a stage\transform stage after the validation to append a value note as we are replacing the the extract stage, it might be tempting to add the append to the list of transformation arguments and save a change this, however, would make our validate stage worthless as every passed value would contain our appended string express the changes as yaml overrides to change the extraction, we need to replicate the entire extract stage definition as we can’t just replace a stage’s property taking the definition from the core code , replace getshortdescription() with getmetadescription() pipeline stage\extract args extraction currentproduct getmetadescription() transformations \ striptags(\["p", "br", "hr", "h1", "h2", "h3", "h4", "h5", "h6", "strong", "em", "ul", "ol", "li", "dl", "dt", "dd", "img", "sub", "sup", "small"], \["script"]) \ escapehtml \ trim to append data, we need a new stage of type stage\transform with a transformation of type append , passing in our desired string pipeline stage\transform args transformation append(" this is a very special product ") we’re using a static value here, but we could just as easily pass in a string from our context (for example, from the magento configuration) or the existing product using an extraction here eg pipeline stage\transform args transformation append(" ", $currentproduct getmetatitle()) implement the changes in a local or staging environment as we are making the changes on a staging (or local) environment, we can take advantage of the auto generated files which contain our custom attribute definitions by adding changes directly to (in this case) var/klevu/indexing/pipeline/product/add update overrides yml , we don’t need to worry about building a module just to test a theory, or recompiling as the file is already included to be safe, you should disable the automatic regeneration of configuration overrides while working to ensure your changes don’t suddenly disappear in the magento backend, set stores > configuration > klevu > developer settings > pipelines > enable autogeneration of configuration overrides to “no” and clear your cache first, replicate the paths identified above in yaml format note if you have autogenerated attribute information in the file, you can use the existing structure without having to remove or duplicate the paths (which would not be valid yaml) stages iterateindexingrecordsbatch stages iterateindexingrecords stages processproduct stages virtualproduct stages attributes shortdescription downloadableproduct stages attributes shortdescription and add the overrides required stages iterateindexingrecordsbatch stages iterateindexingrecords stages processproduct stages virtualproduct stages attributes shortdescription stages generatedata stages default getdefaultlanguagevalue stages getdata stages extract pipeline stage\extract args extraction currentproduct getmetadescription() # this is our change transformations \ striptags(\["p", "br", "hr", "h1", "h2", "h3", "h4", "h5", "h6", "strong", "em", "ul", "ol", "li", "dl", "dt", "dd", "img", "sub", "sup", "small"], \["script"]) \ escapehtml \ trim addstages after validate transform pipeline stage\transform args transformation append(" this is a very special product ") downloadableproduct \# implement the same for downloadable products now, queue one or more products for sync by updating the product data or via cli php bin/magento klevu\ indexing\ entity update entity types=klevu product entity ids=1,2,3 vvv and trigger a sync php bin/magento klevu\ indexing\ entity sync entity types=klevu product vvv we recommend enabling verbose indexing logging and archiving existing logs before triggering the test sync to make identifying the most recent data simpler transfer changes to a dedicated customisation module this guide assumes you are familiar with creating a magento 2 extension if you work better with code in front of you to reference, there is an example module at the end of the article important this example module is not production ready and should not be used it is for reference and illustration purposes only and no guarantee is made to its reliability we strongly recommend testing it in your own environment before deploying it to production we will use the module name vendor klevuindexingoverrides; you should choose a namespace (and module name) suited to your installation dependencies we will reference the klevu indexing products and php pipelines packages, so ensure you include the following in the extension’s composer json file composer json "require" { "php" " 8 1 0| 8 2 0| 8 3 0| 8 4 0", "klevu/module m2 indexing products" "^3 0 0", "klevu/php pipelines" "^1 1 0" }, and the following in the extension’s etc/module xml file etc/module xml \<sequence> \<module name="klevu indexingproducts"/> \<module name="klevu pipelines"/> \</sequence> pipeline yaml note, the framework isn’t opinionated about file locations, but we find the paths specified below are the most logical especially as modules grow and include additional overrides, especially for different record types we will create a parent overrides file at etc/pipeline/indexing/product/add update overrides yml the changes made above should be copied into this file, though to avoid duplication of code (especially if we need to make modifications to every product type), we will extract the actual changes to their own include files create these files at etc/pipeline/indexing/product/attributes/short description/extract yml etc/pipeline/indexing/product/attributes/short description/transform yml (if we were completely replacing the short description attribute’s definition, we could simply create short description yml , however keeping changes small and discrete means better maintainability in the future) our parent add update overrides yml file should look like add update overrides yml stages iterateindexingrecordsbatch stages iterateindexingrecords stages processproduct stages virtualproduct stages attributes shortdescription stages generatedata stages default stages getdefaultlanguagevalue stages getdata stages extract import vendor klevuindexingoverrides etc/pipeline/indexing/product/attributes/short description/extract yml addstages after validate transform import vendor klevuindexingoverrides etc/pipeline/indexing/product/attributes/short description/transform yml downloadableproduct stages attributes shortdescription stages generatedata stages default stages getdefaultlanguagevalue stages getdata stages extract import vendor klevuindexingoverrides etc/pipeline/indexing/product/attributes/short description/extract yml addstages after validate transform import vendor klevuindexingoverrides etc/pipeline/indexing/product/attributes/short description/transform yml extract yml should contain extract yml pipeline stage\extract args extraction currentproduct getmetadescription() # this is our change transformations \ striptags(\["p", "br", "hr", "h1", "h2", "h3", "h4", "h5", "h6", "strong", "em", "ul", "ol", "li", "dl", "dt", "dd", "img", "sub", "sup", "small"], \["script"]) \ escapehtml \ trim and transform yml transform yml pipeline stage\transform args transformation append(" this is a very special product ") defining the overrides now we need to tell the klevu module to use these override files by injecting the add update overrides yml filepath into the provider used when compiling our pipeline in your di xml , add di xml \<virtualtype name="klevu\indexingproducts\service\provider\pipelineconfigurationoverridesfilepathsprovider\add"> \<arguments> \<argument name="pipelineconfigurationoverridefilepaths" xsi\ type="array"> \<item name="vendor klevuindexingoverrides indexing product add update" xsi\ type="string">vendor klevuindexingoverrides etc/pipeline/indexing/product/add update overrides yml\</item> \</argument> \</arguments> \</virtualtype> \<virtualtype name="klevu\indexingproducts\service\provider\pipelineconfigurationoverridesfilepathsprovider\update"> \<arguments> \<argument name="pipelineconfigurationoverridefilepaths" xsi\ type="array"> \<item name="vendor klevuindexingoverrides indexing product add update" xsi\ type="string">vendor klevuindexingoverrides etc/pipeline/indexing/product/add update overrides yml\</item> \</argument> \</arguments> \</virtualtype> note that there are two providers one for klevu product add and klevu product update operations we have created a single pipeline overrides file for both operations, but if you need different values depending on whether a product is being sent to klevu for the first time or not, you could create different overrides files and inject them to the relevant providers as required compiling and testing as we made our changes in the existing overrides file originally, we cannot reliably test our new module until we have reverted these updates to do this, ensure you have re enabled the enable autogeneration of configuration overrides config setting and cleared your cache then, you can either modify an attribute through the magento backend or force a regeneration via cli php bin/magento klevu\ indexing\ configuration overrides regenerate entity type=klevu product vvv install and recompile magento with your new custom module then (again), queue one or more products for sync by updating the product data or via cli php bin/magento klevu\ indexing\ entity update entity types=klevu product entity ids=1,2,3 vvv and trigger a sync php bin/magento klevu\ indexing\ entity sync entity types=klevu product vvv deploy the customisation module to production providing everything has gone as expected, you can deploy your new module to your production environment and all future syncs will incorporate the changes to send updated data for your entire catalog, it is worth triggering an update via cli php bin/magento klevu\ indexing\ entity update entity types=klevu product entity ids=all you can then either trigger a sync on demand, or wait until the next scheduled process to run example module vendor klevuindexingoverrides further reading klevu custom attributes (indexing apis) attributes (klevu magento v4) add, update, or delete indexing attributes (klevu php sdk) managing attributes (klevu php sdk) transformers (klevu pipelines) validators (klevu pipelines)