--- title: Changing Attribute Data Sent to Klevu Indexes slug: klevu-magento-v4/changing-attribute-data-sent-to-klevu-indexes docTags: createdAt: 2025-07-20T17:24:08.193Z --- ## TL;WR 1\. Identify the YAML path(s) to the attribute you wish to modify Eg: `stages.iterateIndexingRecordsBatch.stages.iterateIndexingRecords.stages.processProduct.stages.virtualProduct.stages.attributes.shortDescription` 2\. Determine the changes required to alter the attribute output Eg: Change the source data from using `short_description` to `meta_description`, or append a string 3\. Express the changes as YAML overrides Eg, replacing *just* the specific extraction stage: ```yaml stages: generateData: stages: default: getDefaultLanguageValue: stages: getData: stages: extract: pipeline: Stage\Extract args: extraction: currentProduct::getMetaDescription() transformations: - StripTags(["p", "br", "hr", "h1", "h2", "h3", "h4", "h5", "h6", "strong", "em", "ul", "ol", "li", "dl", "dt", "dd", "img", "sub", "sup", "small"], ["script"]) - EscapeHtml - Trim ``` or appending additional data ```yaml stages: generateData: stages: default: getDefaultLanguageValue: stages: getData: addStages: after: validate transform: pipeline: Stage\Transform args: transformation: Append(" - This is a very special product.") ``` 4\. Implement the changes in a **local** or **staging** environment Eg, add the YAML overrides under the identified attribute path in the `var/klevu/indexing/pipeline/product/add_update.overrides.yml` file and set off a sync 5\. Transfer changes to a dedicated customisation module, including creation of specific overrides files, and registration via di.xml Eg, ```yaml stages: iterateIndexingRecordsBatch: stages: iterateIndexingRecords: stages: processProduct: stages: virtualProduct: stages: attributes: shortDescription: stages: generateData: stages: default: stages: getDefaultLanguageValue: stages: getData: stages: extract: import: Vendor_KlevuIndexingOverrides::etc/pipeline/indexing/product/attributes/short_description/extract.yml addStages: after: validate transform: import: Vendor_KlevuIndexingOverrides::etc/pipeline/indexing/product/attributes/short_description/transform.yml ``` :::CodeblockTabs di.xml ```xml Vendor_KlevuIndexingOverrides::etc/pipeline/indexing/product/add_update.overrides.yml Vendor_KlevuIndexingOverrides::etc/pipeline/indexing/product/add_update.overrides.yml ``` ::: 6\. Deploy the customisation module to production; requeue any affected products; and trigger an entity sync *** ## Background The Klevu Magento integration uses a pipeline system where product records are fed in, converted into a JSON representation, pushed to Klevu via API, and the result returned for processing by the application. These pipelines are defined using one or more YAML files, compiled into a single set of instructions designed to be human readable and easily customisable. :::hint{type="info"} You can view a comprehensive list of YAML config files used to generate all pipelines with the following command ```shell php bin/magento klevu:pipelines:configuration-debug list-files ``` ::: :::hint{type="info"} You can view the full, compiled configuration for any (indexing) pipeline using the following command (where `` is replaced with the actual pipeline, eg `KLEVU_PRODUCT::update` for the product update pipeline ```shell php bin/magento klevu:indexing:configuration-dump-pipeline ``` *Generated pipelines can be thousands of lines long - we recommend piping the output into a temporary file in *`var`* to reference more easily* This output is the most reliable way of identifying the paths required (including for custom attributes) to modify stages via overrides - which we will need in this guide ::: Generation of all attribute data - including extraction and transformation - can be found in these pipelines, separated into various stages which can be removed or replaced with a small bespoke module. :::hint{type="info"} These changes can also be incorporated into existing customisation modules, though we recommend keeping customisations separate to allow easier maintenance, debugging, and removal if required. ::: :::hint{type="warning"} Changes can be made directly to the autogenerated `var/klevu/*/pipeline/*/*.overrides.yml` files (for example `var/klevu/indexing/pipeline/product/add_update.overrides.yml` file, for product indexing add and update operations). While this is a valid way of quickly testing a change in a non-production environment, be aware that any files in var are intentionally transitory and modifications should be moved to a permanent location within a separate extension before pushing to production ::: ### Identifying the YAML Configuration Path :::hint{type="info"} For this guide, we will be assuming you are changing a product attribute. Category and CMS pipelines can be modified in the same manner, though you should use `KLEVU_CATEGORY` / `KLEVU_CMS` identifiers instead of `KLEVU_PRODUCT` . ::: The first step is to identify the full YAML path to the attribute you wish to modify. We can do this by referencing the parent add\_update.yml file and its included children, or generate a configuration dump and trace the path through the consolidated file (the more reliable option). :::hint{type="info"} When reading a path, we record the name of the stage and then its stages node, which tells the system that there are child pipelines to to be processed. For example: ```yaml stages: logStart: pipeline: Indexing\Stage\Log # log that the pipeline has begun args: message: "Start Add/Update Products Pipeline" iterateIndexingRecordsBatch: pipeline: Pipeline\Iterate # LOOP OVER ALL PROVIDED BATCHES args: continueOnException: ~ stages: iterateIndexingRecords: pipeline: Pipeline\Iterate # LOOP OVER ALL PROVIDED RECORDS args: continueOnException: ~ stages: processIndexingRecordStart: # set some variables before processing product data ``` This snippet gives us various paths, culminating in `stages.iterateIndexingRecordsBatch.stages.iterateIndexingRecords.stages.processIndexingRecordStart` More clearly illustrated as ```yaml stages: # logStart: # pipeline: Indexing\Stage\Log # log that the pipeline has begun # args: # message: "Start Add/Update Products Pipeline" iterateIndexingRecordsBatch: # pipeline: Pipeline\Iterate # LOOP OVER ALL PROVIDED BATCHES # args: # continueOnException: ~ stages: iterateIndexingRecords: # pipeline: Pipeline\Iterate # LOOP OVER ALL PROVIDED RECORDS # args: # continueOnException: ~ stages: processIndexingRecordStart: # set some variables before processing product data ``` ::: We’re initially looking for the `processProduct` stage, which will perform different steps depending on the product type being processed. At time of writing, this path is: `stages.iterateIndexingRecordsBatch.stages.iterateIndexingRecords.stages.processProduct` *Reference: [module-m2-indexing-products/etrc/pipeline/add\_update.yml](https://github.com/klevu/module-m2-indexing-products/blob/main/etc/pipeline/add_update.yml#L62)* We will need to record the paths for each of the different product types that this modification will support (eg `…stages.variantProduct`). Within each product, attribute output definitions are defined at `...stages.attributes`, with an identifier corresponding to the Klevu attribute name (eg `…stages.shortDescription`). This stage should receive the current product object as its payload, which is passed through the defined child stages, and the output is the generated value which will be sent to Klevu’s indexes. :::hint{type="info"} As a rule, core attributes' definitions can be found in individual YAML files which are included into the main pipeline. This allows us to re-use the same steps for different product types, as well as giving an easy reference point if you ever need to check how Klevu generates an attribute value. *Reference: [module-m2-indexing-products/etc/pipeline/product/attributes](https://github.com/klevu/module-m2-indexing-products/tree/main/etc/pipeline/product/attributes)* ::: ### Before Making Our Changes Before we make our changes, a quick reminder of the different types of pipeline stage you may wish to use or replace - `Stage\Extract` : extracts a value from the passed payload (or, in many cases, from context - such as grabbing an attribute value from the current product object) May contain transformations - `Stage\Transform` : passes the passed payload through one or more transformation functions (each transformation maps to a PHP class of the same name) - `Stage\Validate` : validates the passed payload against one or more conditions, throwing an exception on failure (which may be caught and handled, or left to break an iteration or the pipeline overall) - `Pipeline\Fallback` : a parent pipeline containing multiple stages intended to be validated and, on failure, fall back to the next stage Also, a quick reminder of the override operations we can perform on an existing pipeline stage (this applies to *any* pipeline stage, not just those generating attribute values, and at any depth - allowing us to target specific parts of an attribute generation if required. Remember, the more specific we can be, the smaller the overall impact of our customisation) - **Add** (`addStages`) : insert new stages before or after a named target (for example, adding an additional transformation) - **Delete** (`removeStages`) : remove existing stages (for example, removing a case from a fallback pipeline) - **Override** : Straight-up replace the targeted stage with a new definition *** ## Implementing the Customisation Finally, we get to making a change, which involves the following steps 1. Identify the YAML path(s) to the attribute you wish to modify 2. Determine what changes we need to make 3. Implement and refine those changes in a staging environment 4. Transfer the changes to a customisation module and retest in a staging environment before deployment 5. Deploy changes, requeue affected products, and trigger an entity sync :::hint{type="info"} In our example, we are going to modify the way that the shortDescription attribute is generated for virtual and downloadable products only. The changes we want to make are to - Use the meta description attribute, rather than short description, as our source value - Append the string “ - This is a very special product.” at the end of the generated value *if the meta description is not empty* - Apply this change to virtual and downloadable product types ::: ### Identify the YAML path(s) to the attribute you wish to modify - `stages.iterateIndexingRecordsBatch.stages.iterateIndexingRecords.stages.processProduct.stages.virtualProduct.stages.attributes.shortDescription` - `stages.iterateIndexingRecordsBatch.stages.iterateIndexingRecords.stages.processProduct.stages.downloadableProduct.stages.attributes.shortDescription` :::hint{type="info"} If you don’t know how we got these paths, go re-read the **Identifying the YAML Configuration Path** section above. *Reference: [module-m2-indexing-products/etc/pipeline/product/attributes/short\_description.yml](https://github.com/klevu/module-m2-indexing-products/blob/main/etc/pipeline/product/attributes/short_description.yml)* ::: ### Determine the changes required to alter the attribute output - **Change** the extraction from `currentProduct::getShortDescription()` to `currentProduct::getMetaDescription()` - **Add** a Stage\Transform stage after the validation to Append a value *Note: as we are replacing the the *extract* stage, it might be tempting to add the Append to the list of transformation arguments and save a change. This, however, would make our Validate stage worthless as every passed value would contain our appended string* ### Express the changes as YAML overrides To change the extraction, we need to replicate the entire extract stage definition as we can’t just replace a stage’s property. Taking the definition [from the core code](https://github.com/klevu/module-m2-indexing-products/blob/main/etc/pipeline/product/attributes/short_description.yml#L12), replace getShortDescription() with getMetaDescription() ```yaml pipeline: Stage\Extract args: extraction: currentProduct::getMetaDescription() transformations: - StripTags(["p", "br", "hr", "h1", "h2", "h3", "h4", "h5", "h6", "strong", "em", "ul", "ol", "li", "dl", "dt", "dd", "img", "sub", "sup", "small"], ["script"]) - EscapeHtml - Trim ``` To append data, we need a new stage of type `Stage\Transform` with a transformation of type [Append](https://github.com/klevu/php-pipelines/blob/master/src/Transformer/Append.php), passing in our desired string ```yaml pipeline: Stage\Transform args: transformation: Append(" - This is a very special product.") ``` :::hint{type="info"} We’re using a static value here, but we could just as easily pass in a string from our context (for example, from the Magento configuration) or the existing product using an extraction here. Eg ```yaml pipeline: Stage\Transform args: transformation: Append(" - ", $currentProduct.getMetaTitle()) ``` ::: ### Implement the changes in a local or staging environment As we are making the changes on a staging (or local) environment, we can take advantage of the auto-generated files which contain our custom attribute definitions. By adding changes directly to (in this case) `var/klevu/indexing/pipeline/product/add_update.overrides.yml`, we don’t need to worry about building a module just to test a theory, or recompiling as the file is already included. :::hint{type="warning"} To be safe, you should disable the automatic regeneration of configuration overrides while working to ensure your changes don’t suddenly disappear. In the Magento backend, set **Stores** > **Configuration** > **Klevu** > **Developer Settings** > **Pipelines** > **Enable autogeneration of configuration overrides **to “No” and clear your cache ::: First, replicate the paths identified above in YAML format. **Note:** if you have autogenerated attribute information in the file, you can use the existing structure without having to remove or duplicate the paths (which would not be valid YAML) ```yaml stages: iterateIndexingRecordsBatch: stages: iterateIndexingRecords: stages: processProduct: stages: virtualProduct: stages: attributes: shortDescription: downloadableProduct: stages: attributes: shortDescription: ``` and add the overrides required ```yaml stages: iterateIndexingRecordsBatch: stages: iterateIndexingRecords: stages: processProduct: stages: virtualProduct: stages: attributes: shortDescription: stages: generateData: stages: default: getDefaultLanguageValue: stages: getData: stages: extract: pipeline: Stage\Extract args: extraction: currentProduct::getMetaDescription() # This is our change transformations: - StripTags(["p", "br", "hr", "h1", "h2", "h3", "h4", "h5", "h6", "strong", "em", "ul", "ol", "li", "dl", "dt", "dd", "img", "sub", "sup", "small"], ["script"]) - EscapeHtml - Trim addStages: after: validate transform: pipeline: Stage\Transform args: transformation: Append(" - This is a very special product.") downloadableProduct: # Implement the same for downloadable products ``` Now, queue one or more products for sync by updating the product data or via CLI ```shell php bin/magento klevu:indexing:entity-update --entity-types=KLEVU_PRODUCT --entity-ids=1,2,3 -vvv ``` and trigger a sync ```shell php bin/magento klevu:indexing:entity-sync --entity-types=KLEVU_PRODUCT -vvv ``` :::hint{type="info"} We recommend enabling verbose indexing logging and archiving existing logs before triggering the test sync to make identifying the most recent data simpler ::: ### Transfer changes to a dedicated customisation module :::hint{type="info"} This guide assumes you are familiar with creating a Magento 2 extension. If you work better with code in front of you to reference, there is an example module at the end of the article ::: :::hint{type="warning"} **IMPORTANT: **This example module is not production-ready and should not be used. It is for reference and illustration purposes only and no guarantee is made to its reliability. We strongly recommend testing it in your own environment before deploying it to production. ::: We will use the module name Vendor\_KlevuIndexingOverrides; you should choose a namespace (and module name) suited to your installation. **Dependencies** We will reference the **Klevu Indexing Products** and **PHP-Pipelines** packages, so ensure you include the following in the extension’s `composer.json` file :::CodeblockTabs composer.json ```json "require": { "php": "~8.1.0|~8.2.0|~8.3.0|~8.4.0", "klevu/module-m2-indexing-products": "^3.0.0", "klevu/php-pipelines": "^1.1.0" }, ``` ::: and the following in the extension’s `etc/module.xml` file :::CodeblockTabs etc/module.xml ```xml ``` ::: **Pipeline YAML** :::hint{type="info"} Note, the framework isn’t opinionated about file locations, but we find the paths specified below are the most logical - especially as modules grow and include additional overrides, especially for different record types. ::: We will create a parent overrides file at - `etc/pipeline/indexing/product/add_update.overrides.yml` . The changes made above should be copied into this file, though to avoid duplication of code (especially if we need to make modifications to every product type), we will extract the actual changes to their own include files. Create these files at - `etc/pipeline/indexing/product/attributes/short_description/extract.yml` - `etc/pipeline/indexing/product/attributes/short_description/transform.yml` (if we were completely replacing the short\_description attribute’s definition, we could simply create `short_description.yml`, however keeping changes small and discrete means better maintainability in the future) Our parent `add_update.overrides.yml` file should look like :::CodeblockTabs add\_update.overrides.yml ```yaml stages: iterateIndexingRecordsBatch: stages: iterateIndexingRecords: stages: processProduct: stages: virtualProduct: stages: attributes: shortDescription: stages: generateData: stages: default: stages: getDefaultLanguageValue: stages: getData: stages: extract: import: Vendor_KlevuIndexingOverrides::etc/pipeline/indexing/product/attributes/short_description/extract.yml addStages: after: validate transform: import: Vendor_KlevuIndexingOverrides::etc/pipeline/indexing/product/attributes/short_description/transform.yml downloadableProduct: stages: attributes: shortDescription: stages: generateData: stages: default: stages: getDefaultLanguageValue: stages: getData: stages: extract: import: Vendor_KlevuIndexingOverrides::etc/pipeline/indexing/product/attributes/short_description/extract.yml addStages: after: validate transform: import: Vendor_KlevuIndexingOverrides::etc/pipeline/indexing/product/attributes/short_description/transform.yml ``` ::: `extract.yml` should contain :::CodeblockTabs extract.yml ```yaml pipeline: Stage\Extract args: extraction: currentProduct::getMetaDescription() # This is our change transformations: - StripTags(["p", "br", "hr", "h1", "h2", "h3", "h4", "h5", "h6", "strong", "em", "ul", "ol", "li", "dl", "dt", "dd", "img", "sub", "sup", "small"], ["script"]) - EscapeHtml - Trim ``` ::: and `transform.yml` :::CodeblockTabs transform.yml ```yaml pipeline: Stage\Transform args: transformation: Append(" - This is a very special product.") ``` ::: **Defining the Overrides** Now we need to tell the Klevu module to use these override files by injecting the `add_update.overrides.yml` filepath into the provider used when compiling our pipeline. In your `di.xml`, add :::CodeblockTabs di.xml ```xml Vendor_KlevuIndexingOverrides::etc/pipeline/indexing/product/add_update.overrides.yml Vendor_KlevuIndexingOverrides::etc/pipeline/indexing/product/add_update.overrides.yml ``` ::: :::hint{type="info"} Note that there are two providers - one for `KLEVU_PRODUCT::add` and `KLEVU_PRODUCT::update` operations. We have created a single pipeline overrides file for both operations, but if you need different values depending on whether a product is being sent to Klevu for the first time or not, you could create different overrides files and inject them to the relevant providers as required. ::: **Compiling and Testing** :::hint{type="warning"} As we made our changes in the existing overrides file originally, we cannot reliably test our new module until we have reverted these updates. To do this, ensure you have re-enabled the **Enable autogeneration of configuration overrides **config setting and cleared your cache. Then, you can either modify an attribute through the Magento backend or force a regeneration via CLI ```shell php bin/magento klevu:indexing:configuration-overrides-regenerate --entity-type=KLEVU_PRODUCT -vvv ``` ::: Install and recompile Magento with your new custom module. Then (again), queue one or more products for sync by updating the product data or via CLI ```shell php bin/magento klevu:indexing:entity-update --entity-types=KLEVU_PRODUCT --entity-ids=1,2,3 -vvv ``` and trigger a sync ```shell php bin/magento klevu:indexing:entity-sync --entity-types=KLEVU_PRODUCT -vvv ``` ### Deploy the customisation module to production Providing everything has gone as expected, you can deploy your new module to your production environment and all future syncs will incorporate the changes. To send updated data for your entire catalog, it is worth triggering an update via CLI ```shell php bin/magento klevu:indexing:entity-update --entity-types=KLEVU_PRODUCT --entity-ids=all ``` You can then either trigger a sync on demand, or wait until the next scheduled process to run. ## Example Module - [Vendor\_KlevuIndexingOverrides](https://github.com/klevultd/m2-customisations/tree/main/Vendor_KlevuIndexingOverrides) ## Further Reading - [Klevu Custom Attributes](https://docs.klevu.com/indexing-apis/custom-attributes) (Indexing APIs) - [Attributes](https://docs.klevu.com/klevu-magento-v4/attributes) (Klevu Magento v4) - [Add, Update, or Delete Indexing Attributes](https://docs.klevu.com/php-sdk/add-update-delete) (Klevu PHP SDK) - [Managing Attributes](https://docs.klevu.com/php-sdk/managing-attributes) (Klevu PHP SDK) - [Transformers](https://github.com/klevu/php-pipelines/tree/master/src/Transformer) (Klevu Pipelines) - [Validators](https://github.com/klevu/php-pipelines/tree/master/src/Validator) (Klevu Pipelines)