Changing Attribute Data Sent to Klevu Indexes

33 min

TL;WR

1. Identify the YAML path(s) to the attribute you wish to modify Eg: stages.iterateIndexingRecordsBatch.stages.iterateIndexingRecords.stages.processProduct.stages.virtualProduct.stages.attributes.shortDescription

2. Determine the changes required to alter the attribute output Eg: Change the source data from using short_description to meta_description, or append a string

3. Express the changes as YAML overrides Eg, replacing just the specific extraction stage:

stages:
  generateData:
    stages:
      default:
        getDefaultLanguageValue:
          stages:
            getData:
              stages:
                extract:
                  pipeline: Stage\Extract
                  args:
                    extraction: currentProduct::getMetaDescription()
                    transformations:
                      - StripTags(["p", "br", "hr", "h1", "h2", "h3", "h4", "h5", "h6", "strong", "em", "ul", "ol", "li", "dl", "dt", "dd", "img", "sub", "sup", "small"], ["script"])
                      - EscapeHtml
                      - Trim

or appending additional data

stages:
  generateData:
    stages:
      default:
        getDefaultLanguageValue:
          stages:
            getData:
              addStages:
                after: validate
                transform:
                  pipeline: Stage\Transform
                  args:
                    transformation: Append(" - This is a very special product.")

4. Implement the changes in a local or staging environment Eg, add the YAML overrides under the identified attribute path in the var/klevu/indexing/pipeline/product/add_update.overrides.yml file and set off a sync

5. Transfer changes to a dedicated customisation module, including creation of specific overrides files, and registration via di.xml Eg,

stages:
  iterateIndexingRecordsBatch:
    stages:
      iterateIndexingRecords:
        stages:
          processProduct:
            stages:
              virtualProduct:
                stages:
                  attributes:
                    shortDescription:
                      stages:
                        generateData:
                          stages:
                            default:
                              stages:
                                getDefaultLanguageValue:
                                  stages:
                                    getData:
                                      stages:
                                        extract:
                                          import: Vendor_KlevuIndexingOverrides::etc/pipeline/indexing/product/attributes/short_description/extract.yml
                                      addStages:
                                        after: validate
                                        transform:
                                          import: Vendor_KlevuIndexingOverrides::etc/pipeline/indexing/product/attributes/short_description/transform.yml

<virtualType name="Klevu\IndexingProducts\Service\Provider\PipelineConfigurationOverridesFilepathsProvider\Add">
    <arguments>
        <argument name="pipelineConfigurationOverrideFilepaths" xsi:type="array">
            <item name="vendor_klevuindexingoverrides_indexing_product_add_update"
                  xsi:type="string">Vendor_KlevuIndexingOverrides::etc/pipeline/indexing/product/add_update.overrides.yml</item>
        </argument>
    </arguments>
</virtualType>
<virtualType name="Klevu\IndexingProducts\Service\Provider\PipelineConfigurationOverridesFilepathsProvider\Update">
    <arguments>
        <argument name="pipelineConfigurationOverrideFilepaths" xsi:type="array">
            <item name="vendor_klevuindexingoverrides_indexing_product_add_update"
                  xsi:type="string">Vendor_KlevuIndexingOverrides::etc/pipeline/indexing/product/add_update.overrides.yml</item>
        </argument>
    </arguments>
</virtualType>

6. Deploy the customisation module to production; requeue any affected products; and trigger an entity sync

Background

The Klevu Magento integration uses a pipeline system where product records are fed in, converted into a JSON representation, pushed to Klevu via API, and the result returned for processing by the application.

These pipelines are defined using one or more YAML files, compiled into a single set of instructions designed to be human readable and easily customisable.

You can view a comprehensive list of YAML config files used to generate all pipelines with the following command

php bin/magento klevu:pipelines:configuration-debug list-files

You can view the full, compiled configuration for any (indexing) pipeline using the following command (where <pipelineIdentifier> is replaced with the actual pipeline, eg KLEVU_PRODUCT::update for the product update pipeline

php bin/magento klevu:indexing:configuration-dump-pipeline <pipelineIdentifier>

Generated pipelines can be thousands of lines long - we recommend piping the output into a temporary file in var to reference more easily

This output is the most reliable way of identifying the paths required (including for custom attributes) to modify stages via overrides - which we will need in this guide

Generation of all attribute data - including extraction and transformation - can be found in these pipelines, separated into various stages which can be removed or replaced with a small bespoke module.

These changes can also be incorporated into existing customisation modules, though we recommend keeping customisations separate to allow easier maintenance, debugging, and removal if required.

Changes can be made directly to the autogenerated var/klevu/*/pipeline/*/*.overrides.yml files (for example var/klevu/indexing/pipeline/product/add_update.overrides.yml file, for product indexing add and update operations).

While this is a valid way of quickly testing a change in a non-production environment, be aware that any files in var are intentionally transitory and modifications should be moved to a permanent location within a separate extension before pushing to production

Identifying the YAML Configuration Path

For this guide, we will be assuming you are changing a product attribute. Category and CMS pipelines can be modified in the same manner, though you should use KLEVU_CATEGORY / KLEVU_CMS identifiers instead of KLEVU_PRODUCT .

The first step is to identify the full YAML path to the attribute you wish to modify. We can do this by referencing the parent add_update.yml file and its included children, or generate a configuration dump and trace the path through the consolidated file (the more reliable option).

When reading a path, we record the name of the stage and then its stages node, which tells the system that there are child pipelines to to be processed.

For example:

stages:
  logStart:
    pipeline: Indexing\Stage\Log # log that the pipeline has begun
    args:
      message: "Start Add/Update Products Pipeline"
  iterateIndexingRecordsBatch:
    pipeline: Pipeline\Iterate # LOOP OVER ALL PROVIDED BATCHES
    args:
      continueOnException: ~
    stages:
      iterateIndexingRecords:
        pipeline: Pipeline\Iterate # LOOP OVER ALL PROVIDED RECORDS
        args:
          continueOnException: ~
        stages:
          processIndexingRecordStart: # set some variables before processing product data

This snippet gives us various paths, culminating in stages.iterateIndexingRecordsBatch.stages.iterateIndexingRecords.stages.processIndexingRecordStart More clearly illustrated as

stages:
#  logStart:
#    pipeline: Indexing\Stage\Log # log that the pipeline has begun
#    args:
#      message: "Start Add/Update Products Pipeline"
  iterateIndexingRecordsBatch:
#    pipeline: Pipeline\Iterate # LOOP OVER ALL PROVIDED BATCHES
#    args:
#      continueOnException: ~
    stages:
      iterateIndexingRecords:
#        pipeline: Pipeline\Iterate # LOOP OVER ALL PROVIDED RECORDS
#        args:
#          continueOnException: ~
        stages:
          processIndexingRecordStart: # set some variables before processing product data

We’re initially looking for the processProduct stage, which will perform different steps depending on the product type being processed. At time of writing, this path is: stages.iterateIndexingRecordsBatch.stages.iterateIndexingRecords.stages.processProduct Reference: module-m2-indexing-products/etrc/pipeline/add_update.yml

We will need to record the paths for each of the different product types that this modification will support (eg …stages.variantProduct).

Within each product, attribute output definitions are defined at ...stages.attributes, with an identifier corresponding to the Klevu attribute name (eg …stages.shortDescription). This stage should receive the current product object as its payload, which is passed through the defined child stages, and the output is the generated value which will be sent to Klevu’s indexes.

As a rule, core attributes' definitions can be found in individual YAML files which are included into the main pipeline. This allows us to re-use the same steps for different product types, as well as giving an easy reference point if you ever need to check how Klevu generates an attribute value.

Reference: module-m2-indexing-products/etc/pipeline/product/attributes

Before Making Our Changes

Before we make our changes, a quick reminder of the different types of pipeline stage you may wish to use or replace

Stage\Extract : extracts a value from the passed payload (or, in many cases, from context - such as grabbing an attribute value from the current product object) May contain transformations
Stage\Transform : passes the passed payload through one or more transformation functions (each transformation maps to a PHP class of the same name)
Stage\Validate : validates the passed payload against one or more conditions, throwing an exception on failure (which may be caught and handled, or left to break an iteration or the pipeline overall)
Pipeline\Fallback : a parent pipeline containing multiple stages intended to be validated and, on failure, fall back to the next stage

Also, a quick reminder of the override operations we can perform on an existing pipeline stage (this applies to any pipeline stage, not just those generating attribute values, and at any depth - allowing us to target specific parts of an attribute generation if required. Remember, the more specific we can be, the smaller the overall impact of our customisation)

Add (addStages) : insert new stages before or after a named target (for example, adding an additional transformation)
Delete (removeStages) : remove existing stages (for example, removing a case from a fallback pipeline)
Override : Straight-up replace the targeted stage with a new definition

Implementing the Customisation

Finally, we get to making a change, which involves the following steps

Identify the YAML path(s) to the attribute you wish to modify
Determine what changes we need to make
Implement and refine those changes in a staging environment
Transfer the changes to a customisation module and retest in a staging environment before deployment
Deploy changes, requeue affected products, and trigger an entity sync

In our example, we are going to modify the way that the shortDescription attribute is generated for virtual and downloadable products only.

The changes we want to make are to

Use the meta description attribute, rather than short description, as our source value
Append the string “ - This is a very special product.” at the end of the generated value if the meta description is not empty
Apply this change to virtual and downloadable product types

Identify the YAML path(s) to the attribute you wish to modify

stages.iterateIndexingRecordsBatch.stages.iterateIndexingRecords.stages.processProduct.stages.virtualProduct.stages.attributes.shortDescription
stages.iterateIndexingRecordsBatch.stages.iterateIndexingRecords.stages.processProduct.stages.downloadableProduct.stages.attributes.shortDescription

If you don’t know how we got these paths, go re-read the Identifying the YAML Configuration Path section above.

Reference: module-m2-indexing-products/etc/pipeline/product/attributes/short_description.yml

Determine the changes required to alter the attribute output

Change the extraction from currentProduct::getShortDescription() to currentProduct::getMetaDescription()
Add a Stage\Transform stage after the validation to Append a value Note: as we are replacing the the extract stage, it might be tempting to add the Append to the list of transformation arguments and save a change. This, however, would make our Validate stage worthless as every passed value would contain our appended string

Express the changes as YAML overrides

To change the extraction, we need to replicate the entire extract stage definition as we can’t just replace a stage’s property. Taking the definition from the core code, replace getShortDescription() with getMetaDescription()

pipeline: Stage\Extract
args:
  extraction: currentProduct::getMetaDescription()
  transformations:
    - StripTags(["p", "br", "hr", "h1", "h2", "h3", "h4", "h5", "h6", "strong", "em", "ul", "ol", "li", "dl", "dt", "dd", "img", "sub", "sup", "small"], ["script"])
    - EscapeHtml
    - Trim

To append data, we need a new stage of type Stage\Transform with a transformation of type Append, passing in our desired string

pipeline: Stage\Transform
args:
  transformation: Append(" - This is a very special product.")

We’re using a static value here, but we could just as easily pass in a string from our context (for example, from the Magento configuration) or the existing product using an extraction here. Eg

pipeline: Stage\Transform
args:
  transformation: Append(" - ", $currentProduct.getMetaTitle())

Implement the changes in a local or staging environment

As we are making the changes on a staging (or local) environment, we can take advantage of the auto-generated files which contain our custom attribute definitions. By adding changes directly to (in this case) var/klevu/indexing/pipeline/product/add_update.overrides.yml, we don’t need to worry about building a module just to test a theory, or recompiling as the file is already included.

To be safe, you should disable the automatic regeneration of configuration overrides while working to ensure your changes don’t suddenly disappear.

In the Magento backend, set Stores > Configuration > Klevu > Developer Settings > Pipelines > Enable autogeneration of configuration overrides to “No” and clear your cache

First, replicate the paths identified above in YAML format.

Note: if you have autogenerated attribute information in the file, you can use the existing structure without having to remove or duplicate the paths (which would not be valid YAML)

stages:
  iterateIndexingRecordsBatch:
    stages:
      iterateIndexingRecords:
        stages:
          processProduct:
            stages:
              virtualProduct:
                stages:
                  attributes:
                    shortDescription:
              downloadableProduct:
                stages:
                  attributes:
                    shortDescription:

and add the overrides required

stages:
  iterateIndexingRecordsBatch:
    stages:
      iterateIndexingRecords:
        stages:
          processProduct:
            stages:
              virtualProduct:
                stages:
                  attributes:
                    shortDescription:
                      stages:
                        generateData:
                          stages:
                            default:
                              getDefaultLanguageValue:
                                stages:
                                  getData:
                                    stages:
                                      extract:
                                        pipeline: Stage\Extract
                                        args:
                                          extraction: currentProduct::getMetaDescription() # This is our change
                                          transformations:
                                            - StripTags(["p", "br", "hr", "h1", "h2", "h3", "h4", "h5", "h6", "strong", "em", "ul", "ol", "li", "dl", "dt", "dd", "img", "sub", "sup", "small"], ["script"])
                                            - EscapeHtml
                                            - Trim
                                    addStages:
                                      after: validate
                                      transform:
                                        pipeline: Stage\Transform
                                        args:
                                          transformation: Append(" - This is a very special product.")
              downloadableProduct:
                # Implement the same for downloadable products

Now, queue one or more products for sync by updating the product data or via CLI

php bin/magento klevu:indexing:entity-update --entity-types=KLEVU_PRODUCT --entity-ids=1,2,3 -vvv

and trigger a sync

php bin/magento klevu:indexing:entity-sync --entity-types=KLEVU_PRODUCT -vvv

We recommend enabling verbose indexing logging and archiving existing logs before triggering the test sync to make identifying the most recent data simpler

Transfer changes to a dedicated customisation module

This guide assumes you are familiar with creating a Magento 2 extension.

If you work better with code in front of you to reference, there is an example module at the end of the article

IMPORTANT: This example module is not production-ready and should not be used. It is for reference and illustration purposes only and no guarantee is made to its reliability.

We strongly recommend testing it in your own environment before deploying it to production.

We will use the module name Vendor_KlevuIndexingOverrides; you should choose a namespace (and module name) suited to your installation.

Dependencies

We will reference the Klevu Indexing Products and PHP-Pipelines packages, so ensure you include the following in the extension’s composer.json file

    "require": {
        "php": "~8.1.0|~8.2.0|~8.3.0|~8.4.0",
        "klevu/module-m2-indexing-products": "^3.0.0",
        "klevu/php-pipelines": "^1.1.0"
    },

and the following in the extension’s etc/module.xml file

        <sequence>
            <module name="Klevu_IndexingProducts"/>
            <module name="Klevu_Pipelines"/>
        </sequence>

Pipeline YAML

Note, the framework isn’t opinionated about file locations, but we find the paths specified below are the most logical - especially as modules grow and include additional overrides, especially for different record types.

We will create a parent overrides file at

etc/pipeline/indexing/product/add_update.overrides.yml .

The changes made above should be copied into this file, though to avoid duplication of code (especially if we need to make modifications to every product type), we will extract the actual changes to their own include files. Create these files at

etc/pipeline/indexing/product/attributes/short_description/extract.yml
etc/pipeline/indexing/product/attributes/short_description/transform.yml

(if we were completely replacing the short_description attribute’s definition, we could simply create short_description.yml, however keeping changes small and discrete means better maintainability in the future)

Our parent add_update.overrides.yml file should look like

stages:
  iterateIndexingRecordsBatch:
    stages:
      iterateIndexingRecords:
        stages:
          processProduct:
            stages:
              virtualProduct:
                stages:
                  attributes:
                    shortDescription:
                      stages:
                        generateData:
                          stages:
                            default:
                              stages:
                                getDefaultLanguageValue:
                                  stages:
                                    getData:
                                      stages:
                                        extract:
                                          import: Vendor_KlevuIndexingOverrides::etc/pipeline/indexing/product/attributes/short_description/extract.yml
                                      addStages:
                                        after: validate
                                        transform:
                                          import: Vendor_KlevuIndexingOverrides::etc/pipeline/indexing/product/attributes/short_description/transform.yml
              downloadableProduct:
                stages:
                  attributes:
                    shortDescription:
                      stages:
                        generateData:
                          stages:
                            default:
                              stages:
                                getDefaultLanguageValue:
                                  stages:
                                    getData:
                                      stages:
                                        extract:
                                          import: Vendor_KlevuIndexingOverrides::etc/pipeline/indexing/product/attributes/short_description/extract.yml
                                      addStages:
                                        after: validate
                                        transform:
                                          import: Vendor_KlevuIndexingOverrides::etc/pipeline/indexing/product/attributes/short_description/transform.yml

extract.yml should contain

pipeline: Stage\Extract
args:
    extraction: currentProduct::getMetaDescription() # This is our change
    transformations:
        - StripTags(["p", "br", "hr", "h1", "h2", "h3", "h4", "h5", "h6", "strong", "em", "ul", "ol", "li", "dl", "dt", "dd", "img", "sub", "sup", "small"], ["script"])
        - EscapeHtml
        - Trim

and transform.yml

pipeline: Stage\Transform
args:
    transformation: Append(" - This is a very special product.")

Defining the Overrides

Now we need to tell the Klevu module to use these override files by injecting the add_update.overrides.yml filepath into the provider used when compiling our pipeline.

In your di.xml, add

    <virtualType name="Klevu\IndexingProducts\Service\Provider\PipelineConfigurationOverridesFilepathsProvider\Add">
        <arguments>
            <argument name="pipelineConfigurationOverrideFilepaths" xsi:type="array">
                <item name="vendor_klevuindexingoverrides_indexing_product_add_update"
                      xsi:type="string">Vendor_KlevuIndexingOverrides::etc/pipeline/indexing/product/add_update.overrides.yml</item>
            </argument>
        </arguments>
    </virtualType>
    <virtualType name="Klevu\IndexingProducts\Service\Provider\PipelineConfigurationOverridesFilepathsProvider\Update">
        <arguments>
            <argument name="pipelineConfigurationOverrideFilepaths" xsi:type="array">
                <item name="vendor_klevuindexingoverrides_indexing_product_add_update"
                      xsi:type="string">Vendor_KlevuIndexingOverrides::etc/pipeline/indexing/product/add_update.overrides.yml</item>
            </argument>
        </arguments>
    </virtualType>

Note that there are two providers - one for KLEVU_PRODUCT::add and KLEVU_PRODUCT::update operations. We have created a single pipeline overrides file for both operations, but if you need different values depending on whether a product is being sent to Klevu for the first time or not, you could create different overrides files and inject them to the relevant providers as required.

Compiling and Testing

As we made our changes in the existing overrides file originally, we cannot reliably test our new module until we have reverted these updates.

To do this, ensure you have re-enabled the Enable autogeneration of configuration overrides config setting and cleared your cache. Then, you can either modify an attribute through the Magento backend or force a regeneration via CLI

php bin/magento klevu:indexing:configuration-overrides-regenerate --entity-type=KLEVU_PRODUCT -vvv

Install and recompile Magento with your new custom module. Then (again), queue one or more products for sync by updating the product data or via CLI

php bin/magento klevu:indexing:entity-update --entity-types=KLEVU_PRODUCT --entity-ids=1,2,3 -vvv

and trigger a sync

php bin/magento klevu:indexing:entity-sync --entity-types=KLEVU_PRODUCT -vvv

Deploy the customisation module to production

Providing everything has gone as expected, you can deploy your new module to your production environment and all future syncs will incorporate the changes.

To send updated data for your entire catalog, it is worth triggering an update via CLI

php bin/magento klevu:indexing:entity-update --entity-types=KLEVU_PRODUCT --entity-ids=all

You can then either trigger a sync on demand, or wait until the next scheduled process to run.

Example Module

Vendor_KlevuIndexingOverrides