Pipelines

19 min

If an attribute has been registered with Klevu for indexing that does not mean that the entity data from that attribute is automatically sent to Klevu. It means it can be sent to Klevu.

Entity data indexed to Klevu is defined in the indexing pipelines YAML files. These files can either be:

auto-generated via the product attribute setting Automatically Generate Pipeline Configuration For see Attribute Admin Settings
manually created if the auto-generated code is not desirable, see Pipeline Customisation.

The auto-generated pipeline YAML files can be found in var/klevu/indexing/pipeline/.

PHP-SDK, Pipelines & Magento Overview

Klevu indexing uses Klevu’s Data Indexing JSON API. Klevu has developed a PHP-SDK to assist developers with integrating Klevu indexing into any PHP Platform. This PHP-SDK is used to index data to Klevu in these Magento modules.

Klevu has also developed PHP-Pipelines, these are generic PHP pipelines that can be used for any purpose and are independent of Magento and the PHP-SDK.

These 2 modules are brought together via php-sdk-pipelines, adding PHP-SDK functionality into the pipelines.

There is a 4th module, module-m2-pipelines which integrates these new components into Magento. This handles dependency injection and adds some Magento context into the pipelines.

Types of Pipeline

Create Record Pipeline: Pipeline\CreateRecord

Create and return a new object. Example: Create an object with keys id and type

pipeline: Pipeline\CreateRecord
stages:
  id:
    pipeline: Stage\Extract
    args:
      extraction: currentProduct::getEntityId()
      transformations: ToString
  type:
    pipeline: Stage\StaticValue
    args:
      value: "KLEVU_PRODUCT"

Output:

{
    "id": "123",
    "type": "KLEVU_PRODUCT"
}

Fallback Pipeline: Pipeline\Fallback

If any validation exception ValidationExceptionInterface is thrown, catch it and move on to the next stage, otherwise return the output from this stage.

Example:

Get the product type,
Validate that the type is “bundle”
- If valid include this file product/bundle.yml
- If not valid move to the next stage which will include this file product/default.yml

bundleProduct:
  stages:
    -
      pipeline: Stage\Extract
      args:
        extraction: currentProduct::getTypeId()
    -
      pipeline: Stage\Validate
      args:
        validation: IsEqualTo("bundle")
    -
      import: Klevu_IndexingProducts::etc/pipeline/product/bundle.yml
default:
  stages:
    -
      import: Klevu_IndexingProducts::etc/pipeline/product/default.yml

Note: the stage names have been removed from this example to make it easier to read

Iterate Pipeline: Pipeline\Iterate

Loop over the provided records.

Example: Iterate over all records provided into the pipeline

stages:
  iterateIndexingRecords:
    pipeline: Pipeline\Iterate
    stages:
     -
        ...

or get all customer groups, iterate over then, setting the current customer group at the start of each loop

getCustomerGroups:
  pipeline: Stage\Extract
  args:
    extraction: store::getForCurrentStore().customer_groups
    transformations: ToString
IterateCustomerGroups:
  pipeline: Pipeline\Iterate # LOOP OVER ALL CUSTOMER GROUPS
  stages:
    SetCustomerGroupId:
      pipeline: Stage\SetCustomerGroupId
    ...

Pipeline Stages

Static Value

Add a value to the output.

Example:

pipeline: Pipeline\CreateRecord
stages:
  type:
    pipeline: Stage\StaticValue
    args:
      value: "KLEVU_PRODUCT"

Output:

{
    "type": "KLEVU_PRODUCT"
}

Extract

Get the value from the extraction and pass it to the next stage Example:

pipeline: Pipeline\CreateRecord
stages:
  id:
    pipeline: Stage\Extract
    args:
      extraction: currentProduct::getEntityId()

Output:

{
    "id": 123
}

Transform

Transform the provided value using one or multiple transformers. Example:

pipeline: Pipeline\CreateRecord
stages:
  id:
    - pipeline: Stage\Extract
      args:
        extraction: currentProduct::getEntityId()
    - pipeline: Stage\Transform 
      args:
        transformation: ToString|Trim|Prepend("ID: ")

Output:

{
    "id": "ID: 123"
}

Alternatively, transformations can be passed into extractions, skipping the need to add an extra stage.

Example:

pipeline: Pipeline\CreateRecord
stages:
  name:
    pipeline: Stage\Extract
    args:
      extraction: currentProduct::getName()
      transformations: 
        - Trim
        - Append(" - ", $currentProduct::getManufacturer())

Output:

{
    "name": "Test Product - Brand Name"
}

Validate

Example:

- pipeline: Stage\Extract
  args:
    extraction: currentProduct::getKlevuImage()
- pipeline: Stage\Validate
  args:
    validation: IsNotEqualTo("no_selection")|IsNotEmpty

If validation fails an exception of type Klevu\Pipelines\Exception\ValidationExceptionInterface will be thrown. Fallback pipelines will catch this an proceed to the next stage. See module-m2-indexing-products/etc/pipeline/attributes/images.yml

Register Context

Example: Extract the current product and then register the context as currentProduct

extractProduct:
  pipeline: Stage\Extract
  args:
    extraction: currentIndexingRecord::getEntity()
registerProduct:
  pipeline: Stage\RegisterContext
  args:
    contextKey: "currentProduct"

Then later in the pipeline (possibly in another file, even another module) the context can be used.

pipeline: Pipeline\CreateRecord
stages:
  default:
    stages:
      -
        pipeline: Stage\Extract
        args:
          extraction: currentProduct::getName()
          transformations: EscapeHtml

Output:

{
    default: 'Product Name'
}

Log

Log the current context. Example: module-m2-indexing/etc/pipeline/process-batch-payload.yml Take whatever the current context is (the output from the previous operation) and log it along with the message “Chunked payload” at log level “debug”.

stages:
  log:
    pipeline: Indexing\Stage\Log
    args:
      message: "Chunked payload"
      level: debug

Output (formatted to make it more readable):

report.DEBUG: Chunked payload {
    "identifier":"processPayload.processRecords.iterate.log",
    "payload":[
        {"id":"47","type":"KLEVU_PRODUCT","relations":{"categories":{"values":["categoryid_15","categoryid_36","categoryid_2","categoryid_12"]}},"attributes":{"name":{"default":"Chaz Kangeroo Hoodie-XS-Black"},"sku":"MH01-XS-Black","images":[{"url":"https://magento.test/media/catalog/product/cache/7fe2def9e1d815fff5838a5b86f54f8b/m/h/mh01-black_main_1.jpg","type":"default","width":800,"height":800}],"prices":[{"amount":52.0,"type":"defaultPrice","currency":"USD"},{"amount":52.0,"type":"salePrice","currency":"USD"}],"url":"https://magento.test/catalog/product/view/id/47/s/chaz-kangeroo-hoodie-xs-black/","inStock":true,"shortDescription":{"default":null},"description":{"default":"Ideal for cold-weather training or work outdoors, the Chaz Hoodie promises superior warmth with every wear. Thick material blocks out the wind as ribbed cuffs and bottom band seal in body heat. &amp;bull; Two-tone gray heather hoodie.&amp;bull; Drawstring-adjustable hood. &amp;bull; Machine wash/dry."},"tags":[],"visibility":[],"rating":4.2,"ratingCount":321}},
        {"id":"2051-2053","type":"KLEVU_PRODUCT","relations":{"parentProduct":{"values":["2051"]}},"attributes":{"sku":"Configurable product link test-Orange-1","images":null,"type":"default","width":800,"height":800}],"prices":[{"amount":45.0,"type":"defaultPrice","currency":"USD"},{"amount":45.0,"type":"salePrice","currency":"USD"}],"url":"https://magento.test/configurable-product-link-test.html","inStock":true,"shortDescription":{"default":null},"description":{"default":null},"rating":4.7,"ratingCount":123}}
    ]
} []