---
title: Pipelines
slug: klevu-magento-v4/entities-pipelines
docTags: 
createdAt: 2024-08-06T15:48:55.412Z
---

If an attribute has been registered with Klevu for indexing that does not mean that the entity data from that attribute is automatically sent to Klevu. It means it can be sent to Klevu. 

Entity data indexed to Klevu is defined in the indexing pipelines YAML files. 
These files can either be:

- auto-generated via the product attribute setting `Automatically Generate Pipeline Configuration For` see [Attribute Admin Settings ](https://docs.klevu.com/klevu-magento-v4/attributes#vOPPl)
- manually created if the auto-generated code is not desirable, see [Pipeline Customisation](https://docs.klevu.com/klevu-magento-v4/entities-pipelines-customisation).

The auto-generated pipeline YAML files can be found in `var/klevu/indexing/pipeline/`. 

***

## PHP-SDK, Pipelines & Magento Overview

Klevu indexing uses [Klevu’s Data Indexing JSON API](https://docs.klevu.com/indexing-apis/overview). Klevu has developed a [PHP-SDK](https://github.com/klevu/php-sdk) to assist developers with integrating Klevu indexing into any PHP Platform. This PHP-SDK is used to index data to Klevu in these Magento modules.

Klevu has also developed [PHP-Pipelines](https://github.com/klevu/php-pipelines), these are generic PHP pipelines that can be used for any purpose and are independent of Magento and the PHP-SDK. 

These 2 modules are brought together via [php-sdk-pipelines](https://github.com/klevu/php-sdk-pipelines), adding PHP-SDK functionality into the pipelines.

There is a 4th module, [module-m2-pipelines](https://github.com/klevu/module-m2-pipelines) which integrates these new components into Magento. This handles dependency injection and adds some Magento context into the pipelines.

***

## Types of Pipeline

### Create Record Pipeline: `Pipeline\CreateRecord`

Create and return a new object.
Example:
Create an object with keys `id` and `type`

```yaml
pipeline: Pipeline\CreateRecord
stages:
  id:
    pipeline: Stage\Extract
    args:
      extraction: currentProduct::getEntityId()
      transformations: ToString
  type:
    pipeline: Stage\StaticValue
    args:
      value: "KLEVU_PRODUCT"
```

Output:

```json
{
    "id": "123",
    "type": "KLEVU_PRODUCT"
}
```

### Fallback Pipeline: Pipeline\Fallback

If any validation exception `ValidationExceptionInterface` is thrown, catch it and move on to the next stage, otherwise return the output from this stage.

Example: 

- Get the product type, 
- Validate that the type is “bundle”
  - If valid include this file `product/bundle.yml`
  - If not valid move to the next stage which will include this file `product/default.yml`

```yaml
bundleProduct:
  stages:
    -
      pipeline: Stage\Extract
      args:
        extraction: currentProduct::getTypeId()
    -
      pipeline: Stage\Validate
      args:
        validation: IsEqualTo("bundle")
    -
      import: Klevu_IndexingProducts::etc/pipeline/product/bundle.yml
default:
  stages:
    -
      import: Klevu_IndexingProducts::etc/pipeline/product/default.yml
```

Note: the stage names have been removed from this example to make it easier to read

### Iterate Pipeline: Pipeline\Iterate

Loop over the provided records.

Example:
Iterate over all records provided into the pipeline

```yaml
stages:
  iterateIndexingRecords:
    pipeline: Pipeline\Iterate
    stages:
     -
        ...
```

or get all customer groups, iterate over then, setting the current customer group at the start of each loop

```yaml
getCustomerGroups:
  pipeline: Stage\Extract
  args:
    extraction: store::getForCurrentStore().customer_groups
    transformations: ToString
IterateCustomerGroups:
  pipeline: Pipeline\Iterate # LOOP OVER ALL CUSTOMER GROUPS
  stages:
    SetCustomerGroupId:
      pipeline: Stage\SetCustomerGroupId
    ...
```

***

## Pipeline Stages

### Static Value

Add a value to the output.

Example:

```yaml
pipeline: Pipeline\CreateRecord
stages:
  type:
    pipeline: Stage\StaticValue
    args:
      value: "KLEVU_PRODUCT"
    
```

Output:

```json
{
    "type": "KLEVU_PRODUCT"
}
```

### Extract

Get the value from the extraction and pass it to the next stage
Example:

```yaml
pipeline: Pipeline\CreateRecord
stages:
  id:
    pipeline: Stage\Extract
    args:
      extraction: currentProduct::getEntityId()
```

Output:

```json
{
    "id": 123
}
```

### Transform

Transform the provided value using one or multiple transformers.
Example:

```yaml
pipeline: Pipeline\CreateRecord
stages:
  id:
    - pipeline: Stage\Extract
      args:
        extraction: currentProduct::getEntityId()
    - pipeline: Stage\Transform 
      args:
        transformation: ToString|Trim|Prepend("ID: ")
```

Output:

```json
{
    "id": "ID: 123"
}
```

Alternatively, transformations can be passed into extractions, skipping the need to add an extra stage.

Example:

```yaml
pipeline: Pipeline\CreateRecord
stages:
  name:
    pipeline: Stage\Extract
    args:
      extraction: currentProduct::getName()
      transformations: 
        - Trim
        - Append(" - ", $currentProduct::getManufacturer())
```

Output:

```json
{
    "name": "Test Product - Brand Name"
}
```

### Validate

Example:

```yaml
- pipeline: Stage\Extract
  args:
    extraction: currentProduct::getKlevuImage()
- pipeline: Stage\Validate
  args:
    validation: IsNotEqualTo("no_selection")|IsNotEmpty

```

If validation fails an exception of type `Klevu\Pipelines\Exception\ValidationExceptionInterface` will be thrown. Fallback pipelines will catch this an proceed to the next stage.
See `module-m2-indexing-products/etc/pipeline/attributes/images.yml`

### Register Context

Register a variable to be used elsewhere in the pipeline.

Example:
Extract the current product and then register the context as currentProduct

```yaml
extractProduct:
  pipeline: Stage\Extract
  args:
    extraction: currentIndexingRecord::getEntity()
registerProduct:
  pipeline: Stage\RegisterContext
  args:
    contextKey: "currentProduct"
```

Then later in the pipeline (possibly in another file, even another module) the context can be used.

```yaml
pipeline: Pipeline\CreateRecord
stages:
  default:
    stages:
      -
        pipeline: Stage\Extract
        args:
          extraction: currentProduct::getName()
          transformations: EscapeHtml
```

Output:

```json
{
    default: 'Product Name'
}
```

### Log

Log the current context. 

Example: 
`module-m2-indexing/etc/pipeline/process-batch-payload.yml`
Take whatever the current context is (the output from the previous operation) and log it along with the message “Chunked payload” at log level “debug”.

```yaml
stages:
  log:
    pipeline: Indexing\Stage\Log
    args:
      message: "Chunked payload"
      level: debug
```

Output (formatted to make it more readable):

```none
report.DEBUG: Chunked payload {
    "identifier":"processPayload.processRecords.iterate.log",
    "payload":[
        {"id":"47","type":"KLEVU_PRODUCT","relations":{"categories":{"values":["categoryid_15","categoryid_36","categoryid_2","categoryid_12"]}},"attributes":{"name":{"default":"Chaz Kangeroo Hoodie-XS-Black"},"sku":"MH01-XS-Black","images":[{"url":"https://magento.test/media/catalog/product/cache/7fe2def9e1d815fff5838a5b86f54f8b/m/h/mh01-black_main_1.jpg","type":"default","width":800,"height":800}],"prices":[{"amount":52.0,"type":"defaultPrice","currency":"USD"},{"amount":52.0,"type":"salePrice","currency":"USD"}],"url":"https://magento.test/catalog/product/view/id/47/s/chaz-kangeroo-hoodie-xs-black/","inStock":true,"shortDescription":{"default":null},"description":{"default":"Ideal for cold-weather training or work outdoors, the Chaz Hoodie promises superior warmth with every wear. Thick material blocks out the wind as ribbed cuffs and bottom band seal in body heat. • Two-tone gray heather hoodie.• Drawstring-adjustable hood. • Machine wash/dry."},"tags":[],"visibility":[],"rating":4.2,"ratingCount":321}},
        {"id":"2051-2053","type":"KLEVU_PRODUCT","relations":{"parentProduct":{"values":["2051"]}},"attributes":{"sku":"Configurable product link test-Orange-1","images":null,"type":"default","width":800,"height":800}],"prices":[{"amount":45.0,"type":"defaultPrice","currency":"USD"},{"amount":45.0,"type":"salePrice","currency":"USD"}],"url":"https://magento.test/configurable-product-link-test.html","inStock":true,"shortDescription":{"default":null},"description":{"default":null},"rating":4.7,"ratingCount":123}}
    ]
} []
```