User interface

General Principles Add Missing Dates Add text column Add Total Rows Aggregate Append Compare Text Columns Argmax Argmin Concatenate columns Convert Convert date to text Convert text to date Cumulated sum Custom step Delete column(s) Geographically dissolve data Duplicate Duration Evolution Extract date information Extract substring Fill null Filter Formula Geographical hierarchy Get unique groups/values Hierarchical rollup ifthenelse Join Keep column(s) Moving Average Percentage Pivot Rank Rename Replace Geographical simplification Sort Split column Column's Statistics To lowercase To uppercase Top N rows Unpivot Waterfall

Technical documentation

Packages Steps Designing a new step Known limitations Variables UI components Custom code editor Python package

Using variables in pipeline

Why

The idea is to be able to build templatable queries with values that could be parametrized.

As often, an example is better than a lot of words. Imagine the following pipeline that filters out lines where age <= 20:

[
  {
    "name": "domain",
    "domain": "team_infos",
  },
  {
    "name": "filter",
    "condition": {
      "and": [
        "operator": "gt",
        "column": "age",
        "value": 20
      ]
    }
  }
]

Now suppose that you include the visual query builder inside a host application that lets you enter the age in an input field. You might then want to parametrize the “age” value and reuse your pipeline with this specific value.

Using lodash template syntax, this could look like:

[
  {
    "name": "domain",
    "domain": "team_infos",
  },
  {
    "name": "filter",
    "condition": {
      "and": [
        "operator": "gt",
        "column": "age",
        "value": "<%= age %>"
      ]
    }
  }
]

How

It is possible to define a variable context (i.e. a bag of variables) in the store that can later be used and referenced in the pipeline steps. We also need to specify an interpolation function that will take a template string and replace it with the corresponding value. This function needs to have the following signature:

type InterpolateFunction = (template: string, context: ScopeContext) => any;

ScopeContext is just a mapping of variable names to values.

Example

Imagine that you want to use the lodash template syntax to reference your variables, you would then need to provide an interpolation function that would take a lodash templated string, the variables map and return the interpolated string:

import _ from 'lodash';

function interpolate(value: string, context: ScopeContext) {
  const compiled = _.template(value);
  return compiled(context);
}

and feed your store with all this:

{
  //...
  variables: {
    age: 42
  },
  interpolationFunc: interpolate,
  //...
}

Steps concerned

For now, only a few steps handle variables interpolation but the support of this feature will be improved progressively in the next releases.

So far, the steps concerned are:

fillna: the value property can be interpolated,
filter: the value property of each condition type can be interpolated,
formula: the formula property can be interpolated,
replace: each value in the to_replace can be interpolated,
top: the limit property can be interpolated.

Another important limitation for now is that except for the top.limit, the interpolation will insert string values, i.e. an interpolated filtered value cannot be an integer.