Advanced Data Pipeline

The Advanced Data Pipeline is an advanced version of our Data Explorer. It is essentially a visual workflow for processing data to build a chart. Merge and transform any number of layers at any point in the pipeline using a simple drag-and-drop interface.

Merge two layers, apply a transformation, then merge a third layer.

Overview

Access the Advanced Data Pipeline by either adding a new chart from a dashboard or navigating to Explore from the top navigation. Then, click Advanced at the top to switch to Advanced Data Pipeline mode.

Click image to expand.

Add a node

Nodes

Nodes are modules you drag onto the canvas to build your chart. Types of nodes include Database Query, Transformation Step, and Merge Step.

Connect nodes

Charts are built by first dragging one or more Database Query nodes onto the canvas, combining those nodes using one or more Merge Step nodes, and connecting the final node to the Chart node. Add Transformation Step nodes as needed to edit your dataset results.

Add a node

To add a node to the canvas, click and drag it from the left sidebar onto the canvas.

Connecting nodes

Disconnect nodes

To connect Node B to the result of Node A, click the output square of Node A and then hover anywhere on Node B. Node B will be highlighted in blue to show the connection is recognized. Then, click on Node B to complete the connection.

Nodes you cannot connect to will be greyed out.

If a node is already connected to another node, its connector square will not be visible.

Disconnect nodes

To disconnect two nodes, hover over the line connecting the nodes and click the scissors icon .

Node types

Edit a Database Query node

Database Query node

Drag a Database Query node onto the canvas to add a dataset to your chart. Database Query nodes are analagous to Layers in the Basic Chart Editor.

Clone a Database Query node

Edit a Database Query node

To edit a Database Query node, click the pencil icon on the node.

Clone a Database Query node

Click the square clone icon on the node to clone it.

Transformation Step node

Transformation Step node

To edit the data output of a node, drag a Transformation Node onto the canvas, hover over the output line of the node you want to edit, and drop it there. Transformation Step nodes are analagous to Pipeline Steps in the Basic Chart Editor. For details on each Transformation step, see our Data Pipeline Steps documentation.

Merge Step node

Combine the results from two or more Database Query nodes by connecting their output to a Merge Step node. You may use as many Merge Nodes as you wish.

Preview Data

Click any node in the Pipeline to view its Input and Output data in the Preview Data drawer, below the canvas.

Run Selected Node

Click any node, then click Run Selected Node to run the pipeline up to that point and view its dataset in the Preview Data drawer. This can be especially useful for troubleshooting and testing a part of your pipeline separately.

Example

Add a new chart to a dashboard, or click Explore from the top navigation. Click Advanced to toggle to the Advanced Data Pipeline.

You'll see an empty Database Query node and a Chart node on your canvas. Click the pencil icon on the Database Query node to edit its query.

Use Interactive Mode or SQL Mode to build your query as you normally would. Click Done when finished.

Add a second Database Query node to the canvas by clicking Database Query in the left sidebar, and dragging it onto the canvas. Edit its query and click Done.

Drag a Merge Step node onto the canvas, and drop it on the line connecting the Database Query node to the Chart node. The connecting line will appear dotted to signify the Merge Step node will connect between the two nodes.

To connect the second Database Query node to the Merge Step, click the square on the bottom middle of the node—the node's output connector—and then click anywhere on the Merge Step node.

ap3.png

To run your queries and update your pipeline, click the Run Pipeline button.

Next, drag any Transformation Step node to the canvas, and hover over the line between the Merge step and the Chart node. Drop the Transformation Step node there, and edit as needed.

Click Run Pipeline to update your chart.