Skip to main content

Making Data Rectangular

· 4 min read
Sergi Kedrov

What’s the problem?

JSON is an extremely lightweight format and de-facto the standard for exchanging data. JSON messages transmit data objects of arbitrary nesting and complexity.

The flexibility of JSON is beneficial when it’s needed to get an order with an variable number of items, a complex application form, or a response from a REST API.

On the other hand, the versatility of JSON becomes problematic when one has to analyze a historical set of data or an online flow of messages.

If one needs the data in a tabular form to ingest into an analytical database or a spreadsheet, the nested JSON messages have to be denormalized.

Multidimensional complex JSON messages need to be flattened to become rectangular.

What’s the complexity?

Imagine that one has a flow of messages containing an array. The array may contain a variable number of elements. Each element may be an object of an arbitrary complexity. For example, the object may contain an embedded array. And the embedded array may again contain a variable number of elements. And each element again may be an object of an arbitrary complexity. And again, and again.

For example, a typical purchase order contains some common attributes such as date-time, branch, cashier, total amount, delivery address, and an array of purchased items, where each item has a name, amount, price, tax, discount, etc. Moreover, each item may be provided with additional information at processing time like promo, category and so on.

JSON format is ideal for transferring such structures. But it requires special efforts to push these multidimensional data into a flat table.

Why Nodes?

Fabrique Nodes are designed to compose and decompose nested structures to make dimensional transformations without a line of code.

In the video we’ll show how to employ the Fabrique Nodes to connect to a cloud message queue, consume messages having nested objects and arrays of an arbitrary number of elements, extract elements from arrays, decompose attributes from nested objects, compose flat messages, and ingest the flat messages into a destination cloud message queue without a line of code.

If it is needed to ingest data into another destination like Database, Message Bus, or Spreadsheet, just choose an appropriate Node from the Collection.

If someone needs to consume a flow of messages from another source or change a processing logic, just choose appropriate Nodes and build a graph of Nodes.

Node Collection is extendable. If some Node is absent, just text us.

Try Nodes!

Fabrique Node Editor is a service for building, testing, exchanging, deploying, executing, and visually documenting the event flow processing scenarios.

  • Go to Fabrique.ai
  • Register to the Fabrique Node Editor. It’s for free
  • Find the “Flatten an Array to Elements” Project in Public
  • Clone and try the Project by your own
  • Find more Public projects to try
  • Read the Blog posts with more cases
  • Contact us if you have any questions

About

Fabrique Nodes is a cloud native scalable event flow processing service for applications like decision automation, real-time analytics, anomaly detection, data transformation, and AI implementation without programming skills.

Event processing logic is built by the extendable collection of no-code Nodes. Fabrique Nodes allow to build direct asynchronous computational graphs of operations, and move the projects to production in 1 click. No code is needed.

Welcome to Fabrique!