Making sense out of nonsense

One of the first thing we found necessary after we created our NB-IoT platform, was fancy graphs. Because you know, people like graphs. It also helps as a sanity check to see that your data makes sense graphically. There was only one problem, Tm9zeSBsaXR0bGUgYmFzdGFyZCBhcmVuJ3QgeW91Pw== doesn’t really translate well into a graph.

Making sense out of your data

Firstly, we need to see what kind of data we can expect from our device. The NB-IoT platform automatically encodes the data (once it leaves the NB-IoT platform) as Base64 to allow for both binary and non-binary data to be represented as a string in the payload, meaning we have the first step fixed for us. Ish. You see, we want something that’s graphable. And we either get

// Decoded as string when the data is binary
"HL>9Á¿?´sAèÌÍBƒÉ€@»™"

or

// Decoded as hexstring when the data is binary
"48144c003e39c1bf3f8db47341e8cccd4283c98040bb990001000000000000000000010002"

dependant of what kind of data was behind the Base64 encoded value.

Enter the data-mapper-chain

Ok, I wasn’t very creative when I came up with the name data-mapper-chain, but it works. The plan was simple. Create a chain of rudimentary transformations which can be applied to a dataset. This can be abstracted into “commands” which again can easily translated into UI elements so a user can configure the transformations (basically we make a DSL). Alright, we’re nearly there!

These were the first commands I came up with that was obviously needed:

  • Base64
    • Encoding and decoding of base64 input
  • Chunk
    • Take a chunk of the input and return it
  • FromJSON
    • Traverse a JSON struct and return value
  • HexToFloat
    • Take a hex input and convert it to a float
  • HexToInt
    • Take a hex input and convert it to an int
  • Offset
    • Take an input and offset it by a positive or negative value

A real life example

Let’s take a real life example to show how it works in action. We have the following Base64 string coming from the one of our devices on the NB-IoT Platform, created to monitor air quality in the Trondheim municipality:

R95+AD45wZA/jbRxQbwAAEKH8ABAkbQAAAzwA5EAAWDMAAEAAg==

First we need to decode this from Base64 and as a hex string giving us

47de7e003e39c1903f8db47141bc00004287f0004091b400000cf00391000160cc00010002

Now what we know (as we’re the one who’s made the device and constructed the payload) is the following:

# Timestamp     Longitude   Latitude    Altitude    Humidity    Temp        
# Float         Float       Float       Float       Float       Float       
47de7e00        3e39c190    3f8db471    41bc0000    4287f000    4091b400

# Status    Jibberjabber    CO2 ppm     TVOC ppb    PM 2.5      PM 10
# Byte      Uknown          UInt16      UInt16      UInt16      UInt16
00          000160cc        0cf0        0391        0001        0002

As you see, it has a lot of different data types just jumbled together and doesn’t really make sense unless you see them as separate parts. Let’s continue the example that we want to visualize the CO2 (equivalents) PPM. The data mapping chain would be the following:

Raw data

-> Decode Base64 as hex (47de7e003e39c190…)

-> Chunk from index 50 with 4 chars (0cf0)

-> HexToInt Unsigned int (3312)

Now look at that. 3312. That is graphable! The result for this is as following when used as a transformation over several data points:

co2 graph

If you feel like experimenting yourself you can go to this codepen. It has some hard coded data points which will be transformed on the fly by the data-mapper-chain and visualized with chart.js.

Making it approachable

Maybe not everyone wants to dabble directly in javascript, so we’ve added the possibility to add and save data mappers directly in our UI at https://nbiot.engineering.

Go to your collection and choose visualization on the left hand menu. There you can add visualizations at your hearts content by clicking Add visualization.

data mapper example

From here you can add another transformation by clicking the plus buttons below or above any transformation. This will add the transformation in the order according to which button you clicked. When clicking add you’ll get this menu:

add data mapper transformation example

As discussed earlier, you can here add which transformations you want to do on your data. Afterwards you can configure the transformation as you much as you need to:

configure data mapper transformation

When you’ve added enough transformations you’ll (hopefully) get the correct number out and you can finally create your data mapper visualization.

data mapper chain result

After clicking create you’ll see your saved data-mapper which might help you make sense of your data one way or another.

end result after configuring data mapper chain

BONUS - Serialization and loading mappers

As you create your data mapper in the UI it is actually saved directly onto the collection. The fantastic thing about this, is that we can load the mapper directly from an API-call.

Typically you’ll get something like this from the collection API:

    {
      "collectionId": "<redacted>",
      "teamId": "<nope>",
      "tags": {
        "data-mapper-chain": "WyJ7XCJuYW1lXC.....",
        "name": "Our collection"
      }
    },

You might spot the tag with the name data-mapper-chain, and yes, you can load that whenever you feel like it by using the following steps:

Directly in your webpage

<script src="https://cdn.jsdelivr.net/npm/@exploratoryengineering/data-mapper-chain@0.7"></script>
<script>
    // Expecting collection to exist and have one mapper
    var configs = window.btoa(collection.tags["data-mapper-chain"]);
    var config = configs[0];
    var myMapper = dmc.create()
        .loadConfig(config);
</script>

As a module dependency

Install the dependency:

npm i -S @exploratoryengineering/data-mapper-chain

Use in code:

import { DataMapperChain } from "@exploratoryengineering/data-mapper-chain";

// Expecting collection to exist and have one mapper
const configs = window.btoa(collection.tags["data-mapper-chain"]);
const config = configs[0];
const myMapper = new DataMapper().loadConfig(config)

After you’ve loaded the saved configuration through either method above, the data mapper is ready to be used through by calling myMapper.mapData(value).

We’ve open sourced the data-mapper-chain on github so you can check it out and see how it works. Contributions are always welcome <3

Hopefully this blog post helps you understand how it’s possible to graph your device payload and will help you on your prototyping journey towards excellence.

As always; happy hacking!