Week 2: Data Storage & Visualisation

The source code for our acquisition device to acquire data for temperature and humidity is located onb GitHub in our group member Tim Baker’s repository:

Click here to view our GitHub for our group

We started the day with some networking issues which evolved into issues with out MQTT broker, mosquitto. As the UTS network blocks certain ports we connected to a private network and changed the port connection to from 1883 to 5269. This fixed our issue with mosquitto but we then needed to secure our connection by adding authorisation for our acquisition device to connect to mosquitto.

Once we had mosquitto back up and running we decided we would resolve our issues with NodeRed.

 
const char* ssid = secret_ssid;
const char* password = secret_password;
const char* mqtt_server = secret_mqtt_passwd;
const char* mqtt_username = secret_mqtt_username;
const char* mqtt_passwd = secret_mqtt_passwd;
const char* sensor_Name = "11.11.403 Temp/Hum";

 
void reconnect() {
  // Loop until we're reconnected
  while (!client.connected()) {
    Serial.print("Attempting MQTT connection...");
    // Create a random client ID
    String clientId = "ESP8266Client-";
    clientId += String(random(0xffff), HEX);
    // Attempt to connect
    if (client.connect(clientId.c_str(), mqtt_username,mqtt_passwd)) {
      Serial.println("connected");
      // Once connected, publish an announcement...
      client.publish("outTopic", "hello world");
      // ... and resubscribe
      client.subscribe("inTopic");
    } else {
      Serial.print("failed, rc=");
      Serial.print(client.state());
      Serial.println(" try again in 5 seconds");
      // Wait 5 seconds before retrying
      delay(5000);
    }
  }
}

Node red is a browser based flow editor for wiring flows together. The flows are then deployed in a single click. JavaScript functions can then be written for functionality between the flows and nodes within.

A function will return a payload as an output to a node such as a database, in our case InfluxDB.

Initially when we installed NodeRed our flow had two MQTT in nodes, one for humidity data and one for Temperature. We tried to join the nodes into one single message payload for the InfluxDB node. The problem we face is how do we get the joined data into different columns in the database by one message payload?

We decided it was best to have a single inject node placing dummy data into the formatting function to create the payload to send to the database with success.

We soon discovered we had our data management the wrong way around in the way we packaged from the sensor acquisition data to InfluxDB.

We wanted to name our sensor so if we had several sensor acquisition devices in different rooms we should identify and pull singular room data if needed. Along with the sensor name we wanted to send the corresponding humidity and temperature data.

For this we had to program the acquisition device to include the three aforementioned types. This meant in NodeRED we needed

This meant in NodeRED we needed to split the incoming packet from Mosquitto into the data types we want to store. To do this we split the data by the new line characters (‘/n’) highlighted in ‘Sample Code 1.1’. This produced the array seen in Sample Code 1.2 with the three types ‘packaged’ into an array sent as a single message payload to InfluxDB.

As we were able to split our message packet into an array for our message payload InfluxDB was able to put our data into the correct columns. By use of database queries in Grafana we can successfully visualise our data appropriately. 

 
if ( (h - hlast) != 0 || (t - tlast) != 0 ) { 
    hlast = h; tlast = t; 
    snprintf (msg, 50, "%s\n%.1f\n%.1f",sensor_Name, h,t ); 
    Serial.println(msg); 
    client.publish("sensor", msg); 
}

 
var partsDHT = msg.payload.split("\n");

msg.payload = {
    Sensor_Name: partsDHT[0],
    Humidity: partsDHT[1],
    temp: partsDHT[2]
}
return msg;