ESPMetRED an Arduino IDE compatible library to communicate ESP8266 and Node-Red via MQTT

Last updated: September 17, 2017

This library can be used to control ESP8266 nodes from Node-Red. Just burn Examples/ESPMetRED/ESPMetRED.ino to your ESP8266 using Arduino IDE and you are ready to go.

Contents:

Features:

  • Arduino IDE compatible
  • Works with all ESP8266 SOICs such as ESP-12E, ESP-12F and boards such as NodeMCU, WeMos D1 Mini etc.
  • Back and forth communication between ESP8266 and Node-Red using MQTT protocol
  • Auto connect to WiFi or MQTT server in case of failure
  • Support Static IP address configurations
  • Deep sleep support (no connection problem on waking up, connects within 2 seconds)
  • Timer for Timely operations, update Sunrise or Sunset data and preserve it in flash memory
  • Save variables in Flash memory
  • Retrieve variables from Flash memory on booting ESP8266
  • Preserve and restore GPIO state i.e. HIGH or LOW
  • Over the Air (OTA) update
  • More are coming soon!!

Downloads:


Client API:


Save a variable to Flash memory:

  • Instance of ESPMetRED Library
  • Save a  variable in Flash memory
  • Where
    • Object: A String type of data
    • Value: Supports “int”, “long” data types
  • Return: true or false

Retrieve a variable value from Flash memory:

  • Instance of ESPMetRED Library
  • Get a variable value out of Flash memory
  • Where
    • Object: A String type of data
  • Return: “int” or “long” type of data

Switch a pin to LOW or HIGH:

  • Instance of ESPMetRED Library
  • Switch a pin to HIGH
  • Where
    • Pin: An “int” data type
    • Value: An “int” data type (0 or 1)
  • Return: true or false
  • New state of a pin will be automatically saved in flash memory
  • To retrieve state of a pin
  • Where
    • 12 is pin number in “String” data type

Note:

  • Don’t forget to set the pin as output

Fetch Time, Sunrise and Sunset:

  • Instance of ESPMetRED Library
  • Retrieve system time
  • Save time to Flash memory
  • Where
    • Time: JSON property used for recording time in Flash memory
  • Retrieve sunrise
  • Retrieve sunset
Note:

  • Retrieved time is in milliseconds
    • Applicable for “system time”, “sunrise” and “sunset”

Publish data to server:

  • Instance of ESPMetRED Library
  • Send data to MQTT_PUBLISH_TOPIC
  • Where
    • Payload: String data type

 

  • Sending JSON data to MQTT_PUBLISH_TOPIC
  • Where
    • Topic: String data type
    • Payload: String data type
  • JsonString(Topic, Payload) function will return a String data into the JSON format i.e. "Topic":"Payload"

 

  • Send data to a specific topic
  • Where
    • MQTT_Topic: const char data type
    • Payload: String data type

 

  • Sending JSON data to a specific topic
  • Where
    • Topic: String data type
    • Payload: String data type
    • MQTT_Topic: const char data type
  • JsonString(Topic, Payload) function will return a String data into the JSON format i.e. "Topic":"Payload"

Custom callback function in Arduino IDE sketch for MQTT message received:

If you want to get data received from MQTT subscribed topic right in your sketch, follow me;

  • Instance of ESPMetRED Library
  • Add callback in void setup()  function
  • Add callback function at the end after or before void loop()  function
Note:

  • Payload received is String data type and is not processed in JSON object
  • Used the following commands to convert Payload into JSON object
  • Suppose Payload contains the following information
  • To get data out of “TOPIC” and “PAYLOAD” i.e. “MyTopic” and “1”, respectively, use the following commands

Static IP Address:

  • To assign static IP address, add three variables to your sketch
  • Change XX with your desired IP address e.g. 192, 168, 50, 100
  • Rest of the values should resembles your WiFi network
  • Now instantiate ESPMetRED Library
 


Server API:


Pulling Node System Information:

  • Add “INJECT” node in Node-Red
  • Choose “{} JSON” option in “Payload”
  • Add the following code in “Payload” field

  • Hit done
  • Add “mqtt out” node in Node-Red
  • Add server settings according to this post
  • In “Topic” field add your subscribed topic (Arduino sketch) e.g. ESPMetRED/IN

  • Hit done
  • Join both nodes and click on “Deploy”

  • Now hit Inject node button and you will get node system information
  • Don’t forget to add “mqtt in” node with “debug” topic attached with debug node as all kind of system information will go on “debug” topic.


Push Time data to ESP8266 units:

  • Create a “MQTT in” node and enter ntp/IN  in Topic field
  • Add json node and link it to “MQTT in” node 
  • Add a “function” node and add following code in to it
  • Name the “function” node as “Time Push”
  • Add “MQTT out” node and leave it as it is
  • Link all of these nodes as shown below and hit “Deploy”

All of the code works fine but  context.global.sunrise and context.global.sunset  (global variables in Node-Red to store sunrise and sunset data) needs to be configured. There are variety of ways to get sunrise and sunset data for example, Bigtimer performs the job with ease.

  • Install Bigtimer and Grab Bigtimer node with in Node-Red and configure it as following

  • Don’t forget to change “Latitude” and “Longitude” according to your location
  • Grab another “function” node and add follow code to it
Now it will save sunrise and sunset to global   context.global.sunrise and context.global.sunset  variables. A “debug” node will show debug information when it will save data to these variables, however, it is not mandatory.

Note:

  • Sunrise or sunset values will be written to global variables for the first time at first sunrise and sunset

Now every time an ESP8266 unit will boot up, It will get time data automatically. Moreover, to update sunrise and sunset data which changes by days, to ESP8266 units, We should do a little more work.

  • Add an “inject” node and select “interval” in “Repeat”
  • Add your desired time interval such as “12” and “hours” and hit done
  • Copy the “function” node as created above with the name of “Time Push” by Ctrl + C keys and paste it using Ctrl + V keys
  • Add “MQTT out” node and add ntp/OUT  in Topic field, hit done
  • Join all of three nodes as follows and “Deploy” changes

Now “Time”, “Sunrise” and “Sunset” data will be updated to all ESP8266 units every 12 hours automatically.


Drive a gpio right from Node-Red:

  • Publish the following code on MQTT_SUBSCRIBE_TOPIC in the same way you do in Pulling Node System Information
  • Change
    • “12”: Your desired gpio
    • “0”: Desired state either “0” or “1”

 

18 thoughts on “ESPMetRED an Arduino IDE compatible library to communicate ESP8266 and Node-Red via MQTT

  1. Bob says:

    Hi, can you tell me why you use JSON formatting to transmit data by MQTT between the ESP and the Node-Red application. As a beginner I would have though simple strings would be easier to use.
    However, a really great library and blog from which I have learnt a great deal.

    Thank you

    • Ahmed says:

      Hi Bob,

      You can use simple String with this library by sending whatever information on your subscribed topic. Have a look on

      ESPMetRED/Examples/CallBack/CallBack.ino

      However all that information will go directly to your own sketch and you’ll have to define all your required functions.

      As far as your question about JSON is concerned, It is very easy to get required information from a JSON data especially when there are number of sensor nodes exchanging a lot of information to each other.

      Let’s suppose, you just want to pull a gpio HIGH and you send a command from Node-Red to the sensor node like this “GPIO-13-HIGH”, you will have to separate all three sub-units of this command which might be “GPIO Change command”, “Pin Number” and “State” which might be not so easy especially for beginners, while if you send a valid JSON string, on parsing you will get all this stuff with ease. One more thing, you can set any type of variable such as int, String etc. etc. from the JSON output. For example;

      {“TOPIC”:”GPIO”, “PIN”:”12″, “VALUE”:”0″}

      On parsing above command, you can get PIN number as integer type variable e.g. int pinNumber = jObject[“PAYLOAD”]. And if our command is like the following;

      {“TOPIC”:”SYSTEM”, “PAYLOAD”:”INFO”}

      Above command contains “INFO” as payload which is a String data type, one can define a String type variable on parsing JSON data e.g. String testCommand = jObject[“PAYLOAD”];

      As beginner, you would prefer simple Strings but soon after you will dive deep into this world, You will feel need of something more organized.

  2. gabriel says:

    I am getting several error when try to compile the example. this is the log:


    Arduino:1.8.1 (Windows 10), Tarjeta:"NodeMCU 1.0 (ESP-12E Module), 80 MHz, 115200, 4M (3M SPIFFS)"
    C:\Users\xinux\Documents\Arduino\libraries\ESPMetRED-master\ESPMetRED.cpp:35:6: error: prototype for 'void ESPMetRED::joinWiFi()' does not match any in class 'ESPMetRED'
    void ESPMetRED::joinWiFi()
    In file included from C:\Users\xinux\Documents\Arduino\libraries\ESPMetRED-master\ESPMetRED.cpp:8:0:
    C:\Users\xinux\Documents\Arduino\libraries\ESPMetRED-master/ESPMetRED.h:57:10: error: candidate is: boolean ESPMetRED::joinWiFi()
    boolean joinWiFi();
    C:\Users\xinux\Documents\Arduino\libraries\ESPMetRED-master\ESPMetRED.cpp:65:6: error: prototype for 'void ESPMetRED::joinMqTT()' does not match any in class 'ESPMetRED'
    void ESPMetRED::joinMqTT()
    In file included from C:\Users\xinux\Documents\Arduino\libraries\ESPMetRED-master\ESPMetRED.cpp:8:0:
    C:\Users\xinux\Documents\Arduino\libraries\ESPMetRED-master/ESPMetRED.h:59:10: error: candidate is: boolean ESPMetRED::joinMqTT()
    boolean joinMqTT();

    C:\Users\xinux\Documents\Arduino\libraries\ESPMetRED-master\ESPMetRED.cpp:81:6: error: prototype for 'void ESPMetRED::keepalive()' does not match any in class 'ESPMetRED'

    void ESPMetRED::keepalive()

    ^

    In file included from C:\Users\xinux\Documents\Arduino\libraries\ESPMetRED-master\ESPMetRED.cpp:8:0:

    C:\Users\xinux\Documents\Arduino\libraries\ESPMetRED-master/ESPMetRED.h:60:10: error: candidate is: boolean ESPMetRED::keepalive()

    boolean keepalive();

    exit status 1
    Error compilación en tarjeta NodeMCU 1.0 (ESP-12E Module).

    Could you please take a little time to teache me why I am getting this errors? I am a newbie on this 🙂 Thanks in advance

    • gabriel says:

      It works like a charm! I was playing with all the examples. Could you please add some example on how to implement OTA updates in node-red/esp sketch? I was trying but I was rounding in a loop of errors 🙂 Thanks in advance fron the other side of the wolrd!

      • gabriel says:

        What I was trying so far: Using bigtimer to set upgrades at night when ESP are doing nothing , OTA update sketch example fron lmroy pubsubclient with some (not well done, I guess) modifications . My idea is to everytime I have an update spread it to several ESP`s at home overnight and getting a report using node-red-dashboard to know status (updated/on queue/error) to let me know how was done on the morning. (by now nice ideas and bad code 🙂 )

        • Ahmed says:

          OTA update has already implemented and works just sending “OTA” command as “PAYLOAD” on to “SYSTEM” “TOPIC” in Json format i.e. {“TOPIC”:”SYSTEM”, “PAYLOAD”:”OTA”}. Currently OTA works with local web server such as nginx at Port 82 (Normally server port is 80 which We are using for some other purpose and currently using Port 82 for OTA) which means you have to upload binary file to your server and send command to your sensor node afterwards. Sensor node will check for new binary file if it detects new file, it will proceed otherwise cancels OTA. Establishing nginx server needs another tutorial, and We are also working on uploading firmware directly from Arduino IDE via WiFi. As this tutorial is still in progress and many things have not explained yet. Stay tuned for updates.

          • Gabriel says:

            I guess that should have a review cause that is ok if you have one o more of the same type node, but in you have more that one iot device to update, for example one relay controller and 2 temperature sensors (3 ESP8266 for example) the code don´t know what is the correct bin file. I will try to think about it (with my humble tiny knowledge of course, if you don´t mind of course)

          • Waqas Ahmed says:

            I’ll try my best to write down a tutorial as soon as possible on “How to build up a web server for OTA update for ESP8266” and “Where to find the binary file”. After that you’ll be able to accomplish this task with out any issue.

  3. James says:

    Hi
    First a HUGE thank you for your work – this is my very first Arduino project and your code is working GREAT & I have my NodeMCU & RaspberryPi happily talking to each other over MQTT 🙂

    Can I ask one request please – When my NodeMCU connects I would really like that a message is sent to Node-RED to say it has connected – I’ve tried adding the below line to end end of my void setup code, but nothing is sent 🙁

    client.Publish(MQTT_PUBLISH_TOPIC,”NodeMCUconnected”);

    Any help much appreciated!

    Thanks
    James

    • Waqas Ahmed says:

      Hello James. May you please share your sketch so that I see why does it not publish message at your specific topic?

      • James says:

        HI Waqas

        Many thanks for agreeing to take a look 🙂

        Very clumsy code below as Im a real beginner… but everything works apart from the ESP does not ‘announce’ itself when connected 🙁
        However, I can happily send commands from NodeRed and they are received and work fine!(the bit that does not work is the bit below & above the “+++++++” comment

        I suppose it may be a problem on NodeRed side..

        Here’s the code : https://pastebin.com/BdPBEiED

        Thanks!!
        James

      • Waqas Ahmed says:

        I have reviewed your code. There were a lot of mistakes you made. Such as you initialized serial library (Serial.begin(115200)) lot of times. If you use ESPMetRED, you don’t need to initialize it. Library also handles WiFi and MQTT communication by it’s own along with OTA updates. There is no need to include all those stuff (such as WiFi.begin(ssid, password); ArduinoOTA etc. etc.). When you use client. Publish(“topic”, “payload”) function, it publishes data on to your specified topic. If you want to publish something over “const char* MQTT_PUBLISH_TOPIC” already defined by you, just use “Publish(“payload”)” function.I have tried to make your code cleaner a bit and it got compiled successfully. Have a look on it by visiting https://pastebin.com/GceR7qzD.

        • James says:

          Hi Waqas
          So sorry for such a slow response! I have been away for a week & only now get back to my computer..
          Thank you SO much for spending some time on this, it is REALLY appreciated!! I did not have time to test it yet, but I’m really looking forward to trying it out.
          Thank you again!
          James

  4. JMBM says:

    I’m a beginner to IoT and found your library to be very useful and isolate me from some complexities. Thank you for your contribution.

    I think I detected an error in file ESPMetRED/Examples/CallBack/CallBack.ino

    In line 21 program calls “client.setCallback(callback);”, and it should be “client.setcallback(callback);” (no uppercase c)

    This was driving me crazy with a compilation error!

    Thanks again.

    • Waqas Ahmed says:

      Yes. It can be used with any ESP8266 or ESP8285 based product. However, OTA functionality will not work on MCUs with 1M Flash e.g. ESP8285 or 1M ESP8266 ESP-01.

Leave a Reply

Your email address will not be published. Required fields are marked *