Graphite Server Installation

Following on from my LoRa temperature and humidity sensor and CurrentCost SDR man in the middle it became apparent a time series database was needed to store all the data that was being collected. RRDTool seemed the obvious choice and although it has a good track recorded at dealing with time series data previous interactions with the software left scars. Particularly the fact that data cannot be inserted retrospectively, data must be inserted in chronological order, and also the lack of a modern API.

To overcome the concerns with RRDTool two other tools where considered, Graphite and InfluxDB. InfluxDB has a clear advantage in terms of implementation although it seems pretty heavy and involved, on the other hand Graphite offers familiarity as I have used it before and hence administration and maintenance would be a lot simpler and less of a learning curve due to knowledge of the API.

Graphite Components

Lets get on with the Graphite server installation, Graphite comes in several components…

  • Whisper – the time series data store, file based like RRDTool but without the limitations.
  • Carbon – an API layer exposed by raw socket which indexes, caches, inserts and retrieves data from the whisper files.
  • Graphite-Web – a web layer that provides a HTTP accessible API including rendering of graphs, and a web front end for exploring the data. The API itself is not RESTful but pretty easy to use and learn.

There are also a few other carbon components which can be used to cluster Graphite deployments, although in this case installation will be to a single server with backups on cron to a NAS.

Installation

Installation on Debian is mega easy as packages exist for Graphite in the distribution’s repo and are actively maintained.

apt-get install graphite-carbon graphite-web

This will install and start the Carbon service, the web service is not configured to run and hence needs to be configured with SystemD to run properly. By default Carbon is configured to store data with an interval of 60 seconds for a maximum time period of 24 hours, after this the space in the whisper file will be overwritten with new data just like a round robin database file in RRDTool. In my case I wanted to store the data with high resolution for essentially ever, so I updated the default Carbon storage schema to have a 10 second interval and store data for 30 years, I expect this server will be replaced by then and we can worry about the data migration at a later date. Of course this comes at a large increase in storage cost, and each metric with this resolution and storage period eats 1.1gb of disk, although if you do not have so many metrics (in my case probably a maximum of 10 or so in the end) then it isn’t a concern with today’s hard drive prices. You should edit the storage schema to fit your use case…

vi /etc/carbon/storage-schemas.conf

[carbon]
pattern = ^carbon\.
retentions = 60:90d

[default_10secs_for_30years]
pattern = .*
retentions = 10s:30y

After the storage schema has been configured you should restart the Carbon service, following this you can start writing data to your Graphite server.

service carbon-cache restart

Verifying Functionality

Writing data to your Graphite server is easy using a raw socket, but be warned this is unprotected and is not encrypted so you should only run it within your local network and not out on the internet. If you want to expose the ability to write to your Graphite server to external devices it is recommended to write a wrapper API which implements authentication and SSL. However due to the input to Carbon being a raw socket we can simply write data to the time series database using netcat… Lets add some random stuff to the database to ensure its working properly.

echo "demo.increments 1 `date +%s`" | nc -q0 127.0.0.1 2003
sleep 10
echo "demo.increments 2 `date +%s`" | nc -q0 127.0.0.1 2003
sleep 10
echo "demo.increments 3 `date +%s`" | nc -q0 127.0.0.1 2003
sleep 10
echo "demo.increments 4 `date +%s`" | nc -q0 127.0.0.1 2003
sleep 10
echo "demo.increments 5 `date +%s`" | nc -q0 127.0.0.1 2003
sleep 10
echo "demo.increments 6 `date +%s`" | nc -q0 127.0.0.1 2003
sleep 10
echo "demo.increments 7 `date +%s`" | nc -q0 127.0.0.1 2003
sleep 10
echo "demo.increments 8 `date +%s`" | nc -q0 127.0.0.1 2003
sleep 10
echo "demo.increments 9 `date +%s`" | nc -q0 127.0.0.1 2003
sleep 10
echo "demo.increments 10 `date +%s`" | nc -q0 127.0.0.1 2003

The sleep in here should be sufficient to write the values into separate intervals defines in your storage schema, if you do not sleep for a sufficient period of time the values will be averaged, of course instead of sleeping you could pass in some pre decided time stamps instead of using the date command. After the inserts have ran check out /var/lib/graphite/whisper/demo you should see a file called increments.

ls -alh /var/lib/graphite/whisper/demo
total 1.1G
drwxr-xr-x 2 _graphite _graphite 4.0K Jun 25 20:05 .
drwxr-xr-x 5 _graphite _graphite 4.0K Jun 25 20:05 ..
-rw-r--r-- 1 _graphite _graphite 1.1G Jun 25 20:07 increments.wsp

Here we can see that a whisper file has been created for our metric “increments”, metrics can be nested and the folders within /var/lib/graphite/whisper are arranged accordingly. Whisper files can be interrogated using the Whisper CLI tools…

whisper-info increments.wsp 
maxRetention: 946080000             # equivalent to 30 years at 10 second resolution.
xFilesFactor: 0.5
aggregationMethod: average
fileSize: 1135296028

Archive 0
retention: 946080000
secondsPerPoint: 10
points: 94608000
size: 1135296000
offset: 28

Using the whisper-dump command we can see all the time series data stored in the file.

whisper-dump increments.wsp
--very long output mostly made up of 0 values--

Ok, this proves Carbon is writing to the Whisper files as required, now lets look at the Web UI and graphing the data, for now lets run graphite-web by hand, we can create a SystemD service for this once we are happy everything is functional. When you first run the web server you will need to run graphite-manage syncdb to run database migrations for the Web UI, the database by default is an SQLite file in /var/lib/graphite, although for larger installations you can swap this out for MySQL or PostgreSQL by editing /etc/graphite/local_settings.py.

graphite-manage syncdb
graphite-manage runserver 0.0.0.0:8000

You should now be able to open the Graphite Web UI in your browser on the port specified in the runserver command, the left hand side of the UI is a tree containing all of your data, and the right hand side pane is used for drawing graphs. Expand the tree and click on increments to load in data from your whisper file, this will ask Carbon to fetch the data either from it’s cache or the disk.

You’ll notice the data isn’t really displayed that well, by default the Graphite Composer draws graphs with a 24 hour time window, obviously we only entered the data over 100 seconds so the data is a very small vertical line on the graph, lets change the graph’s time window to an hour so the data is actually readable. Click the select recent data button (5th from the left in the composer window) and change the time range to the past 30 minutes. The graph will now update, we should see the incrementing data we previously submitted to Carbon, of course as our 10 second intervals do not match exactly the intervals of the whisper file some averaging has occurred, we can see this as the data ends around 9, rather than 10 as submitted.

The basic functionality seems to be working great. Exit the graphite-web server by pressing ctrl-c, lets configuring backup and systemd to start the graphite-web service, of course if you prefer you can use Apache with modwsgi or UWSGI and NGINX, after all the graphite-web server is only a Django application, however as its only to be used internally by myself I’ll just run it standalone.

Configuring Graphite-Web

To configure Graphite-Web to run under SystemD create a new service file in /lib/systemd/system such as…

vi /lib/systemd/system/graphite-web.service
[Unit]
  Description=Graphite Web Service

  [Service]
    Type=simple
    ExecStart=/usr/bin/graphite-manage runserver 0.0.0.0:8000

  [Install]
    WantedBy=multi-user.target

Save the file and then run systemctl daemon-reload to configure the service, following this you can test the service by running the service command.

systemctl daemon-reload
service graphite-web start

Now check it’s up and running in your browser!

If the UI loads as per the above then we know SystemD has the service configured ok, lets make it run at boot time.

systemctl enable graphite-web

Configuring Backup to NAS

If you do not have a clustered Graphite server obviously you have a huge single point of failure, not only that but as the data is only stored on one node so in case of failure data loss is somewhat likely. To combat this it is advised to backup your /var/lib/graphite folder to another host, and preferably offsite. In my case I have installed Graphite on my existing web server which already mounts an NFS share from my ZFS powered NAS. This NAS snapshots changes periodically on the shares and replicates them offsite to another ZFS based NAS. Therefore I will just abuse this for backing up my Graphite data.

The share from the NAS is mounted on /mnt/web_backups via AutoFS and the backups are started by cron at 12 every day. The existing backup script will be altered to tar up the Graphite data and dump it into the share, the configuration is also copied for easy restoration at a later date, most of the file has been redacted.

vi /root/do_backups.sh
#!/bin/bash
mysq...
...ache2
tar -zcvf /mnt/web_backups/graphite-data-`date +%Y-%m-%d_%H-%M-%S`.tar.gz /var/lib/graphite
tar -zcvf /mnt/web_backups/graphite-config-`date +%Y-%m-%d_%H-%M-%S`.tar.gz /etc/graphite
tar -zcvf /mnt/web_backups/carbon-config-`date +%Y-%m-%d_%H-%M-%S`.tar.gz /etc/carbon

Now the backup script is ran, and the resulting tar.gz file compared for completeness. If you’d like more information on mounting the NFS mount from the NAS or AutoFS check out the Debian Wiki.

Fetch data via the render API

Now everything is up and running it is possible to fetch data via the Graphite render API, by default Carbon automatically adds some metrics to the time series database about the local machine and statistics about the Carbon cache. Lets query the machine load average via the render API. First lets fetch the last 1 hour of cpu usage in JSON output.

curl 'http://localhost:8000/render/?target=carbon.agents.web-a.cpuUsage&format=json&from=-1hours'

Now lets use the render API to output a PNG file containing a graph, there are a load of different formats which can be rendered including PDF, SVG and many more.

curl 'http://localhost:8000/render/?target=carbon.agents.web-a.cpuUsage&format=png&from=-1hours&width=800&height=600' > ~/test.png
display ~/test.png

Hopefully you should have a CPU usage graph similar to the below outputted into test.png.

Conclusion

Installing Graphite on my existing Debian webserver was pretty quick and painless and works well. Hopefully the above should give you a leg up compared to storing your IoT sensor submissions in RRDTool or in a MySQL DB. Unfortunately it’s probably out of scope todo clustering in my configuration as I don’t really want several servers running in my house due to the heat / noise / power usage etc… The Graphite web UI is useful for quickly viewing the data and allows saving favourites etc… so you can probably get away without having Grafana running. Next I’ll work on integrating my sensors and making some dashboards using the render API.

Current Cost Power Usage Monitor (Going in the backdoor with Software Defined Radio)

As you have probably guessed from my previous blog posts I am slowly pimping (or in the words of the misses, making a mess) my house with a variety of Internet of Things devices such as Phillips Hue, various sensors and so on. After discussing IoT devices with my work colleague the other day he showed me screenshots his fancy power monitoring solution and gave me a link to the guys who make the device he was using a “Current Cost”, so I promptly ordered a refurbished unit from the Current Cost shop on eBay. I went for the ENVI, a slightly older model but at the bargain price of £20 including delivery.

Today the ENVI arrived and I was eager to get it up and running. Inside the box is a CT clamp with an attached battery operated box, the display base unit and a power adapter for powering the base unit. The base unit has USB serial support for getting an XML feed from the device, however I was curious to see if I could get the power readings without the USB cable as it uses a custom connector and the only cables I could find are selling for £12 on eBay. According to the compliance documentation for the product the CT clamp unit transmits over 433mhz, sounds like an SDR job to me. Installation was easy I attached the CT clamp on the positive inlet cable in the meter cupboard after the meter and main supply fuse, paired it up to the display unit using the included instructions and within a minute I had a frequently updating display of the current power draw in my house.

Next lets see if we can intercept some messages from the CT Clamp to the display unit using the SDR. I used my trusty NooElec NESDR Mini 2+ with the supplied out of the box antenna for the job. I knew from the Current Cost documentation that the display updated every 6 seconds, so lets have a look around the 433mhz frequency for something with a 6 second interval. Here on the waterfall at 433.92mhz we see these lines which match up with every 6 seconds ish with a small red segment which we can assume is the packet from the CT clamp specially as the display updates each time one of these packets appears on the waterfall.

Now we have found the assumed frequency lets try passing it to rtl_433 on the off chance that someone has already reverse engineered / made a decoder in rtl_433 for the Current Cost devices. First I had to boot up my Linux VM and pass the RTL USB device through to the VM as I have never managed to get any librtlsdr stuff to compile on my Mac. I then installed rtl_433 as follows, you’ll need cmake, gcc and librtlsdr-dev etc installed prior to compiling rtl_433.

git clone https://github.com/merbanan/rtl_433.git
cd rtl_433
mkdir build
cd build
cmake ../
make
sudo make install

Now, time to run rtl_433 and see if we get any data.

rputt@rtl-sdr:~$ rtl_433 -f 433920000
Found 1 device(s):
  0:  Realtek, RTL2838UHIDIR, SN: 00000001

Using device 0: Generic RTL2832U OEM
Found Rafael Micro R820T tuner
Exact sample rate is: 250000.000414 Hz
Sample rate set to 250000.
Bit detection level set to 0 (Auto).
Tuner gain set to Auto.
Reading samples in async mode...
Tuned to 433920000 Hz.
2017-06-14 18:37:25 :	CurrentCost TX
	Device Id:	 1636
	Power 0:	 363 W
	Power 1:	 0 W
	Power 2:	 0 W
2017-06-14 18:37:32 :	CurrentCost TX
	Device Id:	 1636
	Power 0:	 373 W
	Power 1:	 0 W
	Power 2:	 0 W
2017-06-14 18:37:37 :	CurrentCost TX
	Device Id:	 1636
	Power 0:	 375 W
	Power 1:	 0 W
	Power 2:	 0 W
2017-06-14 18:37:43 :	CurrentCost TX
	Device Id:	 1636
	Power 0:	 376 W
	Power 1:	 0 W
	Power 2:	 0 W

Seems to have the necessary decoding stuff implement, does the job great without no further configuration. Interestingly even when unplugging the paired base station the CT clamp unit appears to keep sending out readings. I guess the pairing is very simplistic where the base station just remembers the clamp’s Device Id and listens out only for packets from that clamp, great I can unplug the base station and put it back in the box… Next steps to write a simple Python wrapper around rtl_433 to submit the power consumption metrics back to my sensors API.

Internet of Things: TTNMapper Progress

A few weeks ago I decided I’d like to compare the results of Radio Mobile Online and my actual LoRa coverage using TTNMapper, and although I have not walked all roads within the coverage prediction I have walked the roads around my house and the results are quite underwhelming so far. Here is a very primitive overlay of my TTNMapper results vs the shading from Radio Mobile Online.

You can quite clearly see the difference between the Radio Mobile Online and TTNMapper map types and the area where the coverage stops in real life is far inferior compared to the Radio Mobile Online prediction, there are a number of reasons for this…

a) Radio Mobile Online only takes into account terrain not buildings, and unfortunately my house is walled in by other housing and apartment buildings.

b) Radio Mobile Online probably assumes that you are using a proper antenna on a proper mount outside of the building, unfortunately my LoRa gateway is a bad home made antenna inside of a window on the ground floor.

c) I am unsure of the resolution of the terrain data in Radio Mobile Online, it is safe to say I live in a small dip in the landscape that is very slight, but significant enough to affect coverage in my opinion, if the resolution of Radio Mobile Online terrain data is not fairly high it’ll probably miss the fact my house is lower than the surrounding terrain.

d) My tests were only using SF7 as I only have an el cheapo single channel, single spread factor receive only gateway.

All in all quite disappointing compared to the results from others who have had antennas on top of hills with clear line of sight over many miles, but not really a bad thing as all of my LoRa devices are in my house anyway. Hopefully after I move soon we should see some much nicer results (my new house is on top of a hill). However for sure unless you have perfect conditions I would not take the Radio Mobile Online predictions too seriously when working out your LoRa coverage.

Internet of Things: Mapping The Things Network LoRa Coverage with TTN Mapper

Out of curiosity I have been trying to establish the range in which my LoRa gateway can receive packets, first off I started using the tools by Radio Mobile Online which allows you to determine coverage estimates based on the antenna positioning details and the height profile of the area around your antenna, you can give it a go yourself over at http://www.cplus.org/rmw/rmonline.html.

I was quite suprised to see the range of a kilometre or so from my house considering I live in a small dip in the landscape and thought the estimate was unlikely. Therefore I wanted to try and validate the coverage for myself, however I have no GPS module for Arduino so I either had to buy a GPS sensor and get this to periodically send the location over LoRa and then go for a walk with this device powered on or have a complex arrangement of tracking my position on my smart phone and then trying to match my position to the time stamps of the packets received by the gateway, this obviously would take some time to get setup, so I searched to see if anyone else had tried mapping LoRa networks when I discovered ttnmapper.org, a community effort to map The Things Network coverage.

To use TTN Mapper you have to have a LoRa node which periodically sends a packet to The Things Network via a gateway, you then walk with this device powered up and your smart phone running the TTN Mapper app, the app listens for packets from your node via MQTT and when it receives a packet submits the receive signal strength, receiving gateway and GPS location to the TTN Mapper API which publishes the coverage data onto the TTN Mapper website for everyone to view. Looking at the Netherlands and Switzerland on these maps shows how popular LoRa is in these countries and some cities have very good coverage indeed.

Here is the coverage map for Zurich for example…

All in all pretty good mapping for a community driven LoRa “war driving” effort so to speak.

Anyway, what about the coverage for my own gateway? Well I realised I need to somehow power up the LoRa node on the move and due to the laws surrounding duty cycle I will have to rate limit my mapping so it may take some time and walking slowly around the local area to get decent resolution. I am off to charge up the largest USB power bank I own and tommorow I will hit the road in a breif stroll around the neighbourhood to test out how TTN Mapper performs and to get an idea of what my coverage looks like. I will then compare the results to the mapping from Radio Mobile Online to see if there are any similarities. Keep an eye out on my blog over the next few days for the results.

Basic Malware Analysis with Cuckoo Sandbox

Recently I have been working on a project to provide a File Intelligence API within the OpenStack ecosystem (OpenStack Nemesis) which can be used to find out information about a given file based on it’s hashes, the plan is to support multiple pluggable backends for processing files which are unknown to the system and provide a verdict of if the file is malicious and other data about the file like its type, or if it’s an image the contents of the image, or if it’s a document provide a summary of the document and so on.

As part of the process it has been imperative to choose the first pluggable backend for providing file analysis, whilst long term it is hoped there will be multiple backends developed for a range of software including commercial sandboxes and scanners it struck me as important for the first backend available to be opensource and freely available so people can experiment with the API without having to commit financially by more than a few virtual machines. Initial experimentation with ClamAV, Cuckoo Sandbox and a selection of Python libraries for discovering MIME type, header details etc… yielded that a combination of these approaches can be used to construct meaningful meta data about a given file. Cuckoo in particular is very impressive, with some minor modification it was relatively easy to get Cuckoo to run it’s executions inside of Nova and communicate using a Neutron tenant network, post execution the software analyses the artifacts collected against a bunch of community contributed signatures and suggests a score of how malicious a file is and provides enough information to categorise it somewhat based on it’s characteristics.

Lets take a look at Cuckoo, first Cuckoo was installed to a Nova instance as per the documentation at http://docs.cuckoosandbox.org/en/latest/. The Cuckoo server was configured with 2 interfaces, one on the public network and another to a tenant network where the executions would happen. The Cuckoo server acted as NAT for a tenant network configured on the second interface so the execution slaves could access the internet (via Cuckoo rooter) and none of the execution slaves had direct public network access. The execution slave was built as per the Cuckoo documentation and then imaged so further instances could be built and at the end of an execution rebuilt to a known state. Out of the box Cuckoo does not support Nova to provide virtual machines for execution for initial experientation a custom machine type was monkey patched in as a proof of concept, however long term this needs a more major rework to enable Cuckoo to autoscale executions, in it’s current form factor only one image can be used on a fixed number of Nova instances.

You can see my monkey patched code in my fork of Cuckoo here – https://github.com/robputt796/cuckoo/tree/nova_machinery. I would highly suggest if you’d like to give this a go you compare the changes to the upstream master branch of Cuckoo and rebase the machinery addition onto the current master.

Now that the Nova machinery is patched onto Cuckoo and the Cuckoo server and execution slave instances are up and running lets try running some file analysis and see what comes up. It should be mentioned the results may be highly variable and this is in no means a sure way to identify malware with high accuracy, malware commonly has various anti-sandbox techniques such as not running in a virtual machine, not running if the host has unusual usage patterns (e.g. being very new, not having commonly installed applications, or not having any signs of a physical user doing stuff on the machine), or simply just sleeping for longer than a typical analysis window. There are some steps that can be taken to alleviate these such as ensuring the host looks used, installing common software such as Microsoft Office and Adobe Reader (you’ll probably want to execute these file types anyway to see if there are any nasty payloads hidden in them so your execution slaves will probably need these installing anyway) or simply running the execution slave on a physical machine (Ironic can come in handy here). Long term it will be up to the deployer of Nemesis which pluggable backends they would like to use and how these backends are configured.

For the test samples were submitted via the Cuckoo CLI tool and then the meta data and artifacts considered in the web UI, long term the plan would be to upload the artifacts to Swift for storage and the meta data, along with meta data from other scanners and file analysis tools, to be passed to Nemesis API so it can be queried. Let’s submit some samples, first up lets send it some benign executable like PuTTY.

root@cuckoo-server:~# cuckoo submit putty.exe
'putty.exe' added as task with ID #53

Ok, so what results came back for PuTTY? Well it scored 1.2/10 on the maliciousness signature score in Cuckoo suggesting it is “potentially malicious”, lets have a look at the signatures which were picked up from the execution artifacts…

Here we can see there are a few signatures for PuTTY and some screenshots of the software running in the execution environment. The signatures are as followed:
(information): The executable is signed.
(warning): Potentially malicious URLs were found in the process memory dump.
(critical): PuTTY Files, registry keys or mutexes detected.

These signatures may suprise you, of course PuTTY contains stuff related to PuTTY but why is this dangerous? It can only be assumed a fair bit of malware out there is using PuTTY for communication of some kind, and who can blame them? SSH is a good protocol, it’s encrypted and allows tunneling of traffic without setting up and complicated tunnels, however overall the signatures appear to be fairly accurate and the score seems justified. Now lets try executing something far more dangerous via Cuckoo, malware flavour of the week WannaCrypt.

root@cuckoo-server:~# cuckoo submit wannacrypt.exe
'wannacrypt.exe' added as task with ID #54

Now to checkout what Cuckoo thought of WannaCrypt. Coming in with a maliciousness score of 9/10 and the following signatures…

Ok, this has a whole load of warnings and critical signatures including installing Tor, deleting shadow copy, changing over 500 files on the system (as it encrypts them), listening on multiple ports, delaying execution, poking WMI and network interfaces and having a high level of entropy. It screams of malicious behaviour, and now for the screenshots of the execution environment running WannaCrypt.

As you can see you can learn alot from the execution of a file and this sort of analysis would be very advantageous to a file intelligence API for catching emergent threats which are not yet identified by typical AV scanners, of course with this approach there are chances of high false positive and high false negative rates so one should be wary and keep this in mind.

Internet of Things: LoRa DHT22 Temperature & Humidity Sensor on Breadboard

Continuing with my mission to convert my ESP8266 based sensors into LoRaWAN enabled sensors I pushed on with getting a DHT22 on breadboard and getting the code written to submit temperature and humidity over LoRa.

If you’d like to replicate this mini project you’ll need the following:

  • Arduino Uno or equivalent
  • SX1272 / SX1276 LoRa module
  • DHT22
  • Breadboard for hooking everything up
  • Miscellaneous electronics stuff such as jumper cables, stuff you’d find in your tool box.
  • IBM’s LMIC & Adafruit’s DHT22 Arduino Libraries (both can be installed via manage libraries in Arduino IDE)

First I hooked up the SX1276 module to my Arduino Uno as per my previous post, and connected the DHT22 to 5v VCC, ground and pin 7 on the relevant DHT22 pins. The pins on the DHT22 can be identified by looking at the meshed plastic side of the device, the pin on the left is VCC followed by data, not connected and ground.

Next I moved onto the code, I started by merging the DHT22 and LMIC samples together however upon uploading the sketch it became apparent the sketch was too big for the 32kb of the ATMega328-PU, so I continued by shrinking the code by removing some of the serial output and removing unused remnants of the LMIC example, this allowed me to fit the sketch on the Arduino (just!) but also made debugging super hard so I turned to TheThingsNetwork Slack channel to discuss an alternative strategy.

As a result Matthijs Kooijman came forward with his fork of LMIC which included the ability to use a more lightweight AES encryption framework which although slower was 8kb smaller. To replace the original LMIC from IBM installed by the Arduino IDE download Matthijs’s fork from https://github.com/matthijskooijman/arduino-lmic and copy and overwrite the files in your Arduino library’s LMIC folder. Now when compiling the sketch it came to a much more acceptable 23kb leaving plenty of room to add back in the debug serial prints.

Here is the results of my efforts… A fully working DHT22 and LMIC combo which periodically transmits temperature and humidity readings over LoRa, remember before using this code to set your network key, application key and device id prior to uploading to your board.

Download DHT22 & LMIC Arduino Sketch

Now to see it in action! With everything connected up I placed the sensor in range of a TTN LoRa Gateway and powered it up. Now to check TheThingsNetwork and see what kind of data is coming from the device…

Ok, so you are probably thinking wtf is that payload. As you know with LoRa you need to keep the payload as small as possible to reduce airtime and hence reduce duty cycle (a legal restriction in the EU), I decided upon having only one decimal place within my measurements so what you see here is the hex of the temperature and humidty float appended to each other, 31 34 2E 30 is the temperature and 35 34 2E 37 is the humidity.

Lets convert these into something human readable using Python.

Python 3.5.2 (v3.5.2:4def2a2901a5, Jun 26 2016, 10:47:25) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import binascii
>>> binascii.unhexlify('31342e30')
b'14.0'
>>> binascii.unhexlify('35342e37')
b'54.7'
>>> 

OK looks like the temperature is 14°C and relative humidity is 54.7%, sounds about right considering the device in next to an open window in the shade and given the current weather today.

Next steps will involve moving the electronics on to a more permanent board (probably veroboard / stripboard), mounting the device in a small enclosure and subscribing to the TTN’s MQTT and publishing the results to a fancy dashboard.

Internet of Things: LoRa Hello World

Another LoRa transceiver module finally arrived from China today, now I have two of the things I decided to crack on with building a LoRa push button using an Arduino Uno and the LoRa module. Eventually this will be developed into a small circuit on strip board consisting of the LoRa module, an ATMega328-PU and a DHT22 to make my first LoRa powered temperature and humidity sensor to replace the ESP8266 based sensors I have today.

First steps as per all the other LoRa modules was to solder on some cables with appropriate headers at the other end, for this purpose I used a ribbon of male to male dupont style cables and chopped the connectors off one end, tinned the wires and soldered them to the VCC, GND, MISO, MOSI, NSS, SCK, RESET, DIO0 and DIO1 pads of the module. This module is a unbranded SX1276 module I found on eBay which is more cost effective than the RFM95W which was used for the single channel gateway, the downside is it has smaller pads than the RFM95W so soldering is a little more tricky, but with practice and a fine tip soldering iron you can acheive an acceptable result. Here is a chance for you to laugh at my soldering again…

Next I hooked the module up to an Arduino Uno as per the following pins… You shouldn’t really do this because the logic pins of the Uno are 5v and could damage the module, but I gathered for a quick test it would be ok, and if I fried it they are cheap enough. When mounting this permanently on a DHT22 sensor board I will use some voltage dividers to bring the 5v ATMega328-PU logic down to 3.3v for the module.

SX1276 Pins Arduino Uno Pins
VCC 3.3v
GND GND
SCK 13
MISO 12
MOSI 11
NSS 10
RESET 9
DIO0 2
DIO1 3

Next I plugged the Uno in to my laptop and launched the Arduino IDE. The intention is to send the packets to The Things Network via my single channel gateway, hence we need to find a LoRaWAN library. The most common appears to be the IBM LMIC Framework, go to Sketch -> Include Library -> Manage Libraries to download LMIC to your Arduino IDE.

Once the library is installed open the IBM LMIC Framework TTN example from File -> Examples, don’t upload it to your board yet, there are a few mods we need to make before we upload the sketch to the board. Open up your browser and visit https://console.thethingsnetwork.org, if you do not have an account already sign up. Once in the console go to the Applications tab and create a new application. Once you have made an application go to the devices tab of the application and register a new device, as over the air activation is not available for these sorts of modules via a single channel gateway just fill out the form with dummy data and click register.

Once you get to the device information page shown above click on Settings to begin the device personalisation. On the settings page change device activation method from OAA to ABP and click save, the page will refresh and the device’s address,  network and application keys will be generated for you. Copy these into your sketch in MSB format, you can get the keys in MSB format by clicking the ‘<>’ button next to their text box. You should end up with something that looks like this…

Scroll down the code a bit further until you get to the pin mapping comment. Modify the NSS pin to 10, RST pin to 9 and DIO pins to 2 as per below.

Next keep scrolling until you find the channel setup statements, comment out all the channels other than the channel served by your single channel gateway, in my case all channels except for 868.1mhz. If you are in range of a full 8 channel gateway you can skip this step.

Now you are ready to upload the sketch to the board, once it boots up it should transmit a packet saying “Hello, world!” once every 60 seconds, however the LMIC library does observe duty cycle limitations, so it may be delayed as appropriate. Before you hit the upload sketch button open TheThingsNetwork console, go to your application and select the device you created and open the data tab. Now upload your sketch, providing you are in range of a TheThingsNetwork LoRaWAN gateway you should see packets arriving in the console, if you do not see packets arriving as expected you can use the Arduino IDE serial monitor to look for debug messages from the Arduino, you’ll need to connect with a baudrate of 115200. Once you see the packets in your TheThingsNetwork console you know everything is working as expected, you can now expand the functionality to initiate the do_send function upon some other trigger rather than the timed callback set on line 105, e.g. a button press.

You can expand each packet to show meta data about the packet, for example which gateway forwarded it and the signal strength. The payload is presented as hex, but if you convert it back to ascii then the original contents of mydata in the Arduino sketch can be read.

echo "48656C6C6F2C20776F726C6421" | xxd -r -p
Hello, world!

Great all seems to be working just fine, next I will move on to combining this with periodic polling of data from a DHT22 and moving the circuit to veroboard with a ATMega328-PU standalone rather than having the bulk of the Uno. I may also do some range test to see how far away I can position a sensor, however as my area has a lot of buildings and not many hills I think the range will be quite small compared to some of the many kilometer range tests we see on YouTube for LoRa. Stay tuned for updates.

Internet of Things: Wohoo, First LoRa Packet Received!

Wohoo, whilst taking a look over the logs today, although I have no LoRa nodes myself, I noticed that my Single Channel LoRa Gateway had received it’s first packet from somebody and here it is…

stat update: {"stat":{"time":"2017-05-08 22:00:49 GMT","lati":0.00000,"long":0.00000,"alti":0,"rxnb":0,"rxok":0,"rxfw":0,"ackr":0.0,"dwnb":0,"txnb":0,"pfrm":"Single Channel Gateway","mail":"","desc":""}}
Packet RSSI: -113, RSSI: -112, SNR: -11, Length: 9
rxpk update: {"rxpk":[{"tmst":1621788918,"chan":0,"rfch":0,"freq":868.100000,"stat":1,"modu":"LORA","datr":"SF7BW125","codr":"4/5","lsnr":-11,"rssi":-113,"size":9,"data":"xoT77qK24c5x"}]}
stat update: {"stat":{"time":"2017-05-08 22:01:19 GMT","lati":0.00000,"long":0.00000,"alti":0,"rxnb":1,"rxok":1,"rxfw":0,"ackr":0.0,"dwnb":0,"txnb":0,"pfrm":"Single Channel Gateway","mail":"","desc":""}}

Proof it works atleast, of course we do not know what the data contains as it is encrypted with another The Things Network user’s application key but atleast somebody nearby is getting heard by my gateway and hopefully their packets are getting forwarded on to their application via The Things Network. Oh well, hopefully only another couple of weeks before my LoRa transceiver shipment from China arrives so we can start converting my current ESP8266 based sensors into LoRa equivalents, then we can see the end to end process.

If you want to check out how I built my Single Channel LoRa Gateway checkout Internet of Things: Single Channel LoRa Gateway Shopping List and Internet of Things: Single Channel LoRa Gateway Build.

Internet of Things: Single Channel LoRa Gateway Build

Today the final outstanding item on the Single Channel LoRa Gateway Shopping List arrived in the post, and I now have everything needed to get my Single Channel LoRa Gateway up and running and posting data back to The Things Network. If you’d like to replicate the project follow the guide below to get your gateway up and running.

First lets get the hardware side of things sorted out. Take 8 female to female jumpers and chop off the connectors at one end, and then tin the wires ready for soldering. The raw ends will be soldered to the LoRa transceiver and the female dupont style connectors will be plugged into the Raspberry Pi’s GPIO header.

Solder a cable to each of the following pins; DI00, 3.3v, MISO, MOSI, SCK, RESET, NSS and the GND pin next to the MISO pin. Do not solder the GND cable to any other GND point of the board, it must be the one next to the MISO pin. I found I had to switch out my soldering iron tip to a very fine tip as the pads on the transceiver module are pretty small.

Once you have all the cables for the GPIO header connected to the RFM95W LoRa module solder an 8cm length of solid core bell wire to the ANT pin on the module, and another 8cm length of solid core bell wire to the GND pin next to the ANT pin, these wires should face opposite directions at a 180 degree angle and will act as a small dipole antenna.

Next connect up the LoRa transceiver to the Raspberry Pi GPIO header as per the following. Pin 1 is marked on the board.

RFM95W Raspberry Pi
3.3V 1
GND 6
DI00 7
RESET 11
NSS 22
MOSI 19
MISO 21
SCK 23

I assembled my Raspberry Pi in a case with a gap for the cables to pop out from the GPIO connector. Here is the result, and also an opportunity for you to laugh at my bad soldering…

If you have not already loaded an OS onto your Raspberry Pi SD card you should do so now, I’ll be using Raspbian Jessie Lite although you are free to use any distribution that supports wiringpi. Once you have the OS on the sdcard connect up the Raspberry Pi to your network and boot it up. I’ll be configuring everything over SSH, if you want SSH to start by default on first boot place an empty file called ssh in the boot partition of the SD card, else on first boot SSH will remain in a stopped state.

After the Pi has booted up login and sudo up to the root user and then run the raspi-config utility.

rputt@loragw:~$ sudo su
root@loragw:~# raspi-config

From the menu select interfacing options and then select SPI, you need to ensure SPI is enabled prior to continuing configuration of the Pi.

If the configuration asks if you’d like to reboot say yes and reboot your Raspberry Pi. Next continue to install the wiringpi, gcc and git packages from your OS repository.

root@loragw:~# apt-get update
root@loragw:~# apt-get install -y wiringpi git gcc

Once the packages are installed we can continue to clone the single channel gateway packet forwarder from GitHub.

root@loragw:~# git clone https://github.com/tftelkamp/single_chan_pkt_fwd

Now lets change directory into the repository we just cloned and check a few items in the source code…

root@loragw:~# cd single_chan_pkt_fwd
root@loragw:~/single_chan_pkt_fwd# vi main.cpp

In the file find the “#Define SERVER1” line and replace the IP address with 52.169.76.203, the IP in the repository is for an old The Things Network Router which has been retired, the IP provided in this blog is for router.eu.thethings.network, you can of course customise this value for a more appropriate host in your region. Next find the “uint32_t freq” line in the file and ensure it is set for the relevant frequency for your region, in the EU the frequency is 868100000 for 868.1mhz. Save the file and run the make command to build a binary from the code.

root@loragw:~/single_chan_pkt_fwd# make
g++ main.o base64.o -lwiringPi -o single_chan_pkt_fwd
root@loragw:~/single_chan_pkt_fwd# 

Next run the single_chan_pkt_fwd binary and once it has done it’s initialisation press ctrl+c to stop its execution, if the hardware has been connected correctly and the software compiled successfully then you should see something like the following…

root@loragw:~/single_chan_pkt_fwd# ./single_chan_pkt_fwd 
SX1276 detected, starting.
Gateway ID: ba:12:e3:ff:ff:42:4e:1b
Listening at SF7 on 868.100000 Mhz.
------------------
stat update: {"stat":{"time":"2017-05-06 14:37:10 GMT","lati":0.00000,"long":0.00000,"alti":0,"rxnb":0,"rxok":0,"rxfw":0,"ackr":0.0,"dwnb":0,"txnb":0,"pfrm":"Single Channel Gateway","mail":"","desc":""}}
^C
root@loragw:~/single_chan_pkt_fwd# 

Open a web browser and visit https://www.thethingsnetwork.com, if you already have an account login, if you are new to The Things Network sign up for a new account, once you are logged in go to the console, click the gateways button and the click register gateway.

Complete the form as per the following:

  • Protocol – Packet Forwarder
  • Gateway EUI – Copy the Gateway ID given when you first run the single_chan_pkt_fwd binary.
  • Description – A friendly name for your gateway.
  • Frequency Plan – Select the frequency plan for your region, this should match the frequency given when you run single_chan_pkt_fwd, if it does not then you probably have the wrong LoRa module for your region, you should order one with the relevant frequency for your region, running a LoRa module in the wrong frequency band may be illegal in your country.

You should end up with something that looks a bit like this…

Next click register gateway. The page will refresh and the gateway status should say “not connected”, now rerun the single_chan_pkt_fwd binary on your Raspberry Pi, the gateway should now check into The Things Network and the webpage should update to show a connected status and say it was last seen within the last 30 seconds or so.

That’s it, your LoRa single channel packet forwarder gateway is now up and running and capable of sending stuff like sensor readings to The Things Network. Remember there are a few limitations to running a single channel gateway compared to a full 8 channel gateway…

  • This particular implementation is uni directional, it can only receive packets from LoRa devices, not transmit data back.
  • The gateway only operates on 1 channel and 1 spread factor, in this case 868.1mhz and SP7, therefore you should update any LoRa node that you want to submit data from to only use the 868.1mhz channel not the usual 8 channel round robin.
  • If you like you can edit the channel it listens on and the spread factor by editing the frequency and spread factor in the main.cpp file and rerunning the make command.

Obviously when you logout of your Pi SSH session the single_chan_pkt_fwd binary stops running, you can either run this under a screen session or configure SystemD to run the binary as a service to make sure it continues running after you logout.

Whats next? Well, its all well and good having a LoRa gateway setup in your house, but without any data flowing over it’s pretty boring, I am currently waiting on another LoRa transceiver module to turn up (I ordered this one from Ali Express as they are much cheaper direct from China via Ali Express rather than ordering from a UK seller on eBay, but the delivery times suck) once that arrives I’ll continue to convert my ESP8266 DHT22 project to a LoRa DHT22 project and have it submit it’s readings over my LoRa gateway. Unfortunately until this arrives I have no real way of proving if my gateway does in fact work or not. Keep watching the blog for updates.

Internet of Things: Single Channel LoRa Gateway Shopping List

Currently my Internet of Things Things are quite limited, in my house I have a DIY DHT22 powered Temperature & Humidity Sensor and Phillips Hue lighting. However soon providing everything goes to plan I should be moving out of rented accomodation into my own home which greatly improves the flexibility for home automation and cool IoT devices. One theme in common with most Internet of Things Things appears to be the use of radio to a hub of some sort rather than hundreds of ESP8266 modules all over the place, and this of course becomes an attractive proposition as low powered but low bandwidth radio can easily achieve better range compared to WiFi and you can have a much higher density of IoT devices multiplexed on some radio technology in comparison to connected WiFi devices on consumer grade WiFi access points.

After a bit of research I decided I should start experimentation with this concept and I stumbled accross LoRa, low power, wide area networking for IoT usage. LoRa is particularly attractive because of The Things Network (TTN), an open network of LoRa gateways which allows your thing to latch on to any TTN LoRa gateway and start transmitting or indeed recieving it’s data, it’s also attractive as LoRa connected devices can be made uber cheap with modules as low as around £10 on eBay from China. My first intention was to convert my DHT22 humidity sensor to a LoRa based device, so I researched if I could use a nearby TTN LoRa gateway for this purpose. Unfortunately the closest gateway is a good 5 miles away from my house and although LoRa has proven to have very good range when line of sight can be achieved and with carefully planned conditions I knew it was almost impossible to contact this gateway from my house purely because of the landscape and the fact in the UK most TTN LoRa gateways are limited range DIY devices. Further more there are no LoRa gateways in the area I plan on moving to, so to go down the LoRa route I’d need to host my own gateway long term anyway.

Back to the drawing board, before I could convert my DHT22 ESP8266 powered sensor to LoRa I’d need to establish my own LoRa gateway. Initially this seemed to be a huge blocker, although the LoRa modules for use in IoT devices are cheap, a decent LoRa gateway seemed to be in the region of £200 for a compliant 8 channel device. Then I found the saviour Andreas Spiess a YouTuber from Switzerland with a selection of LoRa based experiments on his channel. He documented how he found a blog detailing the creation of a cheap 1 channel LoRa gateway using a Raspberry Pi and a cheap LoRa transiever module, he also explains the limitations of a 1 channel gateway, for example on your connected things you need to force 1 channel operation and you will be able to multiplex less devices. However for the low number of devices I plan on creating I am sure this will provide a cheap entry point and with the heads up from Andreas reducing alot of headaches along the way.

So I ordered the parts to start building a single channel gateway. Here is the shopping list, I already had some of the parts laying around in my electronics box… Of course if you would like to replicate the project you’ll need some tools like wire strippers, a soldering iron and so on.

1x Raspberry Pi
1x Raspberry Pi Power Supply
1x MicroSD Card (8GB+)
1x RFM95W LoRa Transceiver Module
8x Female to Female Jumper Wires

These items are due to arrive over the next couple of days, then I will start to build my single channel LoRa gateway and I’ll document progress as I go. Once the gateway is online I’ll order a few other LoRa modules and start playing with sending and receiving packets on a breadboard to ensure it is working as expected.