Current Cost Power Usage Monitor (Going in the backdoor with Software Defined Radio)

As you have probably guessed from my previous blog posts I am slowly pimping (or in the words of the misses, making a mess) my house with a variety of Internet of Things devices such as Phillips Hue, various sensors and so on. After discussing IoT devices with my work colleague the other day he showed me screenshots his fancy power monitoring solution and gave me a link to the guys who make the device he was using a “Current Cost”, so I promptly ordered a refurbished unit from the Current Cost shop on eBay. I went for the ENVI, a slightly older model but at the bargain price of £20 including delivery.

Today the ENVI arrived and I was eager to get it up and running. Inside the box is a CT clamp with an attached battery operated box, the display base unit and a power adapter for powering the base unit. The base unit has USB serial support for getting an XML feed from the device, however I was curious to see if I could get the power readings without the USB cable as it uses a custom connector and the only cables I could find are selling for £12 on eBay. According to the compliance documentation for the product the CT clamp unit transmits over 433mhz, sounds like an SDR job to me. Installation was easy I attached the CT clamp on the positive inlet cable in the meter cupboard after the meter and main supply fuse, paired it up to the display unit using the included instructions and within a minute I had a frequently updating display of the current power draw in my house.

Next lets see if we can intercept some messages from the CT Clamp to the display unit using the SDR. I used my trusty NooElec NESDR Mini 2+ with the supplied out of the box antenna for the job. I knew from the Current Cost documentation that the display updated every 6 seconds, so lets have a look around the 433mhz frequency for something with a 6 second interval. Here on the waterfall at 433.92mhz we see these lines which match up with every 6 seconds ish with a small red segment which we can assume is the packet from the CT clamp specially as the display updates each time one of these packets appears on the waterfall.

Now we have found the assumed frequency lets try passing it to rtl_433 on the off chance that someone has already reverse engineered / made a decoder in rtl_433 for the Current Cost devices. First I had to boot up my Linux VM and pass the RTL USB device through to the VM as I have never managed to get any librtlsdr stuff to compile on my Mac. I then installed rtl_433 as follows, you’ll need cmake, gcc and librtlsdr-dev etc installed prior to compiling rtl_433.

git clone
cd rtl_433
mkdir build
cd build
cmake ../
sudo make install

Now, time to run rtl_433 and see if we get any data.

rputt@rtl-sdr:~$ rtl_433 -f 433920000
Found 1 device(s):
  0:  Realtek, RTL2838UHIDIR, SN: 00000001

Using device 0: Generic RTL2832U OEM
Found Rafael Micro R820T tuner
Exact sample rate is: 250000.000414 Hz
Sample rate set to 250000.
Bit detection level set to 0 (Auto).
Tuner gain set to Auto.
Reading samples in async mode...
Tuned to 433920000 Hz.
2017-06-14 18:37:25 :	CurrentCost TX
	Device Id:	 1636
	Power 0:	 363 W
	Power 1:	 0 W
	Power 2:	 0 W
2017-06-14 18:37:32 :	CurrentCost TX
	Device Id:	 1636
	Power 0:	 373 W
	Power 1:	 0 W
	Power 2:	 0 W
2017-06-14 18:37:37 :	CurrentCost TX
	Device Id:	 1636
	Power 0:	 375 W
	Power 1:	 0 W
	Power 2:	 0 W
2017-06-14 18:37:43 :	CurrentCost TX
	Device Id:	 1636
	Power 0:	 376 W
	Power 1:	 0 W
	Power 2:	 0 W

Seems to have the necessary decoding stuff implement, does the job great without no further configuration. Interestingly even when unplugging the paired base station the CT clamp unit appears to keep sending out readings. I guess the pairing is very simplistic where the base station just remembers the clamp’s Device Id and listens out only for packets from that clamp, great I can unplug the base station and put it back in the box… Next steps to write a simple Python wrapper around rtl_433 to submit the power consumption metrics back to my sensors API.

Internet of Things: TTNMapper Progress

A few weeks ago I decided I’d like to compare the results of Radio Mobile Online and my actual LoRa coverage using TTNMapper, and although I have not walked all roads within the coverage prediction I have walked the roads around my house and the results are quite underwhelming so far. Here is a very primitive overlay of my TTNMapper results vs the shading from Radio Mobile Online.

You can quite clearly see the difference between the Radio Mobile Online and TTNMapper map types and the area where the coverage stops in real life is far inferior compared to the Radio Mobile Online prediction, there are a number of reasons for this…

a) Radio Mobile Online only takes into account terrain not buildings, and unfortunately my house is walled in by other housing and apartment buildings.

b) Radio Mobile Online probably assumes that you are using a proper antenna on a proper mount outside of the building, unfortunately my LoRa gateway is a bad home made antenna inside of a window on the ground floor.

c) I am unsure of the resolution of the terrain data in Radio Mobile Online, it is safe to say I live in a small dip in the landscape that is very slight, but significant enough to affect coverage in my opinion, if the resolution of Radio Mobile Online terrain data is not fairly high it’ll probably miss the fact my house is lower than the surrounding terrain.

d) My tests were only using SF7 as I only have an el cheapo single channel, single spread factor receive only gateway.

All in all quite disappointing compared to the results from others who have had antennas on top of hills with clear line of sight over many miles, but not really a bad thing as all of my LoRa devices are in my house anyway. Hopefully after I move soon we should see some much nicer results (my new house is on top of a hill). However for sure unless you have perfect conditions I would not take the Radio Mobile Online predictions too seriously when working out your LoRa coverage.

Internet of Things: Mapping The Things Network LoRa Coverage with TTN Mapper

Out of curiosity I have been trying to establish the range in which my LoRa gateway can receive packets, first off I started using the tools by Radio Mobile Online which allows you to determine coverage estimates based on the antenna positioning details and the height profile of the area around your antenna, you can give it a go yourself over at

I was quite suprised to see the range of a kilometre or so from my house considering I live in a small dip in the landscape and thought the estimate was unlikely. Therefore I wanted to try and validate the coverage for myself, however I have no GPS module for Arduino so I either had to buy a GPS sensor and get this to periodically send the location over LoRa and then go for a walk with this device powered on or have a complex arrangement of tracking my position on my smart phone and then trying to match my position to the time stamps of the packets received by the gateway, this obviously would take some time to get setup, so I searched to see if anyone else had tried mapping LoRa networks when I discovered, a community effort to map The Things Network coverage.

To use TTN Mapper you have to have a LoRa node which periodically sends a packet to The Things Network via a gateway, you then walk with this device powered up and your smart phone running the TTN Mapper app, the app listens for packets from your node via MQTT and when it receives a packet submits the receive signal strength, receiving gateway and GPS location to the TTN Mapper API which publishes the coverage data onto the TTN Mapper website for everyone to view. Looking at the Netherlands and Switzerland on these maps shows how popular LoRa is in these countries and some cities have very good coverage indeed.

Here is the coverage map for Zurich for example…

All in all pretty good mapping for a community driven LoRa “war driving” effort so to speak.

Anyway, what about the coverage for my own gateway? Well I realised I need to somehow power up the LoRa node on the move and due to the laws surrounding duty cycle I will have to rate limit my mapping so it may take some time and walking slowly around the local area to get decent resolution. I am off to charge up the largest USB power bank I own and tommorow I will hit the road in a breif stroll around the neighbourhood to test out how TTN Mapper performs and to get an idea of what my coverage looks like. I will then compare the results to the mapping from Radio Mobile Online to see if there are any similarities. Keep an eye out on my blog over the next few days for the results.

Basic Malware Analysis with Cuckoo Sandbox

Recently I have been working on a project to provide a File Intelligence API within the OpenStack ecosystem (OpenStack Nemesis) which can be used to find out information about a given file based on it’s hashes, the plan is to support multiple pluggable backends for processing files which are unknown to the system and provide a verdict of if the file is malicious and other data about the file like its type, or if it’s an image the contents of the image, or if it’s a document provide a summary of the document and so on.

As part of the process it has been imperative to choose the first pluggable backend for providing file analysis, whilst long term it is hoped there will be multiple backends developed for a range of software including commercial sandboxes and scanners it struck me as important for the first backend available to be opensource and freely available so people can experiment with the API without having to commit financially by more than a few virtual machines. Initial experimentation with ClamAV, Cuckoo Sandbox and a selection of Python libraries for discovering MIME type, header details etc… yielded that a combination of these approaches can be used to construct meaningful meta data about a given file. Cuckoo in particular is very impressive, with some minor modification it was relatively easy to get Cuckoo to run it’s executions inside of Nova and communicate using a Neutron tenant network, post execution the software analyses the artifacts collected against a bunch of community contributed signatures and suggests a score of how malicious a file is and provides enough information to categorise it somewhat based on it’s characteristics.

Lets take a look at Cuckoo, first Cuckoo was installed to a Nova instance as per the documentation at The Cuckoo server was configured with 2 interfaces, one on the public network and another to a tenant network where the executions would happen. The Cuckoo server acted as NAT for a tenant network configured on the second interface so the execution slaves could access the internet (via Cuckoo rooter) and none of the execution slaves had direct public network access. The execution slave was built as per the Cuckoo documentation and then imaged so further instances could be built and at the end of an execution rebuilt to a known state. Out of the box Cuckoo does not support Nova to provide virtual machines for execution for initial experientation a custom machine type was monkey patched in as a proof of concept, however long term this needs a more major rework to enable Cuckoo to autoscale executions, in it’s current form factor only one image can be used on a fixed number of Nova instances.

You can see my monkey patched code in my fork of Cuckoo here – I would highly suggest if you’d like to give this a go you compare the changes to the upstream master branch of Cuckoo and rebase the machinery addition onto the current master.

Now that the Nova machinery is patched onto Cuckoo and the Cuckoo server and execution slave instances are up and running lets try running some file analysis and see what comes up. It should be mentioned the results may be highly variable and this is in no means a sure way to identify malware with high accuracy, malware commonly has various anti-sandbox techniques such as not running in a virtual machine, not running if the host has unusual usage patterns (e.g. being very new, not having commonly installed applications, or not having any signs of a physical user doing stuff on the machine), or simply just sleeping for longer than a typical analysis window. There are some steps that can be taken to alleviate these such as ensuring the host looks used, installing common software such as Microsoft Office and Adobe Reader (you’ll probably want to execute these file types anyway to see if there are any nasty payloads hidden in them so your execution slaves will probably need these installing anyway) or simply running the execution slave on a physical machine (Ironic can come in handy here). Long term it will be up to the deployer of Nemesis which pluggable backends they would like to use and how these backends are configured.

For the test samples were submitted via the Cuckoo CLI tool and then the meta data and artifacts considered in the web UI, long term the plan would be to upload the artifacts to Swift for storage and the meta data, along with meta data from other scanners and file analysis tools, to be passed to Nemesis API so it can be queried. Let’s submit some samples, first up lets send it some benign executable like PuTTY.

root@cuckoo-server:~# cuckoo submit putty.exe
'putty.exe' added as task with ID #53

Ok, so what results came back for PuTTY? Well it scored 1.2/10 on the maliciousness signature score in Cuckoo suggesting it is “potentially malicious”, lets have a look at the signatures which were picked up from the execution artifacts…

Here we can see there are a few signatures for PuTTY and some screenshots of the software running in the execution environment. The signatures are as followed:
(information): The executable is signed.
(warning): Potentially malicious URLs were found in the process memory dump.
(critical): PuTTY Files, registry keys or mutexes detected.

These signatures may suprise you, of course PuTTY contains stuff related to PuTTY but why is this dangerous? It can only be assumed a fair bit of malware out there is using PuTTY for communication of some kind, and who can blame them? SSH is a good protocol, it’s encrypted and allows tunneling of traffic without setting up and complicated tunnels, however overall the signatures appear to be fairly accurate and the score seems justified. Now lets try executing something far more dangerous via Cuckoo, malware flavour of the week WannaCrypt.

root@cuckoo-server:~# cuckoo submit wannacrypt.exe
'wannacrypt.exe' added as task with ID #54

Now to checkout what Cuckoo thought of WannaCrypt. Coming in with a maliciousness score of 9/10 and the following signatures…

Ok, this has a whole load of warnings and critical signatures including installing Tor, deleting shadow copy, changing over 500 files on the system (as it encrypts them), listening on multiple ports, delaying execution, poking WMI and network interfaces and having a high level of entropy. It screams of malicious behaviour, and now for the screenshots of the execution environment running WannaCrypt.

As you can see you can learn alot from the execution of a file and this sort of analysis would be very advantageous to a file intelligence API for catching emergent threats which are not yet identified by typical AV scanners, of course with this approach there are chances of high false positive and high false negative rates so one should be wary and keep this in mind.

Internet of Things: LoRa DHT22 Temperature & Humidity Sensor on Breadboard

Continuing with my mission to convert my ESP8266 based sensors into LoRaWAN enabled sensors I pushed on with getting a DHT22 on breadboard and getting the code written to submit temperature and humidity over LoRa.

If you’d like to replicate this mini project you’ll need the following:

  • Arduino Uno or equivalent
  • SX1272 / SX1276 LoRa module
  • DHT22
  • Breadboard for hooking everything up
  • Miscellaneous electronics stuff such as jumper cables, stuff you’d find in your tool box.
  • IBM’s LMIC & Adafruit’s DHT22 Arduino Libraries (both can be installed via manage libraries in Arduino IDE)

First I hooked up the SX1276 module to my Arduino Uno as per my previous post, and connected the DHT22 to 5v VCC, ground and pin 7 on the relevant DHT22 pins. The pins on the DHT22 can be identified by looking at the meshed plastic side of the device, the pin on the left is VCC followed by data, not connected and ground.

Next I moved onto the code, I started by merging the DHT22 and LMIC samples together however upon uploading the sketch it became apparent the sketch was too big for the 32kb of the ATMega328-PU, so I continued by shrinking the code by removing some of the serial output and removing unused remnants of the LMIC example, this allowed me to fit the sketch on the Arduino (just!) but also made debugging super hard so I turned to TheThingsNetwork Slack channel to discuss an alternative strategy.

As a result Matthijs Kooijman came forward with his fork of LMIC which included the ability to use a more lightweight AES encryption framework which although slower was 8kb smaller. To replace the original LMIC from IBM installed by the Arduino IDE download Matthijs’s fork from and copy and overwrite the files in your Arduino library’s LMIC folder. Now when compiling the sketch it came to a much more acceptable 23kb leaving plenty of room to add back in the debug serial prints.

Here is the results of my efforts… A fully working DHT22 and LMIC combo which periodically transmits temperature and humidity readings over LoRa, remember before using this code to set your network key, application key and device id prior to uploading to your board.

Download DHT22 & LMIC Arduino Sketch

Now to see it in action! With everything connected up I placed the sensor in range of a TTN LoRa Gateway and powered it up. Now to check TheThingsNetwork and see what kind of data is coming from the device…

Ok, so you are probably thinking wtf is that payload. As you know with LoRa you need to keep the payload as small as possible to reduce airtime and hence reduce duty cycle (a legal restriction in the EU), I decided upon having only one decimal place within my measurements so what you see here is the hex of the temperature and humidty float appended to each other, 31 34 2E 30 is the temperature and 35 34 2E 37 is the humidity.

Lets convert these into something human readable using Python.

Python 3.5.2 (v3.5.2:4def2a2901a5, Jun 26 2016, 10:47:25) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import binascii
>>> binascii.unhexlify('31342e30')
>>> binascii.unhexlify('35342e37')

OK looks like the temperature is 14°C and relative humidity is 54.7%, sounds about right considering the device in next to an open window in the shade and given the current weather today.

Next steps will involve moving the electronics on to a more permanent board (probably veroboard / stripboard), mounting the device in a small enclosure and subscribing to the TTN’s MQTT and publishing the results to a fancy dashboard.

Internet of Things: LoRa Hello World

Another LoRa transceiver module finally arrived from China today, now I have two of the things I decided to crack on with building a LoRa push button using an Arduino Uno and the LoRa module. Eventually this will be developed into a small circuit on strip board consisting of the LoRa module, an ATMega328-PU and a DHT22 to make my first LoRa powered temperature and humidity sensor to replace the ESP8266 based sensors I have today.

First steps as per all the other LoRa modules was to solder on some cables with appropriate headers at the other end, for this purpose I used a ribbon of male to male dupont style cables and chopped the connectors off one end, tinned the wires and soldered them to the VCC, GND, MISO, MOSI, NSS, SCK, RESET, DIO0 and DIO1 pads of the module. This module is a unbranded SX1276 module I found on eBay which is more cost effective than the RFM95W which was used for the single channel gateway, the downside is it has smaller pads than the RFM95W so soldering is a little more tricky, but with practice and a fine tip soldering iron you can acheive an acceptable result. Here is a chance for you to laugh at my soldering again…

Next I hooked the module up to an Arduino Uno as per the following pins… You shouldn’t really do this because the logic pins of the Uno are 5v and could damage the module, but I gathered for a quick test it would be ok, and if I fried it they are cheap enough. When mounting this permanently on a DHT22 sensor board I will use some voltage dividers to bring the 5v ATMega328-PU logic down to 3.3v for the module.

SX1276 Pins Arduino Uno Pins
VCC 3.3v
SCK 13
NSS 10
DIO0 2
DIO1 3

Next I plugged the Uno in to my laptop and launched the Arduino IDE. The intention is to send the packets to The Things Network via my single channel gateway, hence we need to find a LoRaWAN library. The most common appears to be the IBM LMIC Framework, go to Sketch -> Include Library -> Manage Libraries to download LMIC to your Arduino IDE.

Once the library is installed open the IBM LMIC Framework TTN example from File -> Examples, don’t upload it to your board yet, there are a few mods we need to make before we upload the sketch to the board. Open up your browser and visit, if you do not have an account already sign up. Once in the console go to the Applications tab and create a new application. Once you have made an application go to the devices tab of the application and register a new device, as over the air activation is not available for these sorts of modules via a single channel gateway just fill out the form with dummy data and click register.

Once you get to the device information page shown above click on Settings to begin the device personalisation. On the settings page change device activation method from OAA to ABP and click save, the page will refresh and the device’s address,  network and application keys will be generated for you. Copy these into your sketch in MSB format, you can get the keys in MSB format by clicking the ‘<>’ button next to their text box. You should end up with something that looks like this…

Scroll down the code a bit further until you get to the pin mapping comment. Modify the NSS pin to 10, RST pin to 9 and DIO pins to 2 as per below.

Next keep scrolling until you find the channel setup statements, comment out all the channels other than the channel served by your single channel gateway, in my case all channels except for 868.1mhz. If you are in range of a full 8 channel gateway you can skip this step.

Now you are ready to upload the sketch to the board, once it boots up it should transmit a packet saying “Hello, world!” once every 60 seconds, however the LMIC library does observe duty cycle limitations, so it may be delayed as appropriate. Before you hit the upload sketch button open TheThingsNetwork console, go to your application and select the device you created and open the data tab. Now upload your sketch, providing you are in range of a TheThingsNetwork LoRaWAN gateway you should see packets arriving in the console, if you do not see packets arriving as expected you can use the Arduino IDE serial monitor to look for debug messages from the Arduino, you’ll need to connect with a baudrate of 115200. Once you see the packets in your TheThingsNetwork console you know everything is working as expected, you can now expand the functionality to initiate the do_send function upon some other trigger rather than the timed callback set on line 105, e.g. a button press.

You can expand each packet to show meta data about the packet, for example which gateway forwarded it and the signal strength. The payload is presented as hex, but if you convert it back to ascii then the original contents of mydata in the Arduino sketch can be read.

echo "48656C6C6F2C20776F726C6421" | xxd -r -p
Hello, world!

Great all seems to be working just fine, next I will move on to combining this with periodic polling of data from a DHT22 and moving the circuit to veroboard with a ATMega328-PU standalone rather than having the bulk of the Uno. I may also do some range test to see how far away I can position a sensor, however as my area has a lot of buildings and not many hills I think the range will be quite small compared to some of the many kilometer range tests we see on YouTube for LoRa. Stay tuned for updates.

Internet of Things: Wohoo, First LoRa Packet Received!

Wohoo, whilst taking a look over the logs today, although I have no LoRa nodes myself, I noticed that my Single Channel LoRa Gateway had received it’s first packet from somebody and here it is…

stat update: {"stat":{"time":"2017-05-08 22:00:49 GMT","lati":0.00000,"long":0.00000,"alti":0,"rxnb":0,"rxok":0,"rxfw":0,"ackr":0.0,"dwnb":0,"txnb":0,"pfrm":"Single Channel Gateway","mail":"","desc":""}}
Packet RSSI: -113, RSSI: -112, SNR: -11, Length: 9
rxpk update: {"rxpk":[{"tmst":1621788918,"chan":0,"rfch":0,"freq":868.100000,"stat":1,"modu":"LORA","datr":"SF7BW125","codr":"4/5","lsnr":-11,"rssi":-113,"size":9,"data":"xoT77qK24c5x"}]}
stat update: {"stat":{"time":"2017-05-08 22:01:19 GMT","lati":0.00000,"long":0.00000,"alti":0,"rxnb":1,"rxok":1,"rxfw":0,"ackr":0.0,"dwnb":0,"txnb":0,"pfrm":"Single Channel Gateway","mail":"","desc":""}}

Proof it works atleast, of course we do not know what the data contains as it is encrypted with another The Things Network user’s application key but atleast somebody nearby is getting heard by my gateway and hopefully their packets are getting forwarded on to their application via The Things Network. Oh well, hopefully only another couple of weeks before my LoRa transceiver shipment from China arrives so we can start converting my current ESP8266 based sensors into LoRa equivalents, then we can see the end to end process.

If you want to check out how I built my Single Channel LoRa Gateway checkout Internet of Things: Single Channel LoRa Gateway Shopping List and Internet of Things: Single Channel LoRa Gateway Build.

Internet of Things: Single Channel LoRa Gateway Build

Today the final outstanding item on the Single Channel LoRa Gateway Shopping List arrived in the post, and I now have everything needed to get my Single Channel LoRa Gateway up and running and posting data back to The Things Network. If you’d like to replicate the project follow the guide below to get your gateway up and running.

First lets get the hardware side of things sorted out. Take 8 female to female jumpers and chop off the connectors at one end, and then tin the wires ready for soldering. The raw ends will be soldered to the LoRa transceiver and the female dupont style connectors will be plugged into the Raspberry Pi’s GPIO header.

Solder a cable to each of the following pins; DI00, 3.3v, MISO, MOSI, SCK, RESET, NSS and the GND pin next to the MISO pin. Do not solder the GND cable to any other GND point of the board, it must be the one next to the MISO pin. I found I had to switch out my soldering iron tip to a very fine tip as the pads on the transceiver module are pretty small.

Once you have all the cables for the GPIO header connected to the RFM95W LoRa module solder an 8cm length of solid core bell wire to the ANT pin on the module, and another 8cm length of solid core bell wire to the GND pin next to the ANT pin, these wires should face opposite directions at a 180 degree angle and will act as a small dipole antenna.

Next connect up the LoRa transceiver to the Raspberry Pi GPIO header as per the following. Pin 1 is marked on the board.

RFM95W Raspberry Pi
3.3V 1
DI00 7
NSS 22
SCK 23

I assembled my Raspberry Pi in a case with a gap for the cables to pop out from the GPIO connector. Here is the result, and also an opportunity for you to laugh at my bad soldering…

If you have not already loaded an OS onto your Raspberry Pi SD card you should do so now, I’ll be using Raspbian Jessie Lite although you are free to use any distribution that supports wiringpi. Once you have the OS on the sdcard connect up the Raspberry Pi to your network and boot it up. I’ll be configuring everything over SSH, if you want SSH to start by default on first boot place an empty file called ssh in the boot partition of the SD card, else on first boot SSH will remain in a stopped state.

After the Pi has booted up login and sudo up to the root user and then run the raspi-config utility.

rputt@loragw:~$ sudo su
root@loragw:~# raspi-config

From the menu select interfacing options and then select SPI, you need to ensure SPI is enabled prior to continuing configuration of the Pi.

If the configuration asks if you’d like to reboot say yes and reboot your Raspberry Pi. Next continue to install the wiringpi, gcc and git packages from your OS repository.

root@loragw:~# apt-get update
root@loragw:~# apt-get install -y wiringpi git gcc

Once the packages are installed we can continue to clone the single channel gateway packet forwarder from GitHub.

root@loragw:~# git clone

Now lets change directory into the repository we just cloned and check a few items in the source code…

root@loragw:~# cd single_chan_pkt_fwd
root@loragw:~/single_chan_pkt_fwd# vi main.cpp

In the file find the “#Define SERVER1” line and replace the IP address with, the IP in the repository is for an old The Things Network Router which has been retired, the IP provided in this blog is for, you can of course customise this value for a more appropriate host in your region. Next find the “uint32_t freq” line in the file and ensure it is set for the relevant frequency for your region, in the EU the frequency is 868100000 for 868.1mhz. Save the file and run the make command to build a binary from the code.

root@loragw:~/single_chan_pkt_fwd# make
g++ main.o base64.o -lwiringPi -o single_chan_pkt_fwd

Next run the single_chan_pkt_fwd binary and once it has done it’s initialisation press ctrl+c to stop its execution, if the hardware has been connected correctly and the software compiled successfully then you should see something like the following…

root@loragw:~/single_chan_pkt_fwd# ./single_chan_pkt_fwd 
SX1276 detected, starting.
Gateway ID: ba:12:e3:ff:ff:42:4e:1b
Listening at SF7 on 868.100000 Mhz.
stat update: {"stat":{"time":"2017-05-06 14:37:10 GMT","lati":0.00000,"long":0.00000,"alti":0,"rxnb":0,"rxok":0,"rxfw":0,"ackr":0.0,"dwnb":0,"txnb":0,"pfrm":"Single Channel Gateway","mail":"","desc":""}}

Open a web browser and visit, if you already have an account login, if you are new to The Things Network sign up for a new account, once you are logged in go to the console, click the gateways button and the click register gateway.

Complete the form as per the following:

  • Protocol – Packet Forwarder
  • Gateway EUI – Copy the Gateway ID given when you first run the single_chan_pkt_fwd binary.
  • Description – A friendly name for your gateway.
  • Frequency Plan – Select the frequency plan for your region, this should match the frequency given when you run single_chan_pkt_fwd, if it does not then you probably have the wrong LoRa module for your region, you should order one with the relevant frequency for your region, running a LoRa module in the wrong frequency band may be illegal in your country.

You should end up with something that looks a bit like this…

Next click register gateway. The page will refresh and the gateway status should say “not connected”, now rerun the single_chan_pkt_fwd binary on your Raspberry Pi, the gateway should now check into The Things Network and the webpage should update to show a connected status and say it was last seen within the last 30 seconds or so.

That’s it, your LoRa single channel packet forwarder gateway is now up and running and capable of sending stuff like sensor readings to The Things Network. Remember there are a few limitations to running a single channel gateway compared to a full 8 channel gateway…

  • This particular implementation is uni directional, it can only receive packets from LoRa devices, not transmit data back.
  • The gateway only operates on 1 channel and 1 spread factor, in this case 868.1mhz and SP7, therefore you should update any LoRa node that you want to submit data from to only use the 868.1mhz channel not the usual 8 channel round robin.
  • If you like you can edit the channel it listens on and the spread factor by editing the frequency and spread factor in the main.cpp file and rerunning the make command.

Obviously when you logout of your Pi SSH session the single_chan_pkt_fwd binary stops running, you can either run this under a screen session or configure SystemD to run the binary as a service to make sure it continues running after you logout.

Whats next? Well, its all well and good having a LoRa gateway setup in your house, but without any data flowing over it’s pretty boring, I am currently waiting on another LoRa transceiver module to turn up (I ordered this one from Ali Express as they are much cheaper direct from China via Ali Express rather than ordering from a UK seller on eBay, but the delivery times suck) once that arrives I’ll continue to convert my ESP8266 DHT22 project to a LoRa DHT22 project and have it submit it’s readings over my LoRa gateway. Unfortunately until this arrives I have no real way of proving if my gateway does in fact work or not. Keep watching the blog for updates.

Internet of Things: Single Channel LoRa Gateway Shopping List

Currently my Internet of Things Things are quite limited, in my house I have a DIY DHT22 powered Temperature & Humidity Sensor and Phillips Hue lighting. However soon providing everything goes to plan I should be moving out of rented accomodation into my own home which greatly improves the flexibility for home automation and cool IoT devices. One theme in common with most Internet of Things Things appears to be the use of radio to a hub of some sort rather than hundreds of ESP8266 modules all over the place, and this of course becomes an attractive proposition as low powered but low bandwidth radio can easily achieve better range compared to WiFi and you can have a much higher density of IoT devices multiplexed on some radio technology in comparison to connected WiFi devices on consumer grade WiFi access points.

After a bit of research I decided I should start experimentation with this concept and I stumbled accross LoRa, low power, wide area networking for IoT usage. LoRa is particularly attractive because of The Things Network (TTN), an open network of LoRa gateways which allows your thing to latch on to any TTN LoRa gateway and start transmitting or indeed recieving it’s data, it’s also attractive as LoRa connected devices can be made uber cheap with modules as low as around £10 on eBay from China. My first intention was to convert my DHT22 humidity sensor to a LoRa based device, so I researched if I could use a nearby TTN LoRa gateway for this purpose. Unfortunately the closest gateway is a good 5 miles away from my house and although LoRa has proven to have very good range when line of sight can be achieved and with carefully planned conditions I knew it was almost impossible to contact this gateway from my house purely because of the landscape and the fact in the UK most TTN LoRa gateways are limited range DIY devices. Further more there are no LoRa gateways in the area I plan on moving to, so to go down the LoRa route I’d need to host my own gateway long term anyway.

Back to the drawing board, before I could convert my DHT22 ESP8266 powered sensor to LoRa I’d need to establish my own LoRa gateway. Initially this seemed to be a huge blocker, although the LoRa modules for use in IoT devices are cheap, a decent LoRa gateway seemed to be in the region of £200 for a compliant 8 channel device. Then I found the saviour Andreas Spiess a YouTuber from Switzerland with a selection of LoRa based experiments on his channel. He documented how he found a blog detailing the creation of a cheap 1 channel LoRa gateway using a Raspberry Pi and a cheap LoRa transiever module, he also explains the limitations of a 1 channel gateway, for example on your connected things you need to force 1 channel operation and you will be able to multiplex less devices. However for the low number of devices I plan on creating I am sure this will provide a cheap entry point and with the heads up from Andreas reducing alot of headaches along the way.

So I ordered the parts to start building a single channel gateway. Here is the shopping list, I already had some of the parts laying around in my electronics box… Of course if you would like to replicate the project you’ll need some tools like wire strippers, a soldering iron and so on.

1x Raspberry Pi
1x Raspberry Pi Power Supply
1x MicroSD Card (8GB+)
1x RFM95W LoRa Transceiver Module
8x Female to Female Jumper Wires

These items are due to arrive over the next couple of days, then I will start to build my single channel LoRa gateway and I’ll document progress as I go. Once the gateway is online I’ll order a few other LoRa modules and start playing with sending and receiving packets on a breadboard to ensure it is working as expected.

My adventure into SSD caching with ZFS (Home NAS)

Recently I decided to throw away my old defunct 2009 MacBook Pro which was rotting in my cupboard and I decided to retrieve the only useful part before doing so, the 80GB Intel SSD I had installed a few years earlier. Initially I thought about simply adding it to my desktop as a bit of extra space but in 2017 80GB really wasn’t worth it and then I had a brainwave… Lets see if we can squeeze some additional performance out of my HP Microserver Gen8 NAS running ZFS by installing it as a cache disk.

I installed the SSD to the cdrom tray of the Microserver using a floppy disk power to SATA power converter and a SATA cable, unfortunately it seems the CD ROM SATA port on the motherboard is only a 3gbps port although this didn’t matter so much as it was an older 3gbps SSD anyway. Next I booted up the machine and to my suprise the disk was not found in my FreeBSD install, then I realised that the SATA port for the CD drive is actually provided by the RAID controller, so I rebooted into intelligent provisioning and added an additional RAID0 array with just the 1 disk to act as my cache, in fact all of the disks in this machine are individual RAID0 arrays so it looks like just a bunch of disks (JBOD) as ZFS offers additional functionality over normal RAID (mainly scrubbing, deduplication and compression).


Lets have a look at the zpool before adding the cache drive to make sure there are no errors or uglyness…

[root@netdisk] ~# zpool status vol0
  pool: vol0
 state: ONLINE
  scan: scrub repaired 0 in 3h21m with 0 errors on Wed Apr 26 16:40:16 2017

	NAME                                            STATE     READ WRITE CKSUM
	vol0                                            ONLINE       0     0     0
	  mirror-0                                      ONLINE       0     0     0
	    gptid/b8ebf047-25cd-11e7-b7e2-000c29ccceef  ONLINE       0     0     0
	    gptid/c0df3410-25cd-11e7-b7e2-000c29ccceef  ONLINE       0     0     0
	  mirror-1                                      ONLINE       0     0     0
	    gptid/c8b062cb-25cd-11e7-b7e2-000c29ccceef  ONLINE       0     0     0
	    gptid/355426bd-25ce-11e7-b7e2-000c29ccceef  ONLINE       0     0     0

errors: No known data errors

Now lets prep the drive for use in the zpool using gpart. I want to split the SSD into two seperate partitions, one for L2ARC (read caching) and one for ZIL (write caching). I have decided to split the disk into 20GB for ZIL and 50GB for L2ARC. Be warned using 1 SSD like this is considered unsafe because it is a single point of failure in terms of delayed writes (a redundant configuration with 2 SSDs would be more appropriate) and the heavy write cycles on the SSD from the ZIL is likely to kill it over time.

[root@netdisk] ~# gpart create -s gpt da6
[root@netdisk] ~# gpart show da6
=>       34  150994877  da6  GPT  (72G)
         34  150994877       - free -  (72G)
[root@netdisk] ~# gpart add -t freebsd-zfs -s 20G da6
da6p1 added
[root@netdisk] ~# gpart add -t freebsd-zfs -s 50G da6
da6p2 added
[root@netdisk] ~# gpart show da6
=>       34  150994877  da6  GPT  (72G)
         34   41943040    1  freebsd-zfs  (20G)
   41943074  104857600    2  freebsd-zfs  (50G)
  146800674    4194237       - free -  (2.0G)
[root@netdisk] ~# glabel list da6p1
Geom name: da6p1
1. Name: gptid/a3d641e5-2c03-11e7-9146-000c29ccceef
   Mediasize: 21474836480 (20G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 17408
   Mode: r0w0e0
   secoffset: 0
   offset: 0
   seclength: 41943040
   length: 21474836480
   index: 0
1. Name: da6p1
   Mediasize: 21474836480 (20G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 17408
   Mode: r0w0e0
[root@netdisk] ~# glabel list da6p2
Geom name: da6p2
1. Name: gptid/a6dd9f4d-2c03-11e7-9146-000c29ccceef
   Mediasize: 53687091200 (50G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 17408
   Mode: r0w0e0
   secoffset: 0
   offset: 0
   seclength: 104857600
   length: 53687091200
   index: 0
1. Name: da6p2
   Mediasize: 53687091200 (50G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 17408
   Mode: r0w0e0

Now we have a couple of partitions configured on the disk as described above, they can now be added to the ZFS zpool.

[root@netdisk] ~# zpool add vol0 log gptid/a3d641e5-2c03-11e7-9146-000c29ccceef
[root@netdisk] ~# zpool add vol0 cache gptid/a6dd9f4d-2c03-11e7-9146-000c29ccceef
[root@netdisk] ~# zpool status vol0
  pool: vol0
 state: ONLINE
  scan: scrub repaired 0 in 3h21m with 0 errors on Wed Apr 26 16:40:16 2017

	NAME                                            STATE     READ WRITE CKSUM
	vol0                                            ONLINE       0     0     0
	  mirror-0                                      ONLINE       0     0     0
	    gptid/b8ebf047-25cd-11e7-b7e2-000c29ccceef  ONLINE       0     0     0
	    gptid/c0df3410-25cd-11e7-b7e2-000c29ccceef  ONLINE       0     0     0
	  mirror-1                                      ONLINE       0     0     0
	    gptid/c8b062cb-25cd-11e7-b7e2-000c29ccceef  ONLINE       0     0     0
	    gptid/355426bd-25ce-11e7-b7e2-000c29ccceef  ONLINE       0     0     0
	  gptid/a3d641e5-2c03-11e7-9146-000c29ccceef    ONLINE       0     0     0
	  gptid/a6dd9f4d-2c03-11e7-9146-000c29ccceef    ONLINE       0     0     0

errors: No known data errors

Now it’s time to see if adding the cache has made much of a difference. I suspect not as my Home NAS sucks, it is a HP Microserver Gen8 with the crappy Celeron CPU and only 4GB RAM, anyway, lets test it and find out. First off lets throw fio at the mount point for this zpool and see what happens both with the ZIL and L2ARC enabled and disabled.

Caching Disabled FIO Results

[root@netdisk] ~# fio disk_perf.fio
random_rw: (g=0): rw=rw, bs=4K-4K/4K-4K/4K-4K, ioengine=posixaio, iodepth=1
Starting 1 thread
random_rw: Laying out IO file(s) (1 file(s) / 1024MB)
Jobs: 1 (f=1): [M(1)] [100.0% done] [10806KB/10580KB/0KB /s] [2701/2645/0 iops] [eta 00m:00s]
random_rw: (groupid=0, jobs=1): err= 0: pid=101627: Fri Apr 28 12:21:47 2017
  read : io=522756KB, bw=10741KB/s, iops=2685, runt= 48669msec
    slat (usec): min=2, max=2559, avg= 9.31, stdev=21.63
    clat (usec): min=2, max=69702, avg=34.66, stdev=563.31
     lat (usec): min=12, max=69760, avg=43.97, stdev=567.97
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    2], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[   14],
     | 70.00th=[   14], 80.00th=[   15], 90.00th=[   15], 95.00th=[   15],
     | 99.00th=[   27], 99.50th=[   59], 99.90th=[ 9536], 99.95th=[10304],
     | 99.99th=[15040]
  write: io=525820KB, bw=10804KB/s, iops=2701, runt= 48669msec
    slat (usec): min=2, max=8124, avg=15.87, stdev=43.54
    clat (usec): min=2, max=132195, avg=305.77, stdev=1973.41
     lat (usec): min=21, max=132485, avg=321.64, stdev=1977.32
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    4], 60.00th=[   24],
     | 70.00th=[   24], 80.00th=[   25], 90.00th=[   25], 95.00th=[   35],
     | 99.00th=[10304], 99.50th=[10560], 99.90th=[11712], 99.95th=[19584],
     | 99.99th=[50432]
    lat (usec) : 4=49.44%, 10=4.09%, 20=21.92%, 50=22.63%, 100=0.23%
    lat (usec) : 250=0.08%, 500=0.02%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.90%, 20=0.63%, 50=0.02%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=3.08%, sys=2.92%, ctx=265022, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=130689/w=131455/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: io=522756KB, aggrb=10741KB/s, minb=10741KB/s, maxb=10741KB/s, mint=48669msec, maxt=48669msec
  WRITE: io=525820KB, aggrb=10804KB/s, minb=10804KB/s, maxb=10804KB/s, mint=48669msec, maxt=48669msec
[root@netdisk] ~# 

Caching Enabled FIO Results

[root@netdisk] ~# fio disk_perf.fio
random_rw: (g=0): rw=rw, bs=4K-4K/4K-4K/4K-4K, ioengine=posixaio, iodepth=1
Starting 1 thread
random_rw: Laying out IO file(s) (1 file(s) / 1024MB)
Jobs: 1 (f=1): [M(1)] [100.0% done] [10902KB/10854KB/0KB /s] [2725/2713/0 iops] [eta 00m:00s]
random_rw: (groupid=0, jobs=1): err= 0: pid=101640: Fri Apr 28 12:24:18 2017
  read : io=522756KB, bw=10462KB/s, iops=2615, runt= 49967msec
    slat (usec): min=2, max=23575, avg=12.84, stdev=155.63
    clat (usec): min=2, max=135652, avg=34.24, stdev=644.41
     lat (usec): min=12, max=135654, avg=47.08, stdev=663.37
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    2], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    3],
     | 70.00th=[   13], 80.00th=[   14], 90.00th=[   15], 95.00th=[   15],
     | 99.00th=[   27], 99.50th=[   70], 99.90th=[ 9920], 99.95th=[10432],
     | 99.99th=[11968]
  write: io=525820KB, bw=10523KB/s, iops=2630, runt= 49967msec
    slat (usec): min=2, max=139302, avg=22.47, stdev=556.00
    clat (usec): min=2, max=119615, avg=306.30, stdev=1969.21
     lat (usec): min=20, max=139752, avg=328.78, stdev=2046.20
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    3],
     | 70.00th=[   23], 80.00th=[   24], 90.00th=[   25], 95.00th=[   32],
     | 99.00th=[10176], 99.50th=[10560], 99.90th=[19072], 99.95th=[24192],
     | 99.99th=[44800]
    lat (usec) : 4=62.92%, 10=2.93%, 20=15.75%, 50=16.37%, 100=0.28%
    lat (usec) : 250=0.14%, 500=0.03%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.93%, 20=0.57%, 50=0.04%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=3.21%, sys=2.51%, ctx=265920, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=130689/w=131455/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: io=522756KB, aggrb=10462KB/s, minb=10462KB/s, maxb=10462KB/s, mint=49967msec, maxt=49967msec
  WRITE: io=525820KB, aggrb=10523KB/s, minb=10523KB/s, maxb=10523KB/s, mint=49967msec, maxt=49967msec
[root@netdisk] ~# 


Ok, so the initial result is a little dissapointing, but hardly unexpected, my NAS sucks and there are lots of bottle necks, CPU, memory and the fact only 2 of the SATA ports are 6gbps. There is no real difference performance wise in comparison between the results, the IOPS, bandwidth and latency appear very similar. However lets bare in mind fio is a pretty hardcore disk benchmark utility, how about some real world use cases?

Next I decided to test a few typical file transactions that this NAS is used for, Samba shares to my workstation. For the first test I wanted to test reading a 3GB file over the network with both the cache enabled and disabled, I would run this multiple times to ensure the data is hot in the L2ARC and to ensure the test is somewhat repeatable, the network itself is an uncongested 1gbit link and I am copying onto the secondary SSD in my workstation. The dataset for these tests has compression and deduplication disabled.

Samba Read Test

Attempt    without caching    with caching
1          48.1MB/s           52.2MB/s
2          49.6MB/s           66.4MB/s
3          47.4MB/s           65.6MB/s

Not bad once the data becomes hot in the L2ARC cache reads appear to gain a decent advantage compared to reading from the disk directly. How does it perform when writing the same file back accross the network using the ZIL vs no ZIL.

Samba Write Test

Attempt    without caching    with caching
1          34.2MB/s           57.3MB/s
2          33.6MB/s           55.7MB/s
3          36.7MB/s           57.1MB/s

Another good result in the real world test, this certainately helps the write transfer speed however I do wonder what would happen if you filled the ZIL transferring a very large file, however this is unlikely with my use case as I typically only deal with a couple of files of several hundred megabytes at any given time so a 20GB ZIL should suit me reasonably well.

Is ZIL and L2ARC worth it?

I would imagine with a big beefy ZFS server running in a company somewhere with a large disk pool and lots of users with multiple enterprise level SSD ZIL and L2ARC would be well worth the investment, however at home I am not so sure. Yes I did see an increase in read speeds with cached data and a general increase in write speeds however it is use case dependant. In my use case I rarely access the same file frequently, my NAS primarily serves as a backup and for archived data, and although the write speeds are cool I am not sure its a deal breaker. If I built a new home NAS today I’d probably concentrate the budget on a better CPU, more RAM (for ARC cache) and more disks. However if I had a use case where I frequently accessed the same files and needed to do so in a faster fashion then yes, I’d probably invest in an SSD for caching. I think if you have a spare SSD lying around and you want something fun todo with it, sure chuck it in your ZFS based NAS as a cache mechanism. If you were planning on buying an SSD for caching then I’d really consider your needs and decide if the money can be spent on alternative stuff which would improve your experience with your NAS. I know my NAS would benefit more from an extra stick of RAM and a more powerful CPU, but as a quick evening project with some parts I had hanging around adding some SSD cache was worth a go.