Internet of Things: Mapping The Things Network LoRa Coverage with TTN Mapper

Out of curiosity I have been trying to establish the range in which my LoRa gateway can receive packets, first off I started using the tools by Radio Mobile Online which allows you to determine coverage estimates based on the antenna positioning details and the height profile of the area around your antenna, you can give it a go yourself over at

I was quite suprised to see the range of a kilometre or so from my house considering I live in a small dip in the landscape and thought the estimate was unlikely. Therefore I wanted to try and validate the coverage for myself, however I have no GPS module for Arduino so I either had to buy a GPS sensor and get this to periodically send the location over LoRa and then go for a walk with this device powered on or have a complex arrangement of tracking my position on my smart phone and then trying to match my position to the time stamps of the packets received by the gateway, this obviously would take some time to get setup, so I searched to see if anyone else had tried mapping LoRa networks when I discovered, a community effort to map The Things Network coverage.

To use TTN Mapper you have to have a LoRa node which periodically sends a packet to The Things Network via a gateway, you then walk with this device powered up and your smart phone running the TTN Mapper app, the app listens for packets from your node via MQTT and when it receives a packet submits the receive signal strength, receiving gateway and GPS location to the TTN Mapper API which publishes the coverage data onto the TTN Mapper website for everyone to view. Looking at the Netherlands and Switzerland on these maps shows how popular LoRa is in these countries and some cities have very good coverage indeed.

Here is the coverage map for Zurich for example…

All in all pretty good mapping for a community driven LoRa “war driving” effort so to speak.

Anyway, what about the coverage for my own gateway? Well I realised I need to somehow power up the LoRa node on the move and due to the laws surrounding duty cycle I will have to rate limit my mapping so it may take some time and walking slowly around the local area to get decent resolution. I am off to charge up the largest USB power bank I own and tommorow I will hit the road in a breif stroll around the neighbourhood to test out how TTN Mapper performs and to get an idea of what my coverage looks like. I will then compare the results to the mapping from Radio Mobile Online to see if there are any similarities. Keep an eye out on my blog over the next few days for the results.

Basic Malware Analysis with Cuckoo Sandbox

Recently I have been working on a project to provide a File Intelligence API within the OpenStack ecosystem (OpenStack Nemesis) which can be used to find out information about a given file based on it’s hashes, the plan is to support multiple pluggable backends for processing files which are unknown to the system and provide a verdict of if the file is malicious and other data about the file like its type, or if it’s an image the contents of the image, or if it’s a document provide a summary of the document and so on.

As part of the process it has been imperative to choose the first pluggable backend for providing file analysis, whilst long term it is hoped there will be multiple backends developed for a range of software including commercial sandboxes and scanners it struck me as important for the first backend available to be opensource and freely available so people can experiment with the API without having to commit financially by more than a few virtual machines. Initial experimentation with ClamAV, Cuckoo Sandbox and a selection of Python libraries for discovering MIME type, header details etc… yielded that a combination of these approaches can be used to construct meaningful meta data about a given file. Cuckoo in particular is very impressive, with some minor modification it was relatively easy to get Cuckoo to run it’s executions inside of Nova and communicate using a Neutron tenant network, post execution the software analyses the artifacts collected against a bunch of community contributed signatures and suggests a score of how malicious a file is and provides enough information to categorise it somewhat based on it’s characteristics.

Lets take a look at Cuckoo, first Cuckoo was installed to a Nova instance as per the documentation at The Cuckoo server was configured with 2 interfaces, one on the public network and another to a tenant network where the executions would happen. The Cuckoo server acted as NAT for a tenant network configured on the second interface so the execution slaves could access the internet (via Cuckoo rooter) and none of the execution slaves had direct public network access. The execution slave was built as per the Cuckoo documentation and then imaged so further instances could be built and at the end of an execution rebuilt to a known state. Out of the box Cuckoo does not support Nova to provide virtual machines for execution for initial experientation a custom machine type was monkey patched in as a proof of concept, however long term this needs a more major rework to enable Cuckoo to autoscale executions, in it’s current form factor only one image can be used on a fixed number of Nova instances.

You can see my monkey patched code in my fork of Cuckoo here – I would highly suggest if you’d like to give this a go you compare the changes to the upstream master branch of Cuckoo and rebase the machinery addition onto the current master.

Now that the Nova machinery is patched onto Cuckoo and the Cuckoo server and execution slave instances are up and running lets try running some file analysis and see what comes up. It should be mentioned the results may be highly variable and this is in no means a sure way to identify malware with high accuracy, malware commonly has various anti-sandbox techniques such as not running in a virtual machine, not running if the host has unusual usage patterns (e.g. being very new, not having commonly installed applications, or not having any signs of a physical user doing stuff on the machine), or simply just sleeping for longer than a typical analysis window. There are some steps that can be taken to alleviate these such as ensuring the host looks used, installing common software such as Microsoft Office and Adobe Reader (you’ll probably want to execute these file types anyway to see if there are any nasty payloads hidden in them so your execution slaves will probably need these installing anyway) or simply running the execution slave on a physical machine (Ironic can come in handy here). Long term it will be up to the deployer of Nemesis which pluggable backends they would like to use and how these backends are configured.

For the test samples were submitted via the Cuckoo CLI tool and then the meta data and artifacts considered in the web UI, long term the plan would be to upload the artifacts to Swift for storage and the meta data, along with meta data from other scanners and file analysis tools, to be passed to Nemesis API so it can be queried. Let’s submit some samples, first up lets send it some benign executable like PuTTY.

root@cuckoo-server:~# cuckoo submit putty.exe
'putty.exe' added as task with ID #53

Ok, so what results came back for PuTTY? Well it scored 1.2/10 on the maliciousness signature score in Cuckoo suggesting it is “potentially malicious”, lets have a look at the signatures which were picked up from the execution artifacts…

Here we can see there are a few signatures for PuTTY and some screenshots of the software running in the execution environment. The signatures are as followed:
(information): The executable is signed.
(warning): Potentially malicious URLs were found in the process memory dump.
(critical): PuTTY Files, registry keys or mutexes detected.

These signatures may suprise you, of course PuTTY contains stuff related to PuTTY but why is this dangerous? It can only be assumed a fair bit of malware out there is using PuTTY for communication of some kind, and who can blame them? SSH is a good protocol, it’s encrypted and allows tunneling of traffic without setting up and complicated tunnels, however overall the signatures appear to be fairly accurate and the score seems justified. Now lets try executing something far more dangerous via Cuckoo, malware flavour of the week WannaCrypt.

root@cuckoo-server:~# cuckoo submit wannacrypt.exe
'wannacrypt.exe' added as task with ID #54

Now to checkout what Cuckoo thought of WannaCrypt. Coming in with a maliciousness score of 9/10 and the following signatures…

Ok, this has a whole load of warnings and critical signatures including installing Tor, deleting shadow copy, changing over 500 files on the system (as it encrypts them), listening on multiple ports, delaying execution, poking WMI and network interfaces and having a high level of entropy. It screams of malicious behaviour, and now for the screenshots of the execution environment running WannaCrypt.

As you can see you can learn alot from the execution of a file and this sort of analysis would be very advantageous to a file intelligence API for catching emergent threats which are not yet identified by typical AV scanners, of course with this approach there are chances of high false positive and high false negative rates so one should be wary and keep this in mind.

Internet of Things: LoRa DHT22 Temperature & Humidity Sensor on Breadboard

Continuing with my mission to convert my ESP8266 based sensors into LoRaWAN enabled sensors I pushed on with getting a DHT22 on breadboard and getting the code written to submit temperature and humidity over LoRa.

If you’d like to replicate this mini project you’ll need the following:

  • Arduino Uno or equivalent
  • SX1272 / SX1276 LoRa module
  • DHT22
  • Breadboard for hooking everything up
  • Miscellaneous electronics stuff such as jumper cables, stuff you’d find in your tool box.
  • IBM’s LMIC & Adafruit’s DHT22 Arduino Libraries (both can be installed via manage libraries in Arduino IDE)

First I hooked up the SX1276 module to my Arduino Uno as per my previous post, and connected the DHT22 to 5v VCC, ground and pin 7 on the relevant DHT22 pins. The pins on the DHT22 can be identified by looking at the meshed plastic side of the device, the pin on the left is VCC followed by data, not connected and ground.

Next I moved onto the code, I started by merging the DHT22 and LMIC samples together however upon uploading the sketch it became apparent the sketch was too big for the 32kb of the ATMega328-PU, so I continued by shrinking the code by removing some of the serial output and removing unused remnants of the LMIC example, this allowed me to fit the sketch on the Arduino (just!) but also made debugging super hard so I turned to TheThingsNetwork Slack channel to discuss an alternative strategy.

As a result Matthijs Kooijman came forward with his fork of LMIC which included the ability to use a more lightweight AES encryption framework which although slower was 8kb smaller. To replace the original LMIC from IBM installed by the Arduino IDE download Matthijs’s fork from and copy and overwrite the files in your Arduino library’s LMIC folder. Now when compiling the sketch it came to a much more acceptable 23kb leaving plenty of room to add back in the debug serial prints.

Here is the results of my efforts… A fully working DHT22 and LMIC combo which periodically transmits temperature and humidity readings over LoRa, remember before using this code to set your network key, application key and device id prior to uploading to your board.

Download DHT22 & LMIC Arduino Sketch

Now to see it in action! With everything connected up I placed the sensor in range of a TTN LoRa Gateway and powered it up. Now to check TheThingsNetwork and see what kind of data is coming from the device…

Ok, so you are probably thinking wtf is that payload. As you know with LoRa you need to keep the payload as small as possible to reduce airtime and hence reduce duty cycle (a legal restriction in the EU), I decided upon having only one decimal place within my measurements so what you see here is the hex of the temperature and humidty float appended to each other, 31 34 2E 30 is the temperature and 35 34 2E 37 is the humidity.

Lets convert these into something human readable using Python.

Python 3.5.2 (v3.5.2:4def2a2901a5, Jun 26 2016, 10:47:25) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import binascii
>>> binascii.unhexlify('31342e30')
>>> binascii.unhexlify('35342e37')

OK looks like the temperature is 14°C and relative humidity is 54.7%, sounds about right considering the device in next to an open window in the shade and given the current weather today.

Next steps will involve moving the electronics on to a more permanent board (probably veroboard / stripboard), mounting the device in a small enclosure and subscribing to the TTN’s MQTT and publishing the results to a fancy dashboard.

Internet of Things: LoRa Hello World

Another LoRa transceiver module finally arrived from China today, now I have two of the things I decided to crack on with building a LoRa push button using an Arduino Uno and the LoRa module. Eventually this will be developed into a small circuit on strip board consisting of the LoRa module, an ATMega328-PU and a DHT22 to make my first LoRa powered temperature and humidity sensor to replace the ESP8266 based sensors I have today.

First steps as per all the other LoRa modules was to solder on some cables with appropriate headers at the other end, for this purpose I used a ribbon of male to male dupont style cables and chopped the connectors off one end, tinned the wires and soldered them to the VCC, GND, MISO, MOSI, NSS, SCK, RESET, DIO0 and DIO1 pads of the module. This module is a unbranded SX1276 module I found on eBay which is more cost effective than the RFM95W which was used for the single channel gateway, the downside is it has smaller pads than the RFM95W so soldering is a little more tricky, but with practice and a fine tip soldering iron you can acheive an acceptable result. Here is a chance for you to laugh at my soldering again…

Next I hooked the module up to an Arduino Uno as per the following pins… You shouldn’t really do this because the logic pins of the Uno are 5v and could damage the module, but I gathered for a quick test it would be ok, and if I fried it they are cheap enough. When mounting this permanently on a DHT22 sensor board I will use some voltage dividers to bring the 5v ATMega328-PU logic down to 3.3v for the module.

SX1276 Pins Arduino Uno Pins
VCC 3.3v
SCK 13
NSS 10
DIO0 2
DIO1 3

Next I plugged the Uno in to my laptop and launched the Arduino IDE. The intention is to send the packets to The Things Network via my single channel gateway, hence we need to find a LoRaWAN library. The most common appears to be the IBM LMIC Framework, go to Sketch -> Include Library -> Manage Libraries to download LMIC to your Arduino IDE.

Once the library is installed open the IBM LMIC Framework TTN example from File -> Examples, don’t upload it to your board yet, there are a few mods we need to make before we upload the sketch to the board. Open up your browser and visit, if you do not have an account already sign up. Once in the console go to the Applications tab and create a new application. Once you have made an application go to the devices tab of the application and register a new device, as over the air activation is not available for these sorts of modules via a single channel gateway just fill out the form with dummy data and click register.

Once you get to the device information page shown above click on Settings to begin the device personalisation. On the settings page change device activation method from OAA to ABP and click save, the page will refresh and the device’s address,  network and application keys will be generated for you. Copy these into your sketch in MSB format, you can get the keys in MSB format by clicking the ‘<>’ button next to their text box. You should end up with something that looks like this…

Scroll down the code a bit further until you get to the pin mapping comment. Modify the NSS pin to 10, RST pin to 9 and DIO pins to 2 as per below.

Next keep scrolling until you find the channel setup statements, comment out all the channels other than the channel served by your single channel gateway, in my case all channels except for 868.1mhz. If you are in range of a full 8 channel gateway you can skip this step.

Now you are ready to upload the sketch to the board, once it boots up it should transmit a packet saying “Hello, world!” once every 60 seconds, however the LMIC library does observe duty cycle limitations, so it may be delayed as appropriate. Before you hit the upload sketch button open TheThingsNetwork console, go to your application and select the device you created and open the data tab. Now upload your sketch, providing you are in range of a TheThingsNetwork LoRaWAN gateway you should see packets arriving in the console, if you do not see packets arriving as expected you can use the Arduino IDE serial monitor to look for debug messages from the Arduino, you’ll need to connect with a baudrate of 115200. Once you see the packets in your TheThingsNetwork console you know everything is working as expected, you can now expand the functionality to initiate the do_send function upon some other trigger rather than the timed callback set on line 105, e.g. a button press.

You can expand each packet to show meta data about the packet, for example which gateway forwarded it and the signal strength. The payload is presented as hex, but if you convert it back to ascii then the original contents of mydata in the Arduino sketch can be read.

echo "48656C6C6F2C20776F726C6421" | xxd -r -p
Hello, world!

Great all seems to be working just fine, next I will move on to combining this with periodic polling of data from a DHT22 and moving the circuit to veroboard with a ATMega328-PU standalone rather than having the bulk of the Uno. I may also do some range test to see how far away I can position a sensor, however as my area has a lot of buildings and not many hills I think the range will be quite small compared to some of the many kilometer range tests we see on YouTube for LoRa. Stay tuned for updates.

Internet of Things: Wohoo, First LoRa Packet Received!

Wohoo, whilst taking a look over the logs today, although I have no LoRa nodes myself, I noticed that my Single Channel LoRa Gateway had received it’s first packet from somebody and here it is…

stat update: {"stat":{"time":"2017-05-08 22:00:49 GMT","lati":0.00000,"long":0.00000,"alti":0,"rxnb":0,"rxok":0,"rxfw":0,"ackr":0.0,"dwnb":0,"txnb":0,"pfrm":"Single Channel Gateway","mail":"","desc":""}}
Packet RSSI: -113, RSSI: -112, SNR: -11, Length: 9
rxpk update: {"rxpk":[{"tmst":1621788918,"chan":0,"rfch":0,"freq":868.100000,"stat":1,"modu":"LORA","datr":"SF7BW125","codr":"4/5","lsnr":-11,"rssi":-113,"size":9,"data":"xoT77qK24c5x"}]}
stat update: {"stat":{"time":"2017-05-08 22:01:19 GMT","lati":0.00000,"long":0.00000,"alti":0,"rxnb":1,"rxok":1,"rxfw":0,"ackr":0.0,"dwnb":0,"txnb":0,"pfrm":"Single Channel Gateway","mail":"","desc":""}}

Proof it works atleast, of course we do not know what the data contains as it is encrypted with another The Things Network user’s application key but atleast somebody nearby is getting heard by my gateway and hopefully their packets are getting forwarded on to their application via The Things Network. Oh well, hopefully only another couple of weeks before my LoRa transceiver shipment from China arrives so we can start converting my current ESP8266 based sensors into LoRa equivalents, then we can see the end to end process.

If you want to check out how I built my Single Channel LoRa Gateway checkout Internet of Things: Single Channel LoRa Gateway Shopping List and Internet of Things: Single Channel LoRa Gateway Build.

Internet of Things: Single Channel LoRa Gateway Build

Today the final outstanding item on the Single Channel LoRa Gateway Shopping List arrived in the post, and I now have everything needed to get my Single Channel LoRa Gateway up and running and posting data back to The Things Network. If you’d like to replicate the project follow the guide below to get your gateway up and running.

First lets get the hardware side of things sorted out. Take 8 female to female jumpers and chop off the connectors at one end, and then tin the wires ready for soldering. The raw ends will be soldered to the LoRa transceiver and the female dupont style connectors will be plugged into the Raspberry Pi’s GPIO header.

Solder a cable to each of the following pins; DI00, 3.3v, MISO, MOSI, SCK, RESET, NSS and the GND pin next to the MISO pin. Do not solder the GND cable to any other GND point of the board, it must be the one next to the MISO pin. I found I had to switch out my soldering iron tip to a very fine tip as the pads on the transceiver module are pretty small.

Once you have all the cables for the GPIO header connected to the RFM95W LoRa module solder an 8cm length of solid core bell wire to the ANT pin on the module, and another 8cm length of solid core bell wire to the GND pin next to the ANT pin, these wires should face opposite directions at a 180 degree angle and will act as a small dipole antenna.

Next connect up the LoRa transceiver to the Raspberry Pi GPIO header as per the following. Pin 1 is marked on the board.

RFM95W Raspberry Pi
3.3V 1
DI00 7
NSS 22
SCK 23

I assembled my Raspberry Pi in a case with a gap for the cables to pop out from the GPIO connector. Here is the result, and also an opportunity for you to laugh at my bad soldering…

If you have not already loaded an OS onto your Raspberry Pi SD card you should do so now, I’ll be using Raspbian Jessie Lite although you are free to use any distribution that supports wiringpi. Once you have the OS on the sdcard connect up the Raspberry Pi to your network and boot it up. I’ll be configuring everything over SSH, if you want SSH to start by default on first boot place an empty file called ssh in the boot partition of the SD card, else on first boot SSH will remain in a stopped state.

After the Pi has booted up login and sudo up to the root user and then run the raspi-config utility.

rputt@loragw:~$ sudo su
root@loragw:~# raspi-config

From the menu select interfacing options and then select SPI, you need to ensure SPI is enabled prior to continuing configuration of the Pi.

If the configuration asks if you’d like to reboot say yes and reboot your Raspberry Pi. Next continue to install the wiringpi, gcc and git packages from your OS repository.

root@loragw:~# apt-get update
root@loragw:~# apt-get install -y wiringpi git gcc

Once the packages are installed we can continue to clone the single channel gateway packet forwarder from GitHub.

root@loragw:~# git clone

Now lets change directory into the repository we just cloned and check a few items in the source code…

root@loragw:~# cd single_chan_pkt_fwd
root@loragw:~/single_chan_pkt_fwd# vi main.cpp

In the file find the “#Define SERVER1” line and replace the IP address with, the IP in the repository is for an old The Things Network Router which has been retired, the IP provided in this blog is for, you can of course customise this value for a more appropriate host in your region. Next find the “uint32_t freq” line in the file and ensure it is set for the relevant frequency for your region, in the EU the frequency is 868100000 for 868.1mhz. Save the file and run the make command to build a binary from the code.

root@loragw:~/single_chan_pkt_fwd# make
g++ main.o base64.o -lwiringPi -o single_chan_pkt_fwd

Next run the single_chan_pkt_fwd binary and once it has done it’s initialisation press ctrl+c to stop its execution, if the hardware has been connected correctly and the software compiled successfully then you should see something like the following…

root@loragw:~/single_chan_pkt_fwd# ./single_chan_pkt_fwd 
SX1276 detected, starting.
Gateway ID: ba:12:e3:ff:ff:42:4e:1b
Listening at SF7 on 868.100000 Mhz.
stat update: {"stat":{"time":"2017-05-06 14:37:10 GMT","lati":0.00000,"long":0.00000,"alti":0,"rxnb":0,"rxok":0,"rxfw":0,"ackr":0.0,"dwnb":0,"txnb":0,"pfrm":"Single Channel Gateway","mail":"","desc":""}}

Open a web browser and visit, if you already have an account login, if you are new to The Things Network sign up for a new account, once you are logged in go to the console, click the gateways button and the click register gateway.

Complete the form as per the following:

  • Protocol – Packet Forwarder
  • Gateway EUI – Copy the Gateway ID given when you first run the single_chan_pkt_fwd binary.
  • Description – A friendly name for your gateway.
  • Frequency Plan – Select the frequency plan for your region, this should match the frequency given when you run single_chan_pkt_fwd, if it does not then you probably have the wrong LoRa module for your region, you should order one with the relevant frequency for your region, running a LoRa module in the wrong frequency band may be illegal in your country.

You should end up with something that looks a bit like this…

Next click register gateway. The page will refresh and the gateway status should say “not connected”, now rerun the single_chan_pkt_fwd binary on your Raspberry Pi, the gateway should now check into The Things Network and the webpage should update to show a connected status and say it was last seen within the last 30 seconds or so.

That’s it, your LoRa single channel packet forwarder gateway is now up and running and capable of sending stuff like sensor readings to The Things Network. Remember there are a few limitations to running a single channel gateway compared to a full 8 channel gateway…

  • This particular implementation is uni directional, it can only receive packets from LoRa devices, not transmit data back.
  • The gateway only operates on 1 channel and 1 spread factor, in this case 868.1mhz and SP7, therefore you should update any LoRa node that you want to submit data from to only use the 868.1mhz channel not the usual 8 channel round robin.
  • If you like you can edit the channel it listens on and the spread factor by editing the frequency and spread factor in the main.cpp file and rerunning the make command.

Obviously when you logout of your Pi SSH session the single_chan_pkt_fwd binary stops running, you can either run this under a screen session or configure SystemD to run the binary as a service to make sure it continues running after you logout.

Whats next? Well, its all well and good having a LoRa gateway setup in your house, but without any data flowing over it’s pretty boring, I am currently waiting on another LoRa transceiver module to turn up (I ordered this one from Ali Express as they are much cheaper direct from China via Ali Express rather than ordering from a UK seller on eBay, but the delivery times suck) once that arrives I’ll continue to convert my ESP8266 DHT22 project to a LoRa DHT22 project and have it submit it’s readings over my LoRa gateway. Unfortunately until this arrives I have no real way of proving if my gateway does in fact work or not. Keep watching the blog for updates.

Internet of Things: Single Channel LoRa Gateway Shopping List

Currently my Internet of Things Things are quite limited, in my house I have a DIY DHT22 powered Temperature & Humidity Sensor and Phillips Hue lighting. However soon providing everything goes to plan I should be moving out of rented accomodation into my own home which greatly improves the flexibility for home automation and cool IoT devices. One theme in common with most Internet of Things Things appears to be the use of radio to a hub of some sort rather than hundreds of ESP8266 modules all over the place, and this of course becomes an attractive proposition as low powered but low bandwidth radio can easily achieve better range compared to WiFi and you can have a much higher density of IoT devices multiplexed on some radio technology in comparison to connected WiFi devices on consumer grade WiFi access points.

After a bit of research I decided I should start experimentation with this concept and I stumbled accross LoRa, low power, wide area networking for IoT usage. LoRa is particularly attractive because of The Things Network (TTN), an open network of LoRa gateways which allows your thing to latch on to any TTN LoRa gateway and start transmitting or indeed recieving it’s data, it’s also attractive as LoRa connected devices can be made uber cheap with modules as low as around £10 on eBay from China. My first intention was to convert my DHT22 humidity sensor to a LoRa based device, so I researched if I could use a nearby TTN LoRa gateway for this purpose. Unfortunately the closest gateway is a good 5 miles away from my house and although LoRa has proven to have very good range when line of sight can be achieved and with carefully planned conditions I knew it was almost impossible to contact this gateway from my house purely because of the landscape and the fact in the UK most TTN LoRa gateways are limited range DIY devices. Further more there are no LoRa gateways in the area I plan on moving to, so to go down the LoRa route I’d need to host my own gateway long term anyway.

Back to the drawing board, before I could convert my DHT22 ESP8266 powered sensor to LoRa I’d need to establish my own LoRa gateway. Initially this seemed to be a huge blocker, although the LoRa modules for use in IoT devices are cheap, a decent LoRa gateway seemed to be in the region of £200 for a compliant 8 channel device. Then I found the saviour Andreas Spiess a YouTuber from Switzerland with a selection of LoRa based experiments on his channel. He documented how he found a blog detailing the creation of a cheap 1 channel LoRa gateway using a Raspberry Pi and a cheap LoRa transiever module, he also explains the limitations of a 1 channel gateway, for example on your connected things you need to force 1 channel operation and you will be able to multiplex less devices. However for the low number of devices I plan on creating I am sure this will provide a cheap entry point and with the heads up from Andreas reducing alot of headaches along the way.

So I ordered the parts to start building a single channel gateway. Here is the shopping list, I already had some of the parts laying around in my electronics box… Of course if you would like to replicate the project you’ll need some tools like wire strippers, a soldering iron and so on.

1x Raspberry Pi
1x Raspberry Pi Power Supply
1x MicroSD Card (8GB+)
1x RFM95W LoRa Transceiver Module
8x Female to Female Jumper Wires

These items are due to arrive over the next couple of days, then I will start to build my single channel LoRa gateway and I’ll document progress as I go. Once the gateway is online I’ll order a few other LoRa modules and start playing with sending and receiving packets on a breadboard to ensure it is working as expected.

My adventure into SSD caching with ZFS (Home NAS)

Recently I decided to throw away my old defunct 2009 MacBook Pro which was rotting in my cupboard and I decided to retrieve the only useful part before doing so, the 80GB Intel SSD I had installed a few years earlier. Initially I thought about simply adding it to my desktop as a bit of extra space but in 2017 80GB really wasn’t worth it and then I had a brainwave… Lets see if we can squeeze some additional performance out of my HP Microserver Gen8 NAS running ZFS by installing it as a cache disk.

I installed the SSD to the cdrom tray of the Microserver using a floppy disk power to SATA power converter and a SATA cable, unfortunately it seems the CD ROM SATA port on the motherboard is only a 3gbps port although this didn’t matter so much as it was an older 3gbps SSD anyway. Next I booted up the machine and to my suprise the disk was not found in my FreeBSD install, then I realised that the SATA port for the CD drive is actually provided by the RAID controller, so I rebooted into intelligent provisioning and added an additional RAID0 array with just the 1 disk to act as my cache, in fact all of the disks in this machine are individual RAID0 arrays so it looks like just a bunch of disks (JBOD) as ZFS offers additional functionality over normal RAID (mainly scrubbing, deduplication and compression).


Lets have a look at the zpool before adding the cache drive to make sure there are no errors or uglyness…

[root@netdisk] ~# zpool status vol0
  pool: vol0
 state: ONLINE
  scan: scrub repaired 0 in 3h21m with 0 errors on Wed Apr 26 16:40:16 2017

	NAME                                            STATE     READ WRITE CKSUM
	vol0                                            ONLINE       0     0     0
	  mirror-0                                      ONLINE       0     0     0
	    gptid/b8ebf047-25cd-11e7-b7e2-000c29ccceef  ONLINE       0     0     0
	    gptid/c0df3410-25cd-11e7-b7e2-000c29ccceef  ONLINE       0     0     0
	  mirror-1                                      ONLINE       0     0     0
	    gptid/c8b062cb-25cd-11e7-b7e2-000c29ccceef  ONLINE       0     0     0
	    gptid/355426bd-25ce-11e7-b7e2-000c29ccceef  ONLINE       0     0     0

errors: No known data errors

Now lets prep the drive for use in the zpool using gpart. I want to split the SSD into two seperate partitions, one for L2ARC (read caching) and one for ZIL (write caching). I have decided to split the disk into 20GB for ZIL and 50GB for L2ARC. Be warned using 1 SSD like this is considered unsafe because it is a single point of failure in terms of delayed writes (a redundant configuration with 2 SSDs would be more appropriate) and the heavy write cycles on the SSD from the ZIL is likely to kill it over time.

[root@netdisk] ~# gpart create -s gpt da6
[root@netdisk] ~# gpart show da6
=>       34  150994877  da6  GPT  (72G)
         34  150994877       - free -  (72G)
[root@netdisk] ~# gpart add -t freebsd-zfs -s 20G da6
da6p1 added
[root@netdisk] ~# gpart add -t freebsd-zfs -s 50G da6
da6p2 added
[root@netdisk] ~# gpart show da6
=>       34  150994877  da6  GPT  (72G)
         34   41943040    1  freebsd-zfs  (20G)
   41943074  104857600    2  freebsd-zfs  (50G)
  146800674    4194237       - free -  (2.0G)
[root@netdisk] ~# glabel list da6p1
Geom name: da6p1
1. Name: gptid/a3d641e5-2c03-11e7-9146-000c29ccceef
   Mediasize: 21474836480 (20G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 17408
   Mode: r0w0e0
   secoffset: 0
   offset: 0
   seclength: 41943040
   length: 21474836480
   index: 0
1. Name: da6p1
   Mediasize: 21474836480 (20G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 17408
   Mode: r0w0e0
[root@netdisk] ~# glabel list da6p2
Geom name: da6p2
1. Name: gptid/a6dd9f4d-2c03-11e7-9146-000c29ccceef
   Mediasize: 53687091200 (50G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 17408
   Mode: r0w0e0
   secoffset: 0
   offset: 0
   seclength: 104857600
   length: 53687091200
   index: 0
1. Name: da6p2
   Mediasize: 53687091200 (50G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 17408
   Mode: r0w0e0

Now we have a couple of partitions configured on the disk as described above, they can now be added to the ZFS zpool.

[root@netdisk] ~# zpool add vol0 log gptid/a3d641e5-2c03-11e7-9146-000c29ccceef
[root@netdisk] ~# zpool add vol0 cache gptid/a6dd9f4d-2c03-11e7-9146-000c29ccceef
[root@netdisk] ~# zpool status vol0
  pool: vol0
 state: ONLINE
  scan: scrub repaired 0 in 3h21m with 0 errors on Wed Apr 26 16:40:16 2017

	NAME                                            STATE     READ WRITE CKSUM
	vol0                                            ONLINE       0     0     0
	  mirror-0                                      ONLINE       0     0     0
	    gptid/b8ebf047-25cd-11e7-b7e2-000c29ccceef  ONLINE       0     0     0
	    gptid/c0df3410-25cd-11e7-b7e2-000c29ccceef  ONLINE       0     0     0
	  mirror-1                                      ONLINE       0     0     0
	    gptid/c8b062cb-25cd-11e7-b7e2-000c29ccceef  ONLINE       0     0     0
	    gptid/355426bd-25ce-11e7-b7e2-000c29ccceef  ONLINE       0     0     0
	  gptid/a3d641e5-2c03-11e7-9146-000c29ccceef    ONLINE       0     0     0
	  gptid/a6dd9f4d-2c03-11e7-9146-000c29ccceef    ONLINE       0     0     0

errors: No known data errors

Now it’s time to see if adding the cache has made much of a difference. I suspect not as my Home NAS sucks, it is a HP Microserver Gen8 with the crappy Celeron CPU and only 4GB RAM, anyway, lets test it and find out. First off lets throw fio at the mount point for this zpool and see what happens both with the ZIL and L2ARC enabled and disabled.

Caching Disabled FIO Results

[root@netdisk] ~# fio disk_perf.fio
random_rw: (g=0): rw=rw, bs=4K-4K/4K-4K/4K-4K, ioengine=posixaio, iodepth=1
Starting 1 thread
random_rw: Laying out IO file(s) (1 file(s) / 1024MB)
Jobs: 1 (f=1): [M(1)] [100.0% done] [10806KB/10580KB/0KB /s] [2701/2645/0 iops] [eta 00m:00s]
random_rw: (groupid=0, jobs=1): err= 0: pid=101627: Fri Apr 28 12:21:47 2017
  read : io=522756KB, bw=10741KB/s, iops=2685, runt= 48669msec
    slat (usec): min=2, max=2559, avg= 9.31, stdev=21.63
    clat (usec): min=2, max=69702, avg=34.66, stdev=563.31
     lat (usec): min=12, max=69760, avg=43.97, stdev=567.97
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    2], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[   14],
     | 70.00th=[   14], 80.00th=[   15], 90.00th=[   15], 95.00th=[   15],
     | 99.00th=[   27], 99.50th=[   59], 99.90th=[ 9536], 99.95th=[10304],
     | 99.99th=[15040]
  write: io=525820KB, bw=10804KB/s, iops=2701, runt= 48669msec
    slat (usec): min=2, max=8124, avg=15.87, stdev=43.54
    clat (usec): min=2, max=132195, avg=305.77, stdev=1973.41
     lat (usec): min=21, max=132485, avg=321.64, stdev=1977.32
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    4], 60.00th=[   24],
     | 70.00th=[   24], 80.00th=[   25], 90.00th=[   25], 95.00th=[   35],
     | 99.00th=[10304], 99.50th=[10560], 99.90th=[11712], 99.95th=[19584],
     | 99.99th=[50432]
    lat (usec) : 4=49.44%, 10=4.09%, 20=21.92%, 50=22.63%, 100=0.23%
    lat (usec) : 250=0.08%, 500=0.02%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.90%, 20=0.63%, 50=0.02%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=3.08%, sys=2.92%, ctx=265022, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=130689/w=131455/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: io=522756KB, aggrb=10741KB/s, minb=10741KB/s, maxb=10741KB/s, mint=48669msec, maxt=48669msec
  WRITE: io=525820KB, aggrb=10804KB/s, minb=10804KB/s, maxb=10804KB/s, mint=48669msec, maxt=48669msec
[root@netdisk] ~# 

Caching Enabled FIO Results

[root@netdisk] ~# fio disk_perf.fio
random_rw: (g=0): rw=rw, bs=4K-4K/4K-4K/4K-4K, ioengine=posixaio, iodepth=1
Starting 1 thread
random_rw: Laying out IO file(s) (1 file(s) / 1024MB)
Jobs: 1 (f=1): [M(1)] [100.0% done] [10902KB/10854KB/0KB /s] [2725/2713/0 iops] [eta 00m:00s]
random_rw: (groupid=0, jobs=1): err= 0: pid=101640: Fri Apr 28 12:24:18 2017
  read : io=522756KB, bw=10462KB/s, iops=2615, runt= 49967msec
    slat (usec): min=2, max=23575, avg=12.84, stdev=155.63
    clat (usec): min=2, max=135652, avg=34.24, stdev=644.41
     lat (usec): min=12, max=135654, avg=47.08, stdev=663.37
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    2], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    3],
     | 70.00th=[   13], 80.00th=[   14], 90.00th=[   15], 95.00th=[   15],
     | 99.00th=[   27], 99.50th=[   70], 99.90th=[ 9920], 99.95th=[10432],
     | 99.99th=[11968]
  write: io=525820KB, bw=10523KB/s, iops=2630, runt= 49967msec
    slat (usec): min=2, max=139302, avg=22.47, stdev=556.00
    clat (usec): min=2, max=119615, avg=306.30, stdev=1969.21
     lat (usec): min=20, max=139752, avg=328.78, stdev=2046.20
    clat percentiles (usec):
     |  1.00th=[    2],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    3],
     | 70.00th=[   23], 80.00th=[   24], 90.00th=[   25], 95.00th=[   32],
     | 99.00th=[10176], 99.50th=[10560], 99.90th=[19072], 99.95th=[24192],
     | 99.99th=[44800]
    lat (usec) : 4=62.92%, 10=2.93%, 20=15.75%, 50=16.37%, 100=0.28%
    lat (usec) : 250=0.14%, 500=0.03%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.93%, 20=0.57%, 50=0.04%
    lat (msec) : 100=0.01%, 250=0.01%
  cpu          : usr=3.21%, sys=2.51%, ctx=265920, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=130689/w=131455/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: io=522756KB, aggrb=10462KB/s, minb=10462KB/s, maxb=10462KB/s, mint=49967msec, maxt=49967msec
  WRITE: io=525820KB, aggrb=10523KB/s, minb=10523KB/s, maxb=10523KB/s, mint=49967msec, maxt=49967msec
[root@netdisk] ~# 


Ok, so the initial result is a little dissapointing, but hardly unexpected, my NAS sucks and there are lots of bottle necks, CPU, memory and the fact only 2 of the SATA ports are 6gbps. There is no real difference performance wise in comparison between the results, the IOPS, bandwidth and latency appear very similar. However lets bare in mind fio is a pretty hardcore disk benchmark utility, how about some real world use cases?

Next I decided to test a few typical file transactions that this NAS is used for, Samba shares to my workstation. For the first test I wanted to test reading a 3GB file over the network with both the cache enabled and disabled, I would run this multiple times to ensure the data is hot in the L2ARC and to ensure the test is somewhat repeatable, the network itself is an uncongested 1gbit link and I am copying onto the secondary SSD in my workstation. The dataset for these tests has compression and deduplication disabled.

Samba Read Test

Attempt    without caching    with caching
1          48.1MB/s           52.2MB/s
2          49.6MB/s           66.4MB/s
3          47.4MB/s           65.6MB/s

Not bad once the data becomes hot in the L2ARC cache reads appear to gain a decent advantage compared to reading from the disk directly. How does it perform when writing the same file back accross the network using the ZIL vs no ZIL.

Samba Write Test

Attempt    without caching    with caching
1          34.2MB/s           57.3MB/s
2          33.6MB/s           55.7MB/s
3          36.7MB/s           57.1MB/s

Another good result in the real world test, this certainately helps the write transfer speed however I do wonder what would happen if you filled the ZIL transferring a very large file, however this is unlikely with my use case as I typically only deal with a couple of files of several hundred megabytes at any given time so a 20GB ZIL should suit me reasonably well.

Is ZIL and L2ARC worth it?

I would imagine with a big beefy ZFS server running in a company somewhere with a large disk pool and lots of users with multiple enterprise level SSD ZIL and L2ARC would be well worth the investment, however at home I am not so sure. Yes I did see an increase in read speeds with cached data and a general increase in write speeds however it is use case dependant. In my use case I rarely access the same file frequently, my NAS primarily serves as a backup and for archived data, and although the write speeds are cool I am not sure its a deal breaker. If I built a new home NAS today I’d probably concentrate the budget on a better CPU, more RAM (for ARC cache) and more disks. However if I had a use case where I frequently accessed the same files and needed to do so in a faster fashion then yes, I’d probably invest in an SSD for caching. I think if you have a spare SSD lying around and you want something fun todo with it, sure chuck it in your ZFS based NAS as a cache mechanism. If you were planning on buying an SSD for caching then I’d really consider your needs and decide if the money can be spent on alternative stuff which would improve your experience with your NAS. I know my NAS would benefit more from an extra stick of RAM and a more powerful CPU, but as a quick evening project with some parts I had hanging around adding some SSD cache was worth a go.

Download YouTube Videos using VLC

I recently stumbled accross an amazing feature of VLC which I didn’t notice existed until I accidently pasted a YouTube webpage URL into the VLC open dialog… VLC is fully able to play YouTube videos and you can even save them using the transcoding feature, pretty cool.

Warning: I am sharing this with you as I thought it is a cool feature, however please do not use this method to download any copyright protected materials…


Play YouTube Videos in VLC

First off lets try playing a YouTube video in VLC. Find a YouTube video you’d like to play in VLC and copy it’s URL directly out of your web browser’s address bar. For this example I’ll use a video of some creative commons licensed music.

Next open VLC and select “Open Network…” from the file menu.

Paste the YouTube URL you copied from your browser window into the URL text box and click open, your YouTube video should now start to play within VLC.

Great, now you can watch your favourite YouTube videos within VLC, you can even add them to playlists and best of all it seems to skip the ads every time. But whatou if you want to save the videos offline to view later? Keep reading to learn how to save YouTube videos to your computer using VLC.


Downloading YouTube videos using VLC

Downloading YouTube videos using VLC is almost as easy as playing them, copy the YouTube URL from your browser and then open VLC, this time open the Streaming/Exporting Wizard from the file menu.

On the first page of the wizard select “Transcode/save to file” and click Next.

On the next page paste the YouTube URL into the “Select a stream” textbox and click Next.

Tick the Transcode Video and Transcode Audio checkboxes and select the codec of the file you’d like to save, personally I prefer mpeg4 although if you prefer a different format / bitrate you should select it here. If you don’t know what to select here use the stuff from my screenshot and you should have a working mpeg4 file out the other end. Remember to check transcode audio too else you will end up with a video file with no audio.

Next select a location to save the video file… Clicking the choose button will open a file save dialog so you can more easily select a location for the file. Once you are happy click Next.

A breif summary will be displayed with all the details you entered in the wizard, dismiss this by clicking the finish button, in your main VLC window you’ll now have a new Steaming/Transcode Wizard item in your playlist. Double click on this playlist item to start the transcoding of the YouTube video, when the transcoding starts the Title of the playlist item will update to the title of the YouTube video. No video will play but progress will be displayed at the bottom of the VLC window in the area highlighted in the screenshot… Once the progress indicated in the playback position bar is complete and the time returns to 00:00 you know transcoding the YouTube video to the file is complete.

You can now locate the video file you saved from YouTube in the location you selected in the wizard and you can open it in any video player capable of playing back the codec you selected. It feels a bit clunky but after you have downloaded a couple of videos the process gets alot easier.

Congratulations, you have now downloaded a video from YouTube using VLC!


As I previously suggested please do not download copyrighted materials from YouTube, if you want a video / song please purchase it through official means, however if you are after a creative commons video, or some old content you uploaded to YouTube but lost the source files please go ahead and use this method to pull down video from YouTube.

New Nokia 3310, genius or ludicrous?

A few days ago my other better half was most excited to tell me the joyful news that some random company somewhere was “bringing back the Nokia 3310” and initially I thought, well thats kind of cool as a nastolgic kind of idea, but then after reading the hype online I soon turned to think its an awful idea and a cash cow. The damn thing is going to retail for around £35… £35 for a feature phone are you having a laugh?

Anyway since then I have heard alot of stuff about the new release specially as my generation where around in the highlight of the 3310, here is what I’ve been hearing along with my thoughts…

  1. It looks like a classic Nokia 3310, how cool!
    • Sure this is kind of nice I guess, but you’ll soon get annoyed with the lack of screen size and clunky buttons I am sure.
  2. The Nokia 3310 was the best phone I ever owned!
    • Hang on a minute, you’d probably say that about your first car too, just because of your fond memories of loosing your virginity on the back seat at the age of 17 and having a “burn up with your mates” in the local industrial estate after dark, the reality is living with that car today would be really shit, even if you retrofitted a CD player.
  3. But the battery will last for ever…
    • Yep the battery life is nice, specially if you want a 2G phone for your latest hiking adventure which will last more than 7 hours to airlift you to safety when you freeze at 2 in the morning, or if you want a burner phone to smuggle into prison and you didn’t fancy wedging that horrible 3 pin charger up your butt, but lets look at the reality, is super long battery life really worth loosing all the features you love today and £35? If you want a phone which will still send an SMS and make a phone call after 2 and a half days of being powered on without a charge check out other feature phones which are out there, they are just as bad as the 3310 remake without the nastolgic 3310 case, but only £5 (see here if your interested).
  4. They’ve pimped it with a colour screen!
    • Damn, so orginally I was sold on the nostolgia, but now it’s just a standard feature phone in a 3310 case, if you really wanna be part of the cool kids buy an original Nokia 3310 off of eBay and get a new battery for it, then you really get my respect.
  5. It would be good to escape my smart phone and have a simpler phone.
    • Sure, I understand your motivation here, smart phones are a big drain on your time and they interupt everything with their notifications, apps and short battery life. But there are much cheaper feature phones out there so you may as well save £30 in the process and buy something else to entertain you with the lack of apps / internet / anything useful other than calling your mum on your new handset.
  6. SNAKE
    • Damn, you aren’t gonna enjoy this new snake, its on a colour screen, in the pre-release videos it seems to behave differently to the original snake, and don’t you remember how fustrating it was when you mis-pushed one of those crappy rubbery buttons and your snake ran into the side of the screen after you’d carefully curated the spiral of doom for the last 2 minutes of your life? Trust me, everything is capable of running a snake like game these days, you really don’t need a 3310 remake with a snake remake on it to enjoy a worm based, dot eating time filler.


My verdict…

If you want it because it looks like a 3310, fine, for any other reason do not buy this phone.


What do I recommend instead?

If you live within reasonably easy access to a power socket (and lets face it most of us do) then for the same money as the 3310 remake you can buy a really low end Android phone which will do most stuff you want day to day like calling, SMS, normally a pretty crappy camera and it’ll run some of the not so fancy Android apps so atleast you’ll be able to check up on the awful lives of your friends on Facebook or Tweet photos of your dog licking icecream off of it’s nose. – Yep they really are available for the same price as the 3310 remake: Cheap Android Phone

If you need a phone that will last for a few days and be reliable for making calls and SMS then checkout cheaper feature phones which as proven above start from just £5, with the £30 saving you can go and enjoy yourself. – Cheap Feature Phone

If you want a phone with good battery life and smart phone features… Sorry your out of luck, get a time machine and bring some future batteries back with you please, we’d really like to know how they work! – Time Machine

And lastly, when should you buy a 3310 remake? Well if you want something that looks like a 3310 to take on your trip to some beach in Thailand to “find yourself” and you are living on the generous donations of mummy and daddy to waste away your gap year at uni then sure… Blow yourself away and get the 3310 remake if that’s your kind of thing. – Nokia 3310 Remake, of course if you are the real mccoy you’d look around Amazon used section, eBay or even your own drawer of crap at home to find an original Nokia 3310 in all it’s early 2000s glory.