gdp-if / mqtt-gateway / README.md @ master
URBAN HEARTBEAT KIT MQTT-GDP GATEWAY
This directory contains applications and scripts to support the Urban Heartbeat Kit project from University of Michigan and TerraSwarm. This kit is based on Bluetooth Low Energy (BLE) devices such as BLEE, PowerBlade, and Blink.
There are three distinct parts to a UHK install:
The Michigan code to interface with the BLE radio and gateway the data into MQTT, including some processing done by reading the MQTT raw data feed and producing other MQTT feeds. This code is not included in this directory, but a script (
setup-uhk-gateway.sh) will install the prerequisites, download the Michigan code from Github, and install it. In most cases this will run on a BeagleBone Black (BBB), but it has been shown to work on a SwarmBox. This script might need to be tweaked on other platforms.
mqtt-gdp-gatewayprogram to gateway the MQTT data into the GDP. This code is intentionally quite generic, taking one or more MQTT topic patterns and logging them into corresponding GDP logs. This is normally not run on the BBB.
Various installation and startup scripts for
mqtt-gdp-gateway. These need to be configured with the names of the MQTT devices and the names of the devices of interest.
Note that there is no necessity that the gateway code run on the same system that is collecting the data; in fact, it is probably better to limit the BBB to doing data collection and not data storage or GDP gatewaying.
These instructions assume you are starting from source code
rather than packages. To get the source tree from git into
a directory I'll call
cd $src git clone https://repo.eecs.berkeley.edu/git-anon/projects/swarmlab/gdp.git git clone https://repo.eecs.berkeley.edu/git-anon/projects/swarmlab/gdp-if.git
If you don't install these next to each other you may need to tweak some of the scripts.
The Michigan UHK Code
If you got your BBB through Terraswarm labelled as an "Urban Heartbeat Kit" then the Michigan code should be preinstalled for you, and you should skip this section.
There are instructions to install the Michigan code at
https://github.com/terraswarm/urban-heartbeat-kit. They tend
to be somewhat low level, and if you are doing anything that
needs special tweaking you will probably need to follow those
instructions, and you should skip the rest of this section.
This should only be done on the BBB that has the actual BLE radio that will communicate with the devices.
Do the installation using the script in this directory:
cd $src/gdp-if/mqtt-gateway sh setup-uhk-gateway.sh
If everything is reasonably vanilla in your environment, the
install-mqtt-gdp-gateway.shscript should do everything you need, including creating a
gdpuser if necessary, downloading required packages, creating necessary system directories, and so forth. You may have problems with mosquitto (see below).
This script can take a while when it is setting up Node. If you have already done this you can use the
-nflag to skip that step.
Reboot. The system should start several programs, including:
* adv-gateway-ip.js * ble-gateway-mqtt.js * ble-address-sniffer-mqtt.js To check this, use ~~~ ps ax | egrep 'gateway|ble' ~~~
Test. The easiest way to do this is to see if the MQTT broker is producing output:
mosquitto_sub -v -t \#
You may also need a flag
-h hostnameif you are testing from a different system
mosquittopackage is an implementation of MQTT. Unfortunately it isn't well supported by many of the package managers, so there is a small chance you'll need to compile and install it by hand (see https://mosquitto.org/). In particular, on Debian and Ubuntu you'll need to install
mosquitto-clients. On FreeBSD the package is named
mosquitto. On MacOS mosquitto is supported by brew, but not macports. However it is easy to download and compile the library from source.
If you aren't seeing any output, check to make sure the Bluetooth device is enabled using
hcitool dev. This should give you output such as:
Devices: hci0 A4:C4:94:DB:7E:6D
The MQTT-GDP Gateway
This gateway code is generally not run on a BBB; instead, it runs on a larger host, reading the MQTT feed remotely and then logging the results. If you are running the UHK code on a SwarmBox these can be co-resident.
To install the gateway code you will first need to install the GDP libraries. If the GDP code is already installed, skip to step 3.
Install prerequisite packages for the GDP:
cd $src/gdp sh adm/gdp-setup.sh
Compile and install the GDP code (libraries and programs):
make clean all sudo make install
If you are going to be running the GDP server code on your machine (not recommended for a BBB), do the appropriate system setup:
Otherwise you may need to set up the configuration to point to another GDP server. If the file
etc/gdp/params/gdpalready exists, skip to step 4. Otherwise:
# determine default router set routers=`echo gdp-01 gdp-02 gdp-03 gdp-04 | tr ' ' '\n' | shuf | tr '\n' ';' | sed -e 's/;/.eecs.berkeley.edu; /g' -e 's/; $//' ` # create the parameters file echo "swarm.gdp.routers=$routers" | \ sudo cp - /etc/gdp/params/gdp sudo chown gdp:gdp /etc/gdp/params/gdp
The first clause randomizes the set of servers at Berkeley that you can use; this is intended to distribute the load. The important point is to create
/etc/gdp/params/gdpwith an entry for
swarm.gdp.routers. The final file should look something like:
99 are replaced by randomized values.
You should now be able to compile and install the MQTT-GDP gateway code:
cd $src/gdp-if/mqtt-gateway sh install-mqtt-gdp-gateway.sh
This step compiles and installs the code, but does not configure the individual brokers and sensors. For that, see the next section.
Startup and Installation Scripts
This is probably the hardest part. To proceed, you'll need:
The DNS name(s) for the MQTT brokers (i.e., sensor gateways, usually running on a BBB). For convenience, I'll refer to this name as
$fqdnbelow, with just the host name part referred to as
$host. For example, for a given sensor
The MAC addresses for the individual sensors connected to those machines. This is generally a string of hexadecimal digits printed on the sensor itself, often with each pair of digits separated with a colon. For example, it might be
The "root name" for the logs in which to store the data from the sensors. By convention this is your reversed domain name (possibly with additional location information) --- for example,
edu.berkeley.eecs.swarmlab. I'll refer to this as
The installation script in the previous section should have created
/etc/gdp/keys owned by
If these directories do not exist, please create them:
sudo mkdir /etc/gdp /etc/gdp/keys sudo chown gdp:gdp /etc/gdp /etc/gdp/keys
You can then proceed to the Operations section.
This section applies both to initial installation and long term maintenance.
Most operations are driven using one configuration file per
BBB-based MQTT broker host. Those files live in
the name of the machine without the domain. For example, if your
broker is running on
$host would be
uhkbbb000. We'll call this
$conf in the examples below.
The first line of this configuration file must be the root name of the GDP log. The remaining lines have the internal names of the sensors connected to that host.
Comments can be included in the file starting with
#, and blank
lines are ignored.
For the purposes of explanation, you can assume the following definitions:
fqdn=uhkbbb000.eecs.berkeley.edu host=uhkbbb000 conf=/etc/gdp/mqtt-gateway/uhkbbb000.conf logroot=edu.berkeley.eecs.swarmlab
Adding a New BBB-Based MQTT Broker
For each MQTT broker,
Create a configuration file. The first non-comment line needs to be the root name for the GDP log. It should be owned by
sudo -s "echo $logroot > $conf" sudo chown gdp:gdp $conf
For example, afer this step $conf should look something like:
To add individual sensors, use the
sh gdp-provision-sensor.sh $host sensors ...
sensorsis replaced by the list of sensor MAC addresses. This will update the configuration for the indicated $host, create the log, and do other housekeeping. For example, after running the commands:
sudo sh gdp-provision-sensor.sh uhkbbb000 c098e5300061 c098e590009b sudo sh gdp-provision-sensor.sh uhkbbb000 c098e5300078
The file should look like this:
edu.berkeley.eecs.swarmlab c098e5300061 c098e590009b c098e5300078
and the following logs should have been created:
edu.berkeley.eecs.swarmlab.device.c098e5300061 edu.berkeley.eecs.swarmlab.device.c098e590009b edu.berkeley.eecs.swarmlab.device.c098e5300078
.deviceis included in the name. This cannot be changed.
Tell the system startup scripts to process this configuration file. If you are using
sudo systemctl enable mqtt-gdp-gateway@$fqdn
Note that you must use the fully qualified domain name of the MQTT broker host. If you are not using
systemd, you'll have to edit the file
/usr/sbin/start-mqtt-gdp-gateways.shto add a
$STARTline for each MQTT broker (this should be rare).
Once a gateway service is enabled, it will persist over a reboot. To remove a gateway, use
sudo systemctl disable mqtt-gdp-gateway@$fqdn
At this point you can start the MQTT-GDP gateway processes either by rebooting or using:
sudo systemctl start mqtt-gdp-gateway
Test to make sure data is getting into the logs using
gdp-reader. For example,
gdp-reader -t -s edu.berkeley.eecs.swarmlab.device.c098e5300061
should start printing out records approximately once per second for most devices.
Adding a new sensor to an existing BBB gateway
This is just a matter of adding a new line to the corresponding
configuration file --- essentially, by performing step 2 of the previous
section again. Use the
gdp-provision-sensor.sh script to add a new
sensor to an existing broker. The command:
sh gdp-provision-sensor.sh $host sensor ...
takes the host name of an existing MQTT broker and adds the
indicated sensor names.
$host identifies the configuration
file to be modified and the sensor names are the MAC addresses of
the new sensors. Appropriate logs are created as needed. The
script should be run as root (it will change to
Once you have added a new sensor, you need to restart the
mqtt-gdp-gateway instance for that one broker. You can use
sudo systemctl restart mqtt-gdp-gateway@$fqdn
What's in This Directory?
The mqtt-gdp-gateway application that reads one or more MQTT topics (specified by a pattern) and copies all messages it receives to a GDP log.
For the moment mqtt-gdp-gateway assumes that all MQTT messages are formatted as JSON, and (assuming this is true) writes records that are formatted as JSON. The output format is "wrapped" with a JSON object to include metadata.
MQTT topics are specified using a standard topic pattern. Quoting from the mosquitto mqtt(7) man page:
Topics are treated as a hierarchy, using a slash (/) as a separator. This allows sensible arrangement of common themes to be created, much in the same way as a filesystem. For example, multiple computers may all publish their hard drive temperature information on the following topic, with their own computer and hard drive name being replaced as appropriate:sensors/COMPUTER_NAME/temperature/HARDDRIVE_NAME
Clients can receive messages by creating subscriptions. A subscription may be to an explicit topic, in which case only messages to that topic will be received, or it may include wildcards. Two wildcards are available,
+can be used as a wildcard for a single level of hierarchy. It could be used with the topic above to get information on all computers and hard drives as follows:sensors/+/temperature/+
As another example, for a topic of "a/b/c/d", the following example subscriptions will match:
The following subscriptions will not match:
#can be used as a wildcard for all remaining levels of hierarchy. This means that it must be the final character in a subscription. With a topic of "a/b/c/d", the following example subscriptions will match:
Zero length topic levels are valid, which can lead to some slightly non-obvious behaviour. For example, a topic of "a//topic" would correctly match against a subscription of "a/+/topic". Likewise, zero length topic levels can exist at both the beginning and the end of a topic string, so "/a/topic" would match against a subscription of "+/a/topic", "#" or "/#", and a topic "a/topic/" would match against a subscription of "a/topic/+" or "a/topic/#".
See the mqtt-gdp-gateway(1) man page for more details.
Fetch the U. Michigan from the
lab11 repository, install
prerequisite packages, compile everything necessary, and install
the system startup scripts necessary to run that code on reboot.
Note that the U. Michigan packages must run on a properly
configured BeagleBone Black from the
This only installs the U. Michigan code.
This script creates the
gdp user and directories necessary to
run the MQTT-GDP gateway. It then compiles the
code and installs both it and the system startup scripts as
described below. Note that the GDP code must be installed
before this script is run.
This may run on a BeagleBone, but it isn't necessary. A typical
scenario will be to run the
mqtt-gdp-gateway on an Ubuntu machine
which will pull from the BeagleBone.
The Systemd configuration file templates. The former starts a single
mqtt-gdp-gateway, i.e., one connected to a single MQTT
broker. The latter will startup all mqtt-gdp-gateway instances
running on multiple brokers, as defined by
Deprecated --- used only by Upstart.
The Upstart configuration file templates. The former starts a single
mqtt-gdp-gateway, i.e., one connected to a single MQTT
broker. The latter just invokes
(below) to start up multiple brokers.
Deprecated --- used only by Upstart. A simple script to start multiple GDP gateway instances. It must be modified when initially installed or to to add new MQTT brokers.
A helper script used to start up the MQTT-GDP gateway program. It
is normally invoked by Systemd or Upstart, but it can also be invoked
by hand. It takes the fully qualified domain name of the machine
running the MQTT broker as its only argument. The domain part of
that name is stripped off and the remaining host name is used to
find a configuration in
/etc/gdp.) The first line of that file is the base
name for the GDP log names that will be created; for example,
it might be
edu.berkeley.eecs.swarmlab, which will result in
remaining lines of that file are the names of devices of interest.
Those names will be used both for building the MQTT topic name
and the GDP log name.
Logs are not created if they cannot be found to avoid having logs
created on more than one server if a router happens to be down
when this script is run. The
will create the logs for you when you are doing the initial
For example, if the file
edu.berkeley.eecs.swarmlab dev23 dev37
then you will have to create the logs
start-mqtt-gdp-gateway.sh foo.example.org will start
reading the MQTT broker at
foo.example.org for the topics
Those topics will be copied into the corresponding logs.
A Systemd configuration file to start up the MQTT broker. Generally used on the BBB.
A configuration to do daily log rotation using logrotate(8).
The source code and documentation for the actual MQTT to GDP gateway program.
A helper script used by several of the other shell scripts.
- The start-mqtt-gdp-gateway.sh script only discovers configuration changes on startup, not on the fly. To add or delete devices the script must be restarted.
- The Upstart scripts assume that they are running on the same
gdplogd; in particular, they start and stop based on an Upstart job named
gdplogdrunning on the same machine. Although this is the common case, there is no reason these scripts cannot run on a GDP client-only system.
- Note: The Debian systems running on the BeagleBone use systemd as their startup system; hence, the U. Michigan code uses systemd. Similarly, Ubuntu 16.04 (our preferred platform) uses systemd. However, Ubuntu 14.04 uses Upstart; hence the support for that startup mechanism. Upstart support is deprecated and will probably be removed in the future.