Project

General

Profile

Statistics
| Branch: | Revision:

gdp-if / mqtt-gateway / README.md @ master

History | View | Annotate | Download (18.6 KB)

URBAN HEARTBEAT KIT MQTT-GDP GATEWAY

This directory contains applications and scripts to support the Urban Heartbeat Kit project from University of Michigan and TerraSwarm. This kit is based on Bluetooth Low Energy (BLE) devices such as BLEE, PowerBlade, and Blink.

There are three distinct parts to a UHK install:

  1. The Michigan code to interface with the BLE radio and gateway the data into MQTT, including some processing done by reading the MQTT raw data feed and producing other MQTT feeds. This code is not included in this directory, but a script (setup-uhk-gateway.sh) will install the prerequisites, download the Michigan code from Github, and install it. In most cases this will run on a BeagleBone Black (BBB), but it has been shown to work on a SwarmBox. This script might need to be tweaked on other platforms.

  2. The mqtt-gdp-gateway program to gateway the MQTT data into the GDP. This code is intentionally quite generic, taking one or more MQTT topic patterns and logging them into corresponding GDP logs. This is normally not run on the BBB.

  3. Various installation and startup scripts for mqtt-gdp-gateway. These need to be configured with the names of the MQTT devices and the names of the devices of interest.

Note that there is no necessity that the gateway code run on the same system that is collecting the data; in fact, it is probably better to limit the BBB to doing data collection and not data storage or GDP gatewaying.

Initial Installation

These instructions assume you are starting from source code rather than packages. To get the source tree from git into a directory I'll call $src:

    cd $src
    git clone https://repo.eecs.berkeley.edu/git-anon/projects/swarmlab/gdp.git
    git clone https://repo.eecs.berkeley.edu/git-anon/projects/swarmlab/gdp-if.git

If you don't install these next to each other you may need to tweak some of the scripts.

The Michigan UHK Code

If you got your BBB through Terraswarm labelled as an "Urban Heartbeat Kit" then the Michigan code should be preinstalled for you, and you should skip this section.

There are instructions to install the Michigan code at https://github.com/terraswarm/urban-heartbeat-kit. They tend to be somewhat low level, and if you are doing anything that needs special tweaking you will probably need to follow those instructions, and you should skip the rest of this section.

This should only be done on the BBB that has the actual BLE radio that will communicate with the devices.

  1. Do the installation using the script in this directory:

        cd $src/gdp-if/mqtt-gateway
        sh setup-uhk-gateway.sh
    

    If everything is reasonably vanilla in your environment, the install-mqtt-gdp-gateway.sh script should do everything you need, including creating a gdp user if necessary, downloading required packages, creating necessary system directories, and so forth. You may have problems with mosquitto (see below).

    This script can take a while when it is setting up Node. If you have already done this you can use the -n flag to skip that step.

  2. Reboot. The system should start several programs, including:

* adv-gateway-ip.js
* ble-gateway-mqtt.js
* ble-address-sniffer-mqtt.js

To check this, use

~~~
    ps ax | egrep 'gateway|ble'
~~~
  1. Test. The easiest way to do this is to see if the MQTT broker is producing output:

        mosquitto_sub -v -t \#
    

    You may also need a flag -h hostname if you are testing from a different system

    The mosquitto package is an implementation of MQTT. Unfortunately it isn't well supported by many of the package managers, so there is a small chance you'll need to compile and install it by hand (see https://mosquitto.org/). In particular, on Debian and Ubuntu you'll need to install libmosquitto-dev and mosquitto-clients. On FreeBSD the package is named mosquitto. On MacOS mosquitto is supported by brew, but not macports. However it is easy to download and compile the library from source.

    If you aren't seeing any output, check to make sure the Bluetooth device is enabled using hcitool dev. This should give you output such as:

        Devices:
            hci0    A4:C4:94:DB:7E:6D
    

The MQTT-GDP Gateway

This gateway code is generally not run on a BBB; instead, it runs on a larger host, reading the MQTT feed remotely and then logging the results. If you are running the UHK code on a SwarmBox these can be co-resident.

To install the gateway code you will first need to install the GDP libraries. If the GDP code is already installed, skip to step 3.

  1. Install prerequisite packages for the GDP:

        cd $src/gdp
        sh adm/gdp-setup.sh
    
  2. Compile and install the GDP code (libraries and programs):

        make clean all
        sudo make install
    
  3. If you are going to be running the GDP server code on your machine (not recommended for a BBB), do the appropriate system setup:

        sh adm/init-gdp-server.sh
    

    Otherwise you may need to set up the configuration to point to another GDP server. If the file etc/gdp/params/gdp already exists, skip to step 4. Otherwise:

        # determine default router set
        routers=`echo gdp-01 gdp-02 gdp-03 gdp-04 |
            tr ' ' '\n' |
            shuf |
            tr '\n' ';' |
            sed -e 's/;/.eecs.berkeley.edu; /g' -e 's/; $//' `
    
        # create the parameters file
        echo "swarm.gdp.routers=$routers" | \
            sudo cp - /etc/gdp/params/gdp
        sudo chown gdp:gdp /etc/gdp/params/gdp
    

    The first clause randomizes the set of servers at Berkeley that you can use; this is intended to distribute the load. The important point is to create /etc/gdp/params/gdp with an entry for swarm.gdp.routers. The final file should look something like:

        swarm.gdp.routers=gdp-88.eecs.berkeley.edu; gdp-99.eecs.berkeley.edu
    

where 88 and 99 are replaced by randomized values.

  1. You should now be able to compile and install the MQTT-GDP gateway code:

        cd $src/gdp-if/mqtt-gateway
        sh install-mqtt-gdp-gateway.sh
    

    This step compiles and installs the code, but does not configure the individual brokers and sensors. For that, see the next section.

Startup and Installation Scripts

This is probably the hardest part. To proceed, you'll need:

  • The DNS name(s) for the MQTT brokers (i.e., sensor gateways, usually running on a BBB). For convenience, I'll refer to this name as $fqdn below, with just the host name part referred to as $host. For example, for a given sensor $fqdn might equal mqtt23.example.com and $host would equal mqtt23.

  • The MAC addresses for the individual sensors connected to those machines. This is generally a string of hexadecimal digits printed on the sensor itself, often with each pair of digits separated with a colon. For example, it might be c0:98:e5:30:00:50.

  • The "root name" for the logs in which to store the data from the sensors. By convention this is your reversed domain name (possibly with additional location information) --- for example, edu.berkeley.eecs.swarmlab. I'll refer to this as $logroot below.

The installation script in the previous section should have created directories named /etc/gdp and /etc/gdp/keys owned by gdp:gdp. If these directories do not exist, please create them:

    sudo mkdir /etc/gdp /etc/gdp/keys
    sudo chown gdp:gdp /etc/gdp /etc/gdp/keys

You can then proceed to the Operations section.

Operations

This section applies both to initial installation and long term maintenance.

Most operations are driven using one configuration file per BBB-based MQTT broker host. Those files live in /etc/gdp, named /etc/gdp/mqtt-gateway/$host.conf, where $host is the name of the machine without the domain. For example, if your broker is running on uhkbbb000.eecs.berkeley.edu (i.e., $fqdn is unkbbb000.eecs.berkeley.edu), then $host would be uhkbbb000. We'll call this $conf in the examples below.

The first line of this configuration file must be the root name of the GDP log. The remaining lines have the internal names of the sensors connected to that host.

Comments can be included in the file starting with #, and blank lines are ignored.

For the purposes of explanation, you can assume the following definitions:

    fqdn=uhkbbb000.eecs.berkeley.edu
    host=uhkbbb000
    conf=/etc/gdp/mqtt-gateway/uhkbbb000.conf
    logroot=edu.berkeley.eecs.swarmlab

Adding a New BBB-Based MQTT Broker

For each MQTT broker,

  1. Create a configuration file. The first non-comment line needs to be the root name for the GDP log. It should be owned by gdp:gdp.

        sudo -s "echo $logroot > $conf"
        sudo chown gdp:gdp $conf
    

    For example, afer this step $conf should look something like:

        edu.berkeley.eecs.swarmlab
    
  2. To add individual sensors, use the gdp-provision-sensor.sh script:

        sh gdp-provision-sensor.sh $host sensors ...
    

    where sensors is replaced by the list of sensor MAC addresses. This will update the configuration for the indicated $host, create the log, and do other housekeeping. For example, after running the commands:

        sudo sh gdp-provision-sensor.sh uhkbbb000 c098e5300061 c098e590009b
        sudo sh gdp-provision-sensor.sh uhkbbb000 c098e5300078
    

    The file should look like this:

        edu.berkeley.eecs.swarmlab
        c098e5300061
        c098e590009b
        c098e5300078
    

    and the following logs should have been created:

        edu.berkeley.eecs.swarmlab.device.c098e5300061
        edu.berkeley.eecs.swarmlab.device.c098e590009b
        edu.berkeley.eecs.swarmlab.device.c098e5300078
    

    Note that .device is included in the name. This cannot be changed.

  3. Tell the system startup scripts to process this configuration file. If you are using systemd,

        sudo systemctl enable mqtt-gdp-gateway@$fqdn
    

    Note that you must use the fully qualified domain name of the MQTT broker host. If you are not using systemd, you'll have to edit the file /usr/sbin/start-mqtt-gdp-gateways.sh to add a $START line for each MQTT broker (this should be rare).

    Once a gateway service is enabled, it will persist over a reboot. To remove a gateway, use

        sudo systemctl disable mqtt-gdp-gateway@$fqdn
    
  4. At this point you can start the MQTT-GDP gateway processes either by rebooting or using:

        sudo systemctl start mqtt-gdp-gateway
    
  5. Test to make sure data is getting into the logs using gdp-reader. For example,

        gdp-reader -t -s edu.berkeley.eecs.swarmlab.device.c098e5300061
    

    should start printing out records approximately once per second for most devices.

Adding a new sensor to an existing BBB gateway

This is just a matter of adding a new line to the corresponding configuration file --- essentially, by performing step 2 of the previous section again. Use the gdp-provision-sensor.sh script to add a new sensor to an existing broker. The command:

    sh gdp-provision-sensor.sh $host sensor ...

takes the host name of an existing MQTT broker and adds the indicated sensor names. $host identifies the configuration file to be modified and the sensor names are the MAC addresses of the new sensors. Appropriate logs are created as needed. The script should be run as root (it will change to gdp).

Once you have added a new sensor, you need to restart the mqtt-gdp-gateway instance for that one broker. You can use that using:

    sudo systemctl restart mqtt-gdp-gateway@$fqdn

What's in This Directory?

mqtt-gdp-gateway.c

The mqtt-gdp-gateway application that reads one or more MQTT topics (specified by a pattern) and copies all messages it receives to a GDP log.

For the moment mqtt-gdp-gateway assumes that all MQTT messages are formatted as JSON, and (assuming this is true) writes records that are formatted as JSON. The output format is "wrapped" with a JSON object to include metadata.

MQTT topics are specified using a standard topic pattern. Quoting from the mosquitto mqtt(7) man page:


Topics are treated as a hierarchy, using a slash (/) as a separator. This allows sensible arrangement of common themes to be created, much in the same way as a filesystem. For example, multiple computers may all publish their hard drive temperature information on the following topic, with their own computer and hard drive name being replaced as appropriate:

    sensors/COMPUTER_NAME/temperature/HARDDRIVE_NAME

Clients can receive messages by creating subscriptions. A subscription may be to an explicit topic, in which case only messages to that topic will be received, or it may include wildcards. Two wildcards are available, + or #.

+ can be used as a wildcard for a single level of hierarchy. It could be used with the topic above to get information on all computers and hard drives as follows:

    sensors/+/temperature/+

As another example, for a topic of "a/b/c/d", the following example subscriptions will match:

  • a/b/c/d
  • +/b/c/d
  • a/+/c/d
  • a/+/+/d
  • +/+/+/+

The following subscriptions will not match:

  • a/b/c
  • b/+/c/d
  • +/+/+

# can be used as a wildcard for all remaining levels of hierarchy. This means that it must be the final character in a subscription. With a topic of "a/b/c/d", the following example subscriptions will match:

  • a/b/c/d
  • #
  • a/#
  • a/b/#
  • a/b/c/#
  • +/b/c/#

Zero length topic levels are valid, which can lead to some slightly non-obvious behaviour. For example, a topic of "a//topic" would correctly match against a subscription of "a/+/topic". Likewise, zero length topic levels can exist at both the beginning and the end of a topic string, so "/a/topic" would match against a subscription of "+/a/topic", "#" or "/#", and a topic "a/topic/" would match against a subscription of "a/topic/+" or "a/topic/#".


See the mqtt-gdp-gateway(1) man page for more details.

setup-uhk-gateway.sh

Fetch the U. Michigan from the lab11 repository, install prerequisite packages, compile everything necessary, and install the system startup scripts necessary to run that code on reboot. Note that the U. Michigan packages must run on a properly configured BeagleBone Black from the /home/debian directory. This only installs the U. Michigan code.

install-mqtt-gdp-gateway.sh

This script creates the gdp user and directories necessary to run the MQTT-GDP gateway. It then compiles the mqtt-gdp-gateway code and installs both it and the system startup scripts as described below. Note that the GDP code must be installed before this script is run.

This may run on a BeagleBone, but it isn't necessary. A typical scenario will be to run the mqtt-gdp-gateway on an Ubuntu machine which will pull from the BeagleBone.

mqtt-gdp-gateway@.service.template, mqtt-gdp-gateway.service.template

The Systemd configuration file templates. The former starts a single instance of mqtt-gdp-gateway, i.e., one connected to a single MQTT broker. The latter will startup all mqtt-gdp-gateway instances running on multiple brokers, as defined by systemctl enable.

mqtt-gdp-gateway.conf.template, mqtt-gdp-gateways.conf.template

Deprecated --- used only by Upstart. The Upstart configuration file templates. The former starts a single instance of mqtt-gdp-gateway, i.e., one connected to a single MQTT broker. The latter just invokes start-mqtt-gdp-gateways.sh (below) to start up multiple brokers.

start-mqtt-gdp-gateways.sh

Deprecated --- used only by Upstart. A simple script to start multiple GDP gateway instances. It must be modified when initially installed or to to add new MQTT brokers.

start-mqtt-gdp-gateway.sh

A helper script used to start up the MQTT-GDP gateway program. It is normally invoked by Systemd or Upstart, but it can also be invoked by hand. It takes the fully qualified domain name of the machine running the MQTT broker as its only argument. The domain part of that name is stripped off and the remaining host name is used to find a configuration in $GDP_ETC/mqtt-gateway.$host ($GDP_ETC defaults to /etc/gdp.) The first line of that file is the base name for the GDP log names that will be created; for example, it might be edu.berkeley.eecs.swarmlab, which will result in logs named edu.berkeley.eecs.swarmlab.device._id_. The remaining lines of that file are the names of devices of interest. Those names will be used both for building the MQTT topic name and the GDP log name.

Logs are not created if they cannot be found to avoid having logs created on more than one server if a router happens to be down when this script is run. The gdp-provision-sensor.sh script will create the logs for you when you are doing the initial provisioning.

For example, if the file /etc/gdp/mqtt-gateway/foo.conf contains:

    edu.berkeley.eecs.swarmlab
    dev23
    dev37

then you will have to create the logs

    edu.berkeley.eecs.swarmlab.device.dev23
    edu.berkeley.eecs.swarmlab.device.dev37

then start-mqtt-gdp-gateway.sh foo.example.org will start reading the MQTT broker at foo.example.org for the topics

    device/+/dev23
    device/+/dev37

Those topics will be copied into the corresponding logs.

mosquitto.service

A Systemd configuration file to start up the MQTT broker. Generally used on the BBB.

mqtt-gdp-gateway-logrotate.conf

A configuration to do daily log rotation using logrotate(8).

mqtt-gdp.gateway.c, mqtt-gdp-gateway.1

The source code and documentation for the actual MQTT to GDP gateway program.

setup-common.sh

A helper script used by several of the other shell scripts.

Bugs/Warnings

  • The start-mqtt-gdp-gateway.sh script only discovers configuration changes on startup, not on the fly. To add or delete devices the script must be restarted.
  • The Upstart scripts assume that they are running on the same machine as gdplogd; in particular, they start and stop based on an Upstart job named gdplogd running on the same machine. Although this is the common case, there is no reason these scripts cannot run on a GDP client-only system.
  • Note: The Debian systems running on the BeagleBone use systemd as their startup system; hence, the U. Michigan code uses systemd. Similarly, Ubuntu 16.04 (our preferred platform) uses systemd. However, Ubuntu 14.04 uses Upstart; hence the support for that startup mechanism. Upstart support is deprecated and will probably be removed in the future.