Compiling the GDP from Source¶
- Compiling the GDP from Source
NOTE: these instructions assume you are starting with the source distribution. This is not appropriate if you are installing the Debian package. (Should add a reference here). Note that our only "Tier One" platform is Ubuntu 16.04. Other platforms may require tweaking as described later.
A very brief summary of the steps:
git clone git://repo.eecs.berkeley.edu/projects/swarmlab/gdp.git cd gdp sh adm/gdp-setup.sh make sudo make install
Getting the Source Code¶
The GDP source code is located in the EECS departmental repository. To get it, use one of the following commands:
git clone git://repo.eecs.berkeley.edu/projects/swarmlab/gdp.git git clone https://repo.eecs.berkeley.edu/git/projects/swarmlab/gdp.git git clone email@example.com:projects/swarmlab/gdp.git
The first form gives you public, read-only access, while the other two require that you have an account on the EECS repository. The third is only available if you have registered your public ssh key with
At the moment, compiles work on many platforms, including Debian, RedHat, MacOS, and FreeBSD. However, some other GDP-related packages work on Debian only (which includes Ubuntu), so you may have difficulties outside the main source tree. See the section on Operating System Quirks below for some hints.
Installing Requisite Packages¶
When compiling from source code, there is no distinction between client and server packages; both are compiled every time. For this reason, you must install all requisite packages before compiling. The easiest way to do this is to run the
adm/gdp-setup.sh script. On some platforms this may not be able to install all the dependencies. If you get warnings or errors, see Operating System Quirks below.
Note that on some systems you may need to install the compile suite as well. See Operating System Quirks below.
Compiling the primary code tree should just be a matter of typing
make in the root of the
gdp tree. If you want to clear out old cruft first, use
make clean all.
If you are going to be debugging it can be convenient to use
O= on the
make command line. This will turn off optimization, which makes debuggers more understandable.
Note: gcc on linux has a bug that causes it to complain about non-constant expressions in an initializer when the
-std=c99 flag is given. Those same expressions are constant in Clang and even in gcc without the
-std=c99 flag. As a result of this problem, we do not use the
-std=c99 flag by default, but this means that not all features of C99 are available. If you want full C99, use
STD=-std=c99 on the make command line.
Further note: At least some versions of gcc give warnings about ignored return values even when the function call has been explicitly voided. We know about this and do not consider it to be a bug in the GDP code. If these warnings bother you we recommend installing clang and using that compiler. (Hint: it gives much better error messages and catches things that gcc does not.)
There are a few compilation flags you can use to turn on additional debugging. You can generally do this using
O=-Dflag on the
make command line; this also turns off optimization so that debuggers can give better information.
EP_OPT_EXTENDED_MUTEX_CHECK— Check for cases such as locking mutexes that are already locked. Also enables some assertions that check lock status. Only works on systems using the NTPL implementation (which means most versions of Linux). This breaks information hiding and hence may fail if the NPTL implementation changes.
GDP_OPT_EXTENDED_CACHE_CHECK— This does some additional checking on the GCL cache. Notably, it fails if a loop in one of the linked lists is detected. This can be expensive, so use it sparingly.
You can install the packages in the system tree using
sudo make install
This command installs all the basics in the tree, including user libraries, server daemons, and documentation, but not all the language bindings, which usually involve additional dependencies. If you prefer installing into something other than the main tree, set the
LOCALROOT variable on the
make command line. For example:
sudo make install LOCALROOT=/usr/local
(Note: this is required on MacOS 10.11 and above.) If you don't need the server code (typical on nearly all client machines), you can use:
sudo make install-client
This installs only the portions required by clients. These will be the applications and libraries needed to link against the GDP. In particular, it does not include the log server.
It is not necessary to install the code into system directories for testing and debugging, but it may be easier.
If you do want to be running a server, simply installing the code is not enough — you need system startup scripts and configuration. First install the binaries using
make install and then:
- Verify that the appropriate servers have been installed into the system directories.
- Create a user named "gdp" to own the database if it does not exist.
- Create any additional system directories needed.
- Set up default system parameters files.
- Install utilities programs used by the running daemons (but not necessarily part of the GDP per se).
- Install startup wrapper scripts.
Other language bindings¶
lang subtree. See the instructions in those directories for compiling and installing.
Operating System Quirks¶
Debian (and derivative systems such as Ubuntu) is the preferred Linux distribution for the GDP, though a nightly build uses Red Hat, so generally speaking it should work. However, it is not the platform on which we do our development, nor do we run our servers on it.
If you are trying to compile on MacOS you will need to install Xcode from the App Store to get the compilers, libraries, and build tools you will need. Instructions for doing this are available at https://www.macports.org/install.php.
Other packages are installed by
adm/gdp-setup.sh. Note that this script will try to determine if you are using
macports. Of the two,
macports is better understood and supports more required packages. If you use
homebrew you will have to install some packages by hand. DO NOT install both package managers; they interfere with each other, and conflicts are nearly guaranteed. The GDP system scripts will refuse to run if both are installed and you do not make an explicit choice.
macports, see https://www.macports.org/install.php. It comes as standard binary packages for most versions: download the
.dmg file, double click on the package, and proceed.
To install brew, see http://brew.sh/ and run
/usr/bin/ruby -e \"$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)\"
Unfortunately, neither of them has all the modules you may need if you are compiling everything, so you may have to download other packages from source code. For example,
macports does not support
mosquitto, which is necessary if you are compiling the MQTT-GDP gateway code (which is not part of the default build). Conversely,
brew does not support Avahi, which is part of the default build.
If you cannot use
macports but you want Zeroconf, you can install Avahi by hand using:
wget https://github.com/lathiat/avahi/releases/download/v0.6.32/avahi-0.6.32.tar.gz tar -zxf avahi-0.6.32.tar.gz cd avahi-0.6.32 ./configure --disable-qt4 --disable-qt3 --disable-gtk \ --disable-gtk3 --disable-gdbm --disable-pygtk \ --disable-python-dbus --disable-mono make -k
cd to the
avahi-0.6.32 directory and run:
sudo make -k install
Note also that
0.6.32 is the current version as of this writing; you may need to find the latest version if that one becomes obsolete.
As an alternative, you can remove Zeroconf from the compilation entirely using:
Be aware that this will increase the need for manual configuration at all user sites.
If make fails because openssl/evp.h is not found, then find evp.h with:
find /usr/local/Cellar -name "evp.h"
bash-3.2$ find /usr/local/Cellar -name "evp.h"
Use the directory above the include directory for the value of LOCAL2:
make LOCAL2=/usr/local/Cellar/openssl/1.0.2k all_noavahi
We do attempt to compile on FreeBSD occasionally in an attempt to promote portability, but some of the other optional packages do not compile or run on FreeBSD. However, the base code should compile.
If you just want to use the GDP client programs, continue reading source:README.md
If you intend to install and maintain your own GDP routers and/or log servers, please continue with source:README-admin.md.
If you plan on debugging the GDP code itself, continue with source:README-developers.md.
The following is a brief explanation of the subdirectories contained in this source tree.
adm — Administrative scripts.
apps — Application programs, including tests.
doc — Some documentation, woefully incomplete.
ep — A library of C utility functions. This is a stripped down version of a library I wrote several years ago. If you look at the code you will see vestiges of some of the stripped out functions. I plan on cleaning this version up and releasing it again.
examples — Some example programs, intended be usable as tutorials.
gdp — A library for GDP manipulation. This is the library that applications must link to access the GDP.
gdplogd — The GDP log daemon. This implements physical (on disk) logs for the GDP. The implementation is still fairly simplistic. It depends on a routing layer (currently gdp_router, in a separate repository).
lang — sub-directories with language-specific application programs and supporting code.
lang/java — Java-specific apps and libraries.
lang/python — Python-specific apps and libraries. See the associated README file for details.
scgilib — An updated version of the SCGI code from http://www.xamuel.com/scgilib/. SCGI permits a web server to access outside programs by opening a socket in a manner much more efficient than basic CGI fork/exec. This is only used for the REST interface.