GDP

% Writing GDP applications

# A typical application

The log interface acts as a proxy to interact with sensors and actuators. We

describe how one would go about designing a simple application. Imagine an

office building with some temperature sensors and some motor controlled window

blinds. We would like to design an application that looks at the current

temperature and some weather prediction feeds from the cloud to make a smart

decision about the window blinds. 

![An example GDP application](GDP_application_example.png)

As shown above, in order to do so, the application subscribes to all the

relevant logs and writes the actuation values to the actuator log. A gateway

working on behalf of the actuator subscribes to this actuation log and controls

the motors for the window blinds. Note that instead of two separate logs, a

composite log can be created for `Sensor 1` and `Sensor 2` in the diagram

above, provided that the gateway implements the single-writer semantics.

---

# Software Installation and Configuration

In this section, we talk about how to get access, install and use the GDP

client side library. As mentioned earlier, we have a C library for clients with

wrappers in other languages around this C-library.

The main GDP repository can be accessed in read-only mode by using

```

git clone https://repo.eecs.berkeley.edu/git-anon/projects/swarmlab/gdp.git

```

Or, in read-write mode either using HTTPS (requires username, password), or

using SSH (requires key setup):

```

git clone https://repo.eecs.berkeley.edu/git/projects/swarmlab/gdp.git

git clone repoman@repo.eecs.berkeley.edu:projects/swarmlab/gdp.git

```

The main GDP repository contains the core GDP library (`libgdp`), client-side

applications and language bindings, and the log-server (`gdplogd`). GDP-router

is maintained in a separate repository (`gdp_router_click.git`).  However, you

should not need to worry about it if you are just playing around with GDP.

The repository `gdp-if.git` contains various interfaces to the GDP.  It is

not really part of the GDP itself, but it may prove instructive.

## Compilation

In summary, assuming you have the required dependencies installed, `make

install-dev-c` should install the C include files, C libraries, and basic

utility applications into system path. `make install-python` should

install the Python bindings in the system path as well. These `make`

targets do not create any necessary configuration files, however (see

below). For more details, refer to README.md in the main git tree.

## Infrastructure information

Refer to README.md in the main git tree.

*Note that the software/infrastructure is still in very early experimental

phase. We do not make any guarantees on data retention/backups at this moment.

As the code stabilizes, we will make better effort to maintain data. In the

meantime, contact us if you would like to use GDP for anything serious.*

## Configuration

The GDP library, log-server, and various other utility programs consult a

configuration file for the correct parameters to use.  At the very minimum, your

configuration file should contain the GDP router that your client should connect

to (unless someone else is running a local router in the same subnet as you are

in, in which case zeroconf should work). Refer to README.md in the main git

tree.

## Creating logs

The main mechanism to create a log is using `gdp-create` (should be in your

system path after `make install-*`). For example,

```

gdp-create -k none org.example.project.log17a

```

will create a log named `org.example.project.log17a` on one of the default

log-servers at Berkeley.

Although you can create logs with any name, please stick to this convention

(with "project" being the project name or the user name, as appropriate) so we

can avoid name collisions. `-k none` means that `gdp-create` will not attempt to

create a new signature key for signing appended data.  Although crucial to the

operation, key-management is better deferred to a stage when you are familiar

with the basic operations of the GDP.  Also, note that `gdp-create` has several

other command-line options that will be useful later on.

Note that if you don't explicitly specify log-placement, a log-server at

Berkeley is picked at random for hosting your log. This is especially important

if you are running your own log-servers and want control over where data goes.

---

# Writing applications in Python

Even though the GDP library is written in C, we provide a python package `gdp`

that acts as a wrapper around the C-library. This python package enables quick

prototyping using an object-oriented interface to GDP. What follows is a quick

how-to on writing simple GDP programs in Python. Note that this document is

just a starting point and is not intended to be a comprehensive guide to the

complete interface. For a more thorough API documentation, refer to

`/lang/python/README`.

## Python package installation

The package `gdp` should be installed in your system path for python packages.

Once you have the required dependencies for compilation installed, something

like `make install-python` from the root of repository should do the trick (note

that running with `sudo` may be required). Note that this also installs the

client side C libraries and various utilities (such as `gdp-create`) in system

path.

## Appending data

Let's start with a simple `Hello world` program, that writes some data to a

log and reads it back. Before we begin, we need to create the log; see *Creating

logs* above. The tutorial uses the logname `edu.berkeley.eecs.mor.01`, but

please replace it with the name of the log you create.

We need to import the package `gdp` to begin with.

```python

>>> import gdp

```

Once imported, we need to initialize the GDP package by calling `gdp_init()`.

An optional argument to `gdp_init()` is the address of a GDP-router. If no

address provided, a default value of the parameter `swarm.gdp.routers` is used

as configured by EP library. (See README.md for details).

```python

>>> # the following picks a router based on EP library configuration

>>> gdp.gdp_init()

>>> # For a specific router, use the following:

>>> # gdp.gdp_init('gdp-01.eecs.berkeley.edu:8007')

```

As mentioned earlier, we support human readable names for logs. The mechanism

for translating a human readable name to a 256-bit name is probably going to

change in the future, however, it is our hope that it should be a simple

change. The Python wrapper uses instances of class `gdp.GDP_NAME` for a name,

which could be initialized using a human readable name.

```python

>>> # Create a GDP_NAME object from a human readable python string

>>> gin_name = gdp.GDP_NAME("edu.berkeley.eecs.mor.01")

```

Once we have a `GDP_NAME`, we can use this to open a handle to a log/GCL. A log

handle works like a file handle in some ways. We need to tell whether we want

to open the GCL in read-only mode (`gdp.GDP_MODE_RO`), or append-only mode

(`gdp.GDP_MODE_AO`), or read-and-append mode (`gdp.GDP_MODE_RA`).

```python

>>> # assume that this log already exists.

>>> gin_handle = gdp.GDP_GIN(gin_name, gdp.GDP_MODE_RA)

```

Next, let's append a few records to the log. The unit of read/write to a log is

called a record--data with some automatically generated metadata--represented

by a `GDP_DATUM` object. The `GDP_DATUM` object contains a `GDP_BUF` that holds

the actual data. (Please see the C-api for more details on the behavior of

buffer objects, and such).

```python

>>> d = gdp.GDP_DATUM()

>>> for idx in xrange(10):

...   d["buf"].reset()

...   d["buf"].write("Hello world " + str(idx)}

...   gin_handle.append(d)

```

That's it. Ideally, it should finish without throwing any errors, resulting in

10 records append to the log specified. 

Look at `/lang/python/apps/writer_test.py` for a full example.

## Reading data by record number

Next, let's read some data back and see if it matches what we wrote. Note that

we need to tell what record number we want to read, and record numbers start

from 1. To read data, we just use `read_by_recno` method of the GDP_GIN instance

with the record number.

```python

>>> for idx in xrange(1,11):

...   datum = gin_handle.read_by_recno(idx)

...   print datum["recno"], datum["buf"].peek()

(1, 'Hello world 0')

(2, 'Hello world 1')

(3, 'Hello world 2')

(4, 'Hello world 3')

(5, 'Hello world 4')

(6, 'Hello world 5')

(7, 'Hello world 6')

(8, 'Hello world 7')

(9, 'Hello world 8')

(10, 'Hello world 9')

```

So far, we saw how to read and write data by record number. However, most of

the times, we are interested in the most recent record. For this, we support

negative record numbers, i.e. `-1` refers to the most recent record, `-2`

refers to the second most recent record, and so on.

Look at `/lang/python/apps/reader_test.py` for a full example.

## Subscriptions

Next, let's see how can we subscribe to a log to get new data from a log as it

gets appended. For this, we use `subscribe_by_recno` method of the `gdp.GDP_GIN`

instance.

```python

>>> # ignore the parameters for the moment

>>> gin_handle.subscribe_by_recno(0, 0, None)

```

This subscription returns events, that we need to process in order to get

notified of the data as it appears.

```python

>>> while True:

...   # this blocks until there is a new event

...   event = gin_handle.get_next_event(None)

...   # Events can be used to get the associated datum

...   if event["type"] == gdp.GDP_EVENT_DATA:

...     datum = event["datum"]

...     print datum["buf"].peek()

...   else: 

...     # we ignore other event types for simplicity

...     break

```

In the above code, `event` is an object of type `GDP_EVENT`, which can be used

to get the associated `GDP_DATUM` (and then `GDP_BUF`). In order to see the

above code in action, open another python console (while this is running), and

append some new data to the log just the way you saw above.

Look at `/lang/python/apps/reader_test_subscribe.py` for a full example.

## Reading multiple records at a time

Reading one record at a time can be very inefficient, especially when reading

large amount of data. For this, we support asynchronous reads to read a range of

records at a time. The interface is similar to `subscribe_by_recno` in some

sense--events are returned as a result of an asynchronous call. 

Look at `/lang/python/apps/reader_test_async.py` for a full example.

## Asynchronous write

*Partially implemented*. In the normal `append` call above, a client sends some

data to the log-server and waits for an acknowledgement before returning

control back to the application. In order to convert this blocking operation to

a non-blocking operation, `append_async` could be used instead of regular

`append`.

Refer to the API documentation at `/lang/python/README` for more details.