GDP

# Tensorflow extensions to use GDP as a file system

This directory contains a set of tools, source files, and patches to

[TensorFlow](https://www.tensorflow.org). Please see [Custom

Filesystem](https://www.tensorflow.org/extend/add_filesys) documentation

for a high level overview.

**Note: This is experimental software. No guarantees are made regarding

  the protection of data. Please use it at your own risk. Specifically,

  do not use this to store sensitive data, or data that you can not

  afford to lose.**

# Mapping files to GDP logs

In an ideal world, one could encapsulate an entire filesystem inside a

single GDP log. This ensures that all the updates, even across different

file, have a sequential ordering.

A slightly less than ideal, but much easier to implement/deploy, soultion

is to dedicate one log for each file. Directories are just special type

of files that contain directory entries (just like any other file system).

The current implementation follows this approach with one extra caveat:

there is only one directory in a filesystem; all subdirectories are merely

empty entries. For example: `gdp://x/y/z/1/2/` is merely a filesystem

rooted in a log `x` with empty entries `y`, `y/z`, `y/z/1`, and `y/z/1/2`.

## Encoding data

See `GDPfs.proto`. Each record is a message of type `GDPfsMsg`. Only the

first message in a log has `GDPfsMeta` set; all other records have either

`GDPfsFchunk` or `GDPfsDentry` set (depending on the type set in the first

record). No mixing and matching of `GDPfsFchunk` and `GDPfsDentry` allowed

in the same log.

# Compiling

There are two high-level ways of providing support for `gdp://` paths

with tensorflow: 

- compile as a standalone library that can be loaded at python runtime

  by using `load_library`. This is easier to use, but requires any

  tensorflow code that requires `gdp://` support to be updated.

- integrate with the tensorflow code and compiled into the main

  tensorflow runtime. This requires custom built tensorflow package, but

  can run existing tensorflow code unmodified.

## Compiling as a loadable library

*See a set of full instructions in a shell script form at the end*

Simply use `make` (assuming GDP is installed properly). Then, at

runtime, use the following:

```

from tensorflow.python.framework import load_library

load_library.load_file_system_library("gdpfs_tf.so")

```

Any tensorflow operations that use, say `tf.gfile.GFile`, should

now be able to resolve paths that begin with `gdp://`.

## Compiling with tensorflow source

**Note: this doesn't quite work yet. Still a work in progress.**

*The instructions are outdated; kept only for future reference*

- First, install GDP on the machine where we are compiling

- Clone tensorflow; revert to git commit `35287be3bb7` (this is the

  commit that is known to work with the patch below).

- Apply patch from `extra/gdp_config.patch` by using `git apply <patch>`

- Copy `extra/gdp.BUILD` to tensorflow root

- Copy `CAAPI` to `~/tensorflow.gdp/CAAPI`

- Copy `tf` to `tensorflow/tensorflow/core/platform/gdp`

- Follow instructions for compiling tensorflow. Briefly:

  - `./configure`. Say `-march=x86-64` for optimization

  - `bazel build --config=opt --cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0" //tensorflow/tools/pip_package:build_pip_package`

  - `bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg`

  - `sudo pip install --upgrade /tmp/tensorflow_pkg/tensorflow-*.whl`

## Testing whether the installation works?

(Based on some instructions [here](https://www.tensorflow.org/versions/master/deploy/s3#smoke_test))

After a successful installation (whether by pip package, or by local

compilation and runtime instantiation), something like the following

should work (with appropriate paths, and loading the `.so` file).

```

$ python

>>> from tensorflow.python.lib.io import file_io

>>> print file_io.stat('gdp://X/') # this should also work with a correct path

```

# Command line utility

For ease of use, a command line utility `gdpfs_cmd` is also provided. At the

very least, this utility allows for viewing the contents of a GDP filesystem,

adding files from local disk to a given GDP filesystem, copying files from a

GDP filesystem to the local disk, and removing files and directories.

Please run `./gdpfs_cmd` for usage instructions.

# Limitations

- The current code should not be relied upon being bug-free. In fact, there

  are a number of seucrity vulnerabilities that are present with the current

  code.

- For the moment, no guarantees are made regarding protection (confidentiality

  and/or integrity) of data. Please use this at your own risk.

- No signature/encryption keys are used at the moment. This will change at some

  point in near future.

- Since signature keys are not implemented right now, the single-writer model

  is not really enforced. Please avoid architecting a solution that relies on

  multiple writers for a given filesystem.

- Recursive operations are not supported from `gdpfs_cmd`. This will change.

- Creating new files in the filesystem has a fixed overhead. Thus, for copying

  a large number of very small files is very slow. Partially, this is because

  of the limited recursive functionality.

# Miscellaneous

Also see: 

- [Tensorflow: extending file system](https://www.tensorflow.org/extend/add_filesys)

- [Tensorflow file_system.h](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/platform/file_system.h)

- [Tensorflow on S3](https://www.tensorflow.org/versions/master/deploy/s3#smoke_test)

- [Tensorflow compilation instructions](https://www.tensorflow.org/install/install_sources)

- [Loadable extensions at runtime](https://www.tensorflow.org/extend/adding_an_op#compile_the_op_using_your_system_compiler_tensorflow_binary_installation)

- [IPTF: IPFS with tensorflow](https://github.com/tesserai/iptf)

- [Bazel: adding external dependencies](https://docs.bazel.build/versions/master/cpp-use-cases.html#adding-dependencies-on-precompiled-libraries)

# Amazon EC2 installation instructions (Ubuntu 16.04)

Here's a list of steps that one can perform on a vanilla EC2 image

```

#!/bin/bash

NUM_PROCS=`cat /proc/cpuinfo  | grep "^processor" | wc -l`

NUM_JOBS=$((NUM_PROCS*2))

# sudo apt-get update && sudo apt-get dist-upgrade -y && sudo reboot

# Download protobuf

wget https://github.com/google/protobuf/releases/download/v3.5.1/protobuf-cpp-3.5.1.tar.gz

tar xvzf protobuf-cpp-3.5.1.tar.gz

## Install protobuf pre-reqs

sudo apt-get -y install autoconf automake libtool curl make g++ unzip

( cd protobuf-3.5.1 && \

    ./configure && \

    make -j $NUM_JOBS && \

    make -j $NUM_JOBS check && \

    sudo make install && \

    sudo ldconfig \

)

# Clone GDP

git clone git://repo.eecs.berkeley.edu/projects/swarmlab/gdp.git

# Install GDP dependencies.

(cd gdp/adm/ && ./gdp-setup.sh)

# Download protobuf-c-compiler; we need a new version

sudo apt-get -y --purge autoremove protobuf-c-compiler

git clone https://github.com/protobuf-c/protobuf-c.git

( cd protobuf-c && \

    ./autogen.sh && \

    ./configure && \

    make && \

    sudo make install\

)

# compile GDP

(cd gdp && make && sudo make install-client)

# configure gdp

( sudo mkdir /etc/ep_adm_params && \

  echo "swarm.gdp.zeroconf.enable=false" >> /tmp/gdp && \

  echo "swarm.gdp.routers=gdp-03.eecs.berkeley.edu:8009" >> /tmp/gdp && \

  echo "swarm.gdp.gdp-create.server=gdplogd.mor" >> /tmp/gdp && \

  sudo mv /tmp/gdp /etc/ep_adm_params \

)

# install tensorflow

sudo apt-get -y install python-pip

sudo pip install --upgrade pip

sudo pip install tensorflow

# Get the gdp-if (GDP interfaces) repository

git clone git://repo.eecs.berkeley.edu/projects/swarmlab/gdp-if.git

# compile

(cd gdp-if/tensorflow && make)

( echo "swarm.gdpfs.logprefix = edu.berkerley.tensorflow." >> /tmp/gdpfs && \

  echo "swarm.gdp.debug = gdp.*=5" >> /tmp/gdpfs && \

  echo "swarm.gdpfs.debug = gdpfs.*=19" >> /tmp/gdpfs && \

  sudo mv /tmp/gdpfs /etc/ep_adm_params \

)

##

echo "Include this in your LD path>>>>"

echo export LD_LIBRARY_PATH=`python -c "import tensorflow as tf; print tf.sysconfig.get_lib()"`:\$LD_LIBRARY_PATH

```