Project

General

Profile

Statistics
| Branch: | Revision:

gdp-if / gstreamer-log-plugin / README.md @ master

History | View | Annotate | Download (6.87 KB)

Proof-of-Concept GDP <=> GStreamer Integration

Introduction

This directory contains a proof of concept GDP-source and a GDP-sink for GStreamer. GStreamer is a framework to handle streaming multimedia, and has support for various audio/video formats. A well designed GDP source/sink plugin can allow one to leverage the existing elements/plugins in the GStreamer ecosystem.

GStreamer Background

GStreamer is a generalized framework for handling streaming multimedia, with a very plugin-oriented design.

A typical GStreamer application contains a pipeline composed of a source-element, a sink-element, and a variable number of other element in between. Elements are connected together via pads; a sink pad accepts incoming data from a peer element upstream in the pipeline, and a source pad emits data to be fed to a peer element downstream. The data is passed among elements using Buffers that have efficient memory allocation/sharing mechanisms. In addition, there are various kind of Events that are to be used for passing around control related information.

GStreamer itself is C-based, however it has good support for other language bindings, including Python. GStreamer 0.10 and GStreamer-1.x are two forked versions currently in use by many applications, however they are not compatible with each other. Generally speaking, version 0.10 has better examples and documentation (especially when it comes to Python), but the development is frozen on 0.10. This is why the current proof-of-concept uses GStreamer-1.x.

The key idea of GStreamer integration with GDP is to create two elements: a GDP sink for storing data from a generalized pipeline into a GDP-log, and a similar GDP source for retreiving data from a GDP-log to be fed to a pipeline elsewhere. This is precisely what this proof-of-concept strives to achieve.

Prerequisites

The two main requirements are 1) GDP (including Python bindings) and 2) GStreamer-1.x (with its Python bindings).

Various elements are categorized and packaged into groups such "gstreamer-plugins-good", etc. Such groups can be installed independently depending on whether a particular application needs the plugins included in that particular group.

Usage

  • make <testname> creates a new log and uses the newly created log to store some streams and play them back.

Note that it is not possible to use a command-line tool such as gst-launch to use the plugins directly; such a feature is not yet available in GStreamer 1.x. Further, setting properties of elements in a concise pipeline specification doesn't work either; it seems to trigger certain bugs--most likely not in the GDP elements. Because of these limitations, passing even the most basic parameters requires additional work.

A script named gst-launch-w.py is included, which should be a good enough replacement of gst-launch to quickly launch pipelines with a concise specification. However, note that this script does not understand command line flags understood by gst-launch, such as -v for a verbose output. Internally, this wrapper script launches a pipeline with modified specification, and then sets the properties of GDP elements by calling set_property on such elements one at a time.

Internal Data Format

Since GStreamer is a generalized framework to support a wide variety of use-cases, there is no strict requirement of contents of a Buffer. Such decisions are made by individual elements, and in many cases, negotiated dynamically. For example, a given video pipeline can pass along each video frame as a single buffer, whereas another video pipeline can encode multiple video frames per buffer.

In an ideal world, each Buffer corresponds to a record in a GDP-log. However, this can lead to tens of--or in some cases, even hundreds of--records being generated each second. It may be required to store a number of buffers in memory and periodically commit such buffers together to a single GDP record.

In addition, there is metadata information that describes the type of media (called 'caps' in GStreamer terminology). Such metadata information is not necessarily static and can change mid-stream in certain scenarios, e.g. a video stream being received via network that adapts the video quality dynamically based on the available bandwidth. A GStreamer pipeline provides ways to communicate and negotiate such changes. However, a sink that stores the stream for later use needs to store such information together with the actual data.

With these considerations of chunking and meta-information, the proof-of-concept GDP-sink element stores a serialized tuple in each record. This serialized tuple is (caps, buflist), where caps is a string representation of the meta-information, and buflist is a variable sized list of Buffers. In the current design, the size of buflist is tuned by parameters that specify the maximum age of a buffer in memory, total byte-size of uncommitted buffers, and number of uncommitted buffers.

When a GDP-source reads records from a GDP log, it parses the available information and signals the metadata information downstream. There are other kind of control information that is currently not implemented (See Limitations). In addition, certain encoders may not be very happy with potentially abrupt changes in the stream information.

Limitations, Bugs, etc.

Note that this is simply a proof-of-concept, hence there are many many limitations on what this can be used for.

  • Minimal control information passed.
  • During recording, if a stream is abruptly terminated and then resumed again with the same target log, it depends on the encoder/decoder if it will work.
  • Memory allocation is handled poorly by the GDP source element. There is a hard coded maximum size for the buffers that are passed, which means large raw video frames do not work.
  • Current proof-of-concept playback only subscribes to a log from the very beginning; new data that gets appended to the log does not work very nicely once the playback has reached the end.
  • Memory leaks, probably many.
  • Only push mode supported for the source element.
  • Buffering introduces inherent delay in the real-time playback.
  • Launching the GDP sink/source as a standalone plugin does not work (a general limitation of Python plugins in GStreamer 1.x). Hence, a pipeline can only be launched from a Python script.
  • Specifying properties in a textual pipeline specification does not work, due to a potential bug in the Python API of GStreamer.
  • Not all the encoding/decoding formats are happy with the other limitations; some trial and error may be required to get things working.

Resources