Project

General

Profile

Statistics
| Branch: | Revision:

gdp-if / gstreamer-log-plugin / README.md @ master

History | View | Annotate | Download (6.87 KB)

1 fdccdf88 Nitesh Mor
Proof-of-Concept GDP <=> GStreamer Integration
2
==============================================
3
4
# Introduction
5
6
This directory contains a proof of concept GDP-source and a GDP-sink for
7
GStreamer. GStreamer is a framework to handle streaming multimedia, and has
8
support for various audio/video formats. A well designed GDP source/sink plugin
9
can allow one to leverage the existing elements/plugins in the GStreamer
10
ecosystem.
11
12
# GStreamer Background
13
14
GStreamer is a generalized framework for handling streaming multimedia, with a
15
very plugin-oriented design.
16
17
A typical GStreamer application contains a pipeline composed of a
18
source-element, a sink-element, and a variable number of other element in
19
between.  Elements are connected together via pads; a sink pad accepts incoming
20
data from a peer element upstream in the pipeline, and a source pad emits data
21
to be fed to a peer element downstream.  The data is passed among elements using
22
Buffers that have efficient memory allocation/sharing mechanisms. In addition,
23
there are various kind of Events that are to be used for passing around control
24
related information.
25
26
GStreamer itself is C-based, however it has good support for other language
27
bindings, including Python. GStreamer 0.10 and GStreamer-1.x are two forked
28
versions currently in use by many applications, however they are not compatible
29
with each other. Generally speaking, version 0.10 has better examples and
30
documentation (especially when it comes to Python), but the development is
31
frozen on 0.10. This is why the current proof-of-concept uses GStreamer-1.x.
32
33
The key idea of GStreamer integration with GDP is to create two elements: a GDP
34
sink for storing data from a generalized pipeline into a GDP-log, and a similar
35
GDP source for retreiving data from a GDP-log to be fed to a pipeline elsewhere.
36
This is precisely what this proof-of-concept strives to achieve.
37
38
# Prerequisites
39
40
The two main requirements are 1) GDP (including Python bindings) and 2)
41
GStreamer-1.x (with its Python bindings).
42
43
Various elements are categorized and packaged into groups such
44
"gstreamer-plugins-good", etc. Such groups can be installed independently
45
depending on whether a particular application needs the plugins included in that
46
particular group.
47
48
# Usage
49
50
* `make <testname>` creates a new log and uses the newly created log to store
51
  some streams and play them back.
52
53
Note that it is not possible to use a command-line tool such as `gst-launch` to
54
use the plugins directly; such a feature is not yet available in GStreamer 1.x.
55
Further, setting properties of elements in a concise pipeline specification
56
doesn't work either; it seems to trigger certain bugs--most likely not in the
57
GDP elements. Because of these limitations, passing even the most basic
58
parameters requires additional work.
59
60
A script named `gst-launch-w.py` is included, which should be a good enough
61
replacement of `gst-launch` to quickly launch pipelines with a concise
62
specification. However, note that this script does not understand command line
63
flags understood by `gst-launch`, such as `-v` for a verbose output. Internally,
64
this wrapper script launches a pipeline with modified specification, and then
65
sets the properties of GDP elements by calling `set_property` on such elements
66
one at a time.
67
68
69
# Internal Data Format
70
71
Since GStreamer is a generalized framework to support a wide variety of
72
use-cases, there is no strict requirement of contents of a Buffer. Such
73
decisions are made by individual elements, and in many cases, negotiated
74
dynamically. For example, a given video pipeline can pass along each video frame
75
as a single buffer, whereas another video pipeline can encode multiple video
76
frames per buffer.
77
78
In an ideal world, each Buffer corresponds to a record in a GDP-log. However,
79
this can lead to tens of--or in some cases, even hundreds of--records being
80
generated each second. It may be required to store a number of buffers in memory
81
and periodically commit such buffers together to a single GDP record.
82
83
In addition, there is metadata information that describes the type of media
84
(called 'caps' in GStreamer terminology). Such metadata information is not
85
necessarily static and can change mid-stream in certain scenarios, e.g. a video
86
stream being received via network that adapts the video quality dynamically
87
based on the available bandwidth. A GStreamer pipeline provides ways to
88
communicate and negotiate such changes. However, a sink that stores the stream
89
for later use needs to store such information together with the actual data.
90
91
With these considerations of chunking and meta-information, the proof-of-concept
92
GDP-sink element stores a serialized tuple in each record.  This serialized
93
tuple is `(caps, buflist)`, where `caps` is a string representation of the
94
meta-information, and `buflist` is a variable sized list of Buffers. In the
95
current design, the size of `buflist` is tuned by parameters that specify the
96
maximum age of a buffer in memory, total byte-size of uncommitted buffers, and
97
number of uncommitted buffers.
98
99
When a GDP-source reads records from a GDP log, it parses the available
100
information and signals the metadata information downstream. There are other
101
kind of control information that is currently not implemented (See Limitations).
102
In addition, certain encoders may not be very happy with potentially abrupt
103
changes in the stream information.
104
105
# Limitations, Bugs, etc.
106
107
Note that this is simply a proof-of-concept, hence there are many many
108
limitations on what this can be used for.
109
110
* Minimal control information passed.
111
* During recording, if a stream is abruptly terminated and then resumed again
112
  with the same target log, it depends on the encoder/decoder if it will work.
113
* Memory allocation is handled poorly by the GDP source element. There is a
114
  hard coded maximum size for the buffers that are passed, which means large
115
  raw video frames do not work.
116
* Current proof-of-concept playback only subscribes to a log from the very
117
  beginning; new data that gets appended to the log does not work very nicely
118
  once the playback has reached the end.
119
* Memory leaks, probably many.
120
* Only push mode supported for the source element.
121
* Buffering introduces inherent delay in the real-time playback.
122
* Launching the GDP sink/source as a standalone plugin does not work (a general
123
  limitation of Python plugins in GStreamer 1.x). Hence, a pipeline can only be
124
  launched from a Python script.
125
* Specifying properties in a textual pipeline specification does not work, due
126
  to a potential bug in the Python API of GStreamer.
127
* Not all the encoding/decoding formats are happy with the other limitations;
128
  some trial and error may be required to get things working.
129
130
# Resources
131
132
* https://gstreamer.freedesktop.org/documentation/plugin-development/
133
* https://gstreamer.freedesktop.org/documentation/plugins.html
134
* https://lazka.github.io/pgi-docs/#GstBase-1.0
135
* http://www.pygtk.org/articles/subclassing-gobject/sub-classing-gobject-in-python.htm
136
* https://github.com/GStreamer/gstreamer