From a8dfa96db0c1ed5ae216a58f4f07a955cd23a212 Mon Sep 17 00:00:00 2001 From: jotaen4tinypilot <83721279+jotaen4tinypilot@users.noreply.github.com> Date: Wed, 13 Apr 2022 12:31:15 +0200 Subject: [PATCH 1/2] Document guide for H.264 (#2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Setup guide for H.264 * Phrasing, structure * Remove MJPEG primer and rewrite intro * Link to library files directly * Adjust page title * Remove todo * Resolve todo (Janus setup) * Comment all JavaScript * Rephrase to use active voice, clarify details * Phrasing and clarifications * Write backend instructions * Grammar * Use generic URL with port number * Use consistent terminology * Use term “V4L2 device” to refer to the video device * Link “building” section of uStreamer README * Use active voice * Change page title to “demo” * Add comment about the `janus.js` library file * Use window location to construct server URL * Move `videoElement` variable closer to it’s usage * Elaborate why we clone the media stream * Drop obsolete `WITH_OMX` option * Correct path of shared object file * Fix shadowing variable of same name * Send `start` request to avoid (harmless) `400` response * Add `refcount.h` fix Co-authored-by: Jan Heuermann --- docs/h264.md | 215 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 215 insertions(+) create mode 100644 docs/h264.md diff --git a/docs/h264.md b/docs/h264.md new file mode 100644 index 0000000..56fed30 --- /dev/null +++ b/docs/h264.md @@ -0,0 +1,215 @@ +# Guide to H.264 streaming + +µStreamer supports bandwidth-efficient streaming using [H.264 compression](https://en.wikipedia.org/wiki/Advanced_Video_Coding) and the Janus WebRTC server. This guide explains how to configure the V4L2 device to provide a H.264 video stream via WebRTC, and how to consume that stream within a web application. + +## Components and Control Flow + +In addition to µStreamer itself, the following components are involved: + +- [**Janus**](https://janus.conf.meetecho.com/): A general-purpose WebRTC server. Its purpose is to facilitate the WebRTC connection and communication, but it doesn’t know anything about µStreamer or other media suppliers. +- [**µStreamer Janus Plugin**](https://github.com/pikvm/ustreamer/tree/master/janus): A Janus plugin to link µStreamer and Janus. It provides the video data via shared memory from µStreamer. +- [**Janus JavaScript-Client**](https://janus.conf.meetecho.com/docs/JS.html): A frontend library for connecting to the Janus server, and to exchange commands and data with the µStreamer Janus plugin. + +This is a high-level overview of the control flow: + +1. The V4L2 device starts the µStreamer service. +1. The V4L2 device starts the Janus WebRTC server with the µStreamer Janus plugin. +1. The client-side JavaScript application establishes a connection to the Janus server. +1. The client-side JavaScript application instructs the Janus WebRTC server to attach the µStreamer Janus plugin and to start the video stream. + +## Server Setup + +### Prerequisites + +The following prerequisites need to be installed on the server: + +- The system packages that µStreamer depends on; see [µStreamer documentation](https://github.com/pikvm/ustreamer/blob/master/README.md#building). +- The Janus WebRTC server, with WebSocket transport enabled; see [Janus documentation](https://github.com/meetecho/janus-gateway). + +For compiling µStreamer (see section “Installation”), the Janus header files have to be available to the C compiler. By default, the base path of Janus is `/opt/janus`. + +```sh +ln -s /opt/janus/include/janus /usr/include/janus +``` + +You might encounter a compiler error raised in `janus/plugins/plugin.h` that reports a missing `refcount.h` header file. To fix this, modify the respective `#include` directive in the `janus/plugins/plugin.h` file and prepend a `../` to the included file name (`#include "refcount.h"` → `#include "../refcount.h"`). + +```sh +cd /usr/include/janus/plugins +sed -i -e 's|^#include "refcount.h"$|#include "../refcount.h"|g' plugin.h +``` + +### Installation + +First, compile µStreamer with the Janus plugin option (`WITH_JANUS`). + +```sh +git clone --depth=1 https://github.com/pikvm/ustreamer +cd ustreamer +make WITH_JANUS=1 +``` + +Next, move the compiled `janus/libjanus_ustreamer.so` shared library file to the plugin directory of your Janus installation. + +```sh +mv janus/libjanus_ustreamer.so /opt/janus/lib/janus/plugins/libjanus_ustreamer.so +``` + +Finally, specify a qualifier for the shared memory object, so that the µStreamer Janus plugin can read the video data of µStreamer. + +```sh +cat > /opt/janus/lib/janus/configs/janus.plugin.ustreamer.jcfg <` element for rendering the H.264 video stream. + +The control flow inside the client-side JavaScript application is this: + +1. The client loads and initiates the Janus client library, and establishes a network connection to the Janus server. +1. The client instructs the server to attach the µStreamer Janus plugin. +1. On success, the client obtains a plugin handle through which it can send requests to the µStreamer Janus plugin directly. The responses are processed via the `attach` callbacks, in this case: + - `onmessage` callback for general messages + - `onremotetrack` for the H.264 video stream +1. The client issues a `watch` request to the µStreamer Janus plugin, which initiates the H.264 stream in the plugin itself. When attaching the plugin for the first time, this doesn’t immediately succeed, but it takes a few seconds for the H.264 media to become available. The client has to retry the `watch` request in this case. +1. Once the H.264 media is available for streaming in the µStreamer Janus plugin, client and server have to negotiate the underlying parameters of the WebRTC session, such as port or codec information. This procedure is called JSEP (JavaScript Session Establishment Protocol), where the server makes a `jsepOffer` to the client, and the client responds with a `jsepAnswer`. +1. After the client and the server complete the JSEP exchange, the WebRTC connection is eventually established. The µStreamer Janus plugin automatically begins to deliver the H.264 video stream to the client via WebRTC. +1. Once the video data starts to arrive on the client, the Janus client library invokes the `onremotetrack` callback. The client attaches the received stream to the `