Document guide for H.264 (#2)

* Setup guide for H.264 * Phrasing, structure * Remove MJPEG primer and rewrite intro * Link to library files directly * Adjust page title * Remove todo * Resolve todo (Janus setup) * Comment all JavaScript * Rephrase to use active voice, clarify details * Phrasing and clarifications * Write backend instructions * Grammar * Use generic URL with port number * Use consistent terminology * Use term “V4L2 device” to refer to the video device * Link “building” section of uStreamer README * Use active voice * Change page title to “demo” * Add comment about the `janus.js` library file * Use window location to construct server URL * Move `videoElement` variable closer to it’s usage * Elaborate why we clone the media stream * Drop obsolete `WITH_OMX` option * Correct path of shared object file * Fix shadowing variable of same name * Send `start` request to avoid (harmless) `400` response * Add `refcount.h` fix Co-authored-by: Jan Heuermann <jan@jotaen.net>
2026-02-18 02:55:46 +00:00 · 2022-04-13 12:31:15 +02:00 · 2022-04-13 12:31:15 +02:00 · a8dfa96db0
commit a8dfa96db0
parent 7ceb8a3da5
1 changed files with 215 additions and 0 deletions
--- a/docs/h264.md
+++ b/docs/h264.md
@ -0,0 +1,215 @@
+# Guide to H.264 streaming
+
+µStreamer supports bandwidth-efficient streaming using [H.264 compression](https://en.wikipedia.org/wiki/Advanced_Video_Coding) and the Janus WebRTC server. This guide explains how to configure the V4L2 device to provide a H.264 video stream via WebRTC, and how to consume that stream within a web application.
+
+## Components and Control Flow
+
+In addition to µStreamer itself, the following components are involved:
+
+- [**Janus**](https://janus.conf.meetecho.com/): A general-purpose WebRTC server. Its purpose is to facilitate the WebRTC connection and communication, but it doesn’t know anything about µStreamer or other media suppliers.
+- [**µStreamer Janus Plugin**](https://github.com/pikvm/ustreamer/tree/master/janus): A Janus plugin to link µStreamer and Janus. It provides the video data via shared memory from µStreamer.
+- [**Janus JavaScript-Client**](https://janus.conf.meetecho.com/docs/JS.html): A frontend library for connecting to the Janus server, and to exchange commands and data with the µStreamer Janus plugin.
+
+This is a high-level overview of the control flow:
+
+1. The V4L2 device starts the µStreamer service.
+1. The V4L2 device starts the Janus WebRTC server with the µStreamer Janus plugin.
+1. The client-side JavaScript application establishes a connection to the Janus server.
+1. The client-side JavaScript application instructs the Janus WebRTC server to attach the µStreamer Janus plugin and to start the video stream.
+
+## Server Setup
+
+### Prerequisites
+
+The following prerequisites need to be installed on the server:
+
+- The system packages that µStreamer depends on; see [µStreamer documentation](https://github.com/pikvm/ustreamer/blob/master/README.md#building).
+- The Janus WebRTC server, with WebSocket transport enabled; see [Janus documentation](https://github.com/meetecho/janus-gateway).
+
+For compiling µStreamer (see section “Installation”), the Janus header files have to be available to the C compiler. By default, the base path of Janus is `/opt/janus`.
+
+```sh
+ln -s /opt/janus/include/janus /usr/include/janus
+```
+
+You might encounter a compiler error raised in `janus/plugins/plugin.h` that reports a missing `refcount.h` header file. To fix this, modify the respective `#include` directive in the `janus/plugins/plugin.h` file and prepend a `../` to the included file name (`#include "refcount.h"` → `#include "../refcount.h"`).
+
+```sh
+cd /usr/include/janus/plugins
+sed -i -e 's|^#include "refcount.h"$|#include "../refcount.h"|g' plugin.h
+```
+
+### Installation
+
+First, compile µStreamer with the Janus plugin option (`WITH_JANUS`).
+
+```sh
+git clone --depth=1 https://github.com/pikvm/ustreamer
+cd ustreamer
+make WITH_JANUS=1
+```
+
+Next, move the compiled `janus/libjanus_ustreamer.so` shared library file to the plugin directory of your Janus installation.
+
+```sh
+mv janus/libjanus_ustreamer.so /opt/janus/lib/janus/plugins/libjanus_ustreamer.so
+```
+
+Finally, specify a qualifier for the shared memory object, so that the µStreamer Janus plugin can read the video data of µStreamer.
+
+```sh
+cat > /opt/janus/lib/janus/configs/janus.plugin.ustreamer.jcfg <<EOF
+memsink: {
+  object = "demo::ustreamer::h264"
+}
+EOF
+```
+
+### Start Up µStreamer and the Janus WebRTC Server
+
+µStreamer needs to be started with the following additional options, so that it can share memory with the µStreamer Janus plugin:
+
+- `--h264-sink` with the qualifier of the shared memory object, in our case: `demo::ustreamer::h264`
+- `--h264-sink-mode` with the permissions bitmask for the shared memory object, e.g. `660`.
+- `--h264-sink-rm` for cleaning up the shared memory object on stop.
+
+The Janus WebRTC server needs to be started with the following additional options, so that it loads the µStreamer Janus plugin and configuration:
+
+- `--configs-folder` with the path to the Janus configuration directory, for example: `/opt/janus/lib/janus/configs/`
+- `--plugins-folder` with the path to the Janus plugin directory, for example: `/opt/janus/lib/janus/plugins/`
+
+## Client Setup
+
+### Prerequisites
+
+The client needs to load the following JavaScript libraries:
+
+- [The WebRTC Adapter library](https://webrtc.github.io/adapter/adapter-8.1.0.js) (`webrtc-adapter.js`, version `8.1.0`)
+- [The JavaScript library of Janus Gateway](https://raw.githubusercontent.com/meetecho/janus-gateway/v1.0.0/html/janus.js) (`janus.js`, version `1.0.0`)
+
+### Control Flow
+
+The only HTML we need is a `<video>` element for rendering the H.264 video stream.
+
+The control flow inside the client-side JavaScript application is this:
+
+1. The client loads and initiates the Janus client library, and establishes a network connection to the Janus server.
+1. The client instructs the server to attach the µStreamer Janus plugin.
+1. On success, the client obtains a plugin handle through which it can send requests to the µStreamer Janus plugin directly. The responses are processed via the `attach` callbacks, in this case:
+   - `onmessage` callback for general messages
+   - `onremotetrack` for the H.264 video stream
+1. The client issues a `watch` request to the µStreamer Janus plugin, which initiates the H.264 stream in the plugin itself. When attaching the plugin for the first time, this doesn’t immediately succeed, but it takes a few seconds for the H.264 media to become available. The client has to retry the `watch` request in this case.
+1. Once the H.264 media is available for streaming in the µStreamer Janus plugin, client and server have to negotiate the underlying parameters of the WebRTC session, such as port or codec information. This procedure is called JSEP (JavaScript Session Establishment Protocol), where the server makes a `jsepOffer` to the client, and the client responds with a `jsepAnswer`.
+1. After the client and the server complete the JSEP exchange, the WebRTC connection is eventually established. The µStreamer Janus plugin automatically begins to deliver the H.264 video stream to the client via WebRTC.
+1. Once the video data starts to arrive on the client, the Janus client library invokes the `onremotetrack` callback. The client attaches the received stream to the `<video>` element. This will render the video stream on the screen.
+
+### Sample Code
+
+```html
+<!DOCTYPE html>
+<html>
+<head>
+  <meta charset="utf-8">
+  <title>µStreamer H.264 demo</title>
+  <script src="https://webrtc.github.io/adapter/adapter-8.1.0.js"></script>
+  <!-- `janus.js` is the JavaScript client library of Janus, as specified above
+       in the prerequisites section of the client setup. You might need to
+       change the `src` path, depending on where you serve this file from. -->
+  <script src="janus.js"></script>
+  <style>
+    video {
+      height: 100%;
+      width: 100%;
+      object-fit: contain;
+    }
+  </style>
+</head>
+<body>
+  <video id="webrtc-output" autoplay playsinline muted></video>
+  <script type="text/javascript">
+    // Initialize Janus library.
+    Janus.init({
+      // Turn on debug logs in the browser console.
+      debug: true,
+
+      // Configure Janus to use standard browser APIs internally.
+      dependencies: Janus.useDefaultDependencies(),
+    });
+
+    // Establish a network connection to the server.
+    const janus = new Janus({
+      // The URL of the server’s websocket endpoint.
+      server: `ws://${window.location.hostname}:8188/`,
+
+      // Callback function, for when the client connected successfully.
+      success: attachUStreamerPlugin,
+
+      // Callback function, for when the client failed to connect.
+      error: console.error,
+    });
+
+    let uStreamerPluginHandle = null;
+
+    function attachUStreamerPlugin() {
+      // Instruct the server to attach the µStreamer Janus plugin.
+      janus.attach({
+        // Qualifier of the plugin.
+        plugin: "janus.plugin.ustreamer",
+
+        // Callback function, for when the server attached the plugin
+        // successfully.
+        success: function (pluginHandle) {
+          uStreamerPluginHandle = pluginHandle;
+          // Instruct the µStreamer Janus plugin to initiate H.264 stream.
+          uStreamerPluginHandle.send({ message: { request: "watch" } });
+        },
+
+        // Callback function, for when the server failed to attach the plugin.
+        error: console.error,
+
+        // Callback function, for when a message arrived from the server.
+        onmessage: function (msg, jsepOffer) {
+          // `503` indicates that the plugin is not ready to stream yet. Retry
+          // the watch request, until the H.264 stream is available.
+          if (msg.error_code === 503) {
+            uStreamerPluginHandle.send({ message: { request: "watch" } });
+            return;
+          }
+
+          // If there is a JSEP offer, respond to it. This will eventually
+          // start the WebRTC connection.
+          if (jsepOffer) {
+            uStreamerPluginHandle.createAnswer({
+              jsep: jsepOffer,
+              // Prevent the client from sending audio and video, as this would
+              // trigger a permission dialog in the browser.
+              media: { audioSend: false, videoSend: false },
+              success: function (jsepAnswer) {
+                uStreamerPluginHandle.send({
+                  message: { request: "start" },
+                  jsep: jsepAnswer,
+                });
+              },
+              error: console.error,
+            });
+          }
+        },
+
+        // Callback function, for when the video stream arrives.
+        onremotetrack: function (mediaStreamTrack, mediaId, isAdded) {
+          if (isAdded) {
+            // Attach the received media track to the video element. By cloning
+            // the `mediaStreamTrack`, we ensure that the stream gets a
+            // distinct, globally unique stream id.
+            const videoElement = document.getElementById("webrtc-output");
+            const stream = new MediaStream();
+            stream.addTrack(mediaStreamTrack.clone());
+            videoElement.srcObject = stream;
+          }
+        },
+      });
+    }
+  </script>
+</body>
+</html>
+```