ustreamer/docs/h264.md

Streaming H.264 Video over WebRTC with Janus

µStreamer supports bandwidth-efficient streaming using H.264 compression and the Janus WebRTC server.

This guide explains how to configure µStreamer to provide an H.264 video stream and consume it within a web application using WebRTC.

Components

In addition to µStreamer, H.264 streaming involves the following components:

• Janus: A general-purpose WebRTC server.
• µStreamer Janus Plugin: A Janus plugin that allows Janus to consume a video stream from µStreamer via shared memory.
• Janus JavaScript Client: A frontend library that communicates with the Janus server and the µStreamer Janus plugin.

Control Flow

To connect to a µStreamer video stream over WebRTC, an application must establish the following control flow:

1. The backend server starts the µStreamer service.
2. The backend server starts the Janus WebRTC server with the µStreamer Janus plugin.
3. The client-side JavaScript application establishes a connection to the Janus server.
4. The client-side JavaScript application instructs the Janus server to attach the µStreamer Janus plugin and start the video stream.
5. The client-side JavaScript application renders the video stream in the web browser.

Server Setup

Prerequisites

To integrate with Janus, µStreamer has the following prerequisites:

• The system packages that µStreamer depends on (see the main build instructions).
• The Janus WebRTC server with WebSockets transport enabled (see the Janus documentation).

Fixing Janus C Headers

To compile µStreamer with the Janus option, (see “Installation”), you need to make the Janus header files available within µStreamer's build context.

First, create a symbolic link from /usr/include to the Janus install directory. By default, Janus installs to /opt/janus.

ln -s /opt/janus/include/janus /usr/include/janus

Janus uses #include paths that make it difficult for third-party libraries to build against its headers. To fix this, modify the #include directive in janus/plugins/plugin.h to prepend a ../ to the included file name (#include "refcount.h" → #include "../refcount.h").

sed \
  --in-place \
  --expression 's|^#include "refcount.h"$|#include "../refcount.h"|g' \
  /usr/include/janus/plugins/plugin.h

Installation

First, compile µStreamer with the Janus plugin option (WITH_JANUS).

git clone --depth=1 https://github.com/pikvm/ustreamer
cd ustreamer
make WITH_JANUS=1

The WITH_JANUS option compiles the µStreamer Janus plugin and outputs it to janus/libjanus_ustreamer.so. Move this file to the plugin directory of your Janus installation.

mv \
  janus/libjanus_ustreamer.so \
  /opt/janus/lib/janus/plugins/libjanus_ustreamer.so

Finally, specify a qualifier for the shared memory object so that the µStreamer Janus plugin can read µStreamer's video data.

cat > /opt/janus/lib/janus/configs/janus.plugin.ustreamer.jcfg <<EOF
memsink: {
  object = "demo::ustreamer::h264"
}
EOF

If you're using a TC358743-based video capture device that supports audio capture, run the following command to enable audio streaming:

cat << EOF >> /opt/janus/lib/janus/configs/janus.plugin.ustreamer.jcfg
audio: {
    device = "hw:1"
    tc358743 = "/dev/video0"
}
EOF

Start µStreamer and the Janus WebRTC Server

For µStreamer to share the video stream with the µStreamer Janus plugin, µStreamer must run with the following command-line flags:

• --h264-sink with the qualifier of the shared memory object you specified above (demo::ustreamer::h264)
• --h264-sink-mode with the permissions bitmask for the shared memory object (e.g., 660)
• --h264-sink-rm to clean up the shared memory object when the µStreamer process exits

To load the µStreamer Janus plugin and configuration, the Janus WebRTC server must run with the following command-line flags:

• --configs-folder with the path to the Janus configuration directory (e.g., /opt/janus/lib/janus/configs/)
• --plugins-folder with the path to the Janus plugin directory (e.g., /opt/janus/lib/janus/plugins/)

Client Setup

Once an application's backend server is running µStreamer and Janus, a browser-based JavaScript application can consume the server's video stream.

Prerequisites

The client needs to load the following JavaScript libraries:

• WebRTC Adapter library (webrtc-adapter.js, version 8.1.0)
• Janus Gateway JavaScript library (janus.js, version 1.0.0)

Control Flow

The client-side JavaScript application uses the following control flow:

1. The client loads and initializes the Janus client library.
2. The client establishes a WebSockets connection to the Janus server.
3. The client instructs the Janus server to attach the µStreamer Janus plugin.
4. On success, the client obtains a plugin handle through which it can send requests directly to the µStreamer Janus plugin. The client processes responses via the attach callbacks:
   • onmessage for general messages
   • onremotetrack for the H.264 video stream and (optionally) an Opus audio stream
5. The client issues a watch request to the µStreamer Janus plugin, which initiates the H.264 stream in the plugin itself.
   • It takes a few seconds for uStreamer's video stream to become available to Janus. The first watch request may fail, so the client must retry the watch request.
6. The client and server negotiate the underlying parameters of the WebRTC session. This procedure is called JavaScript Session Establishment Protocol (JSEP). The server makes a jsepOffer to the client, and the client responds with a jsepAnswer.
7. The client issues a start request to the µStreamer Janus plugin to indicate that the client wants to begin consuming the video stream.
8. The µStreamer Janus plugin delivers the H.264 video stream and (optionally) an Opus audio stream to the client via WebRTC.
9. The Janus client library invokes the onremotetrack callback with the video stream. The client attaches the video stream to the <video> element, rendering the video stream in the browser window.
10. (if an audio track is available) The Janus client library invokes the onremotetrack callback with the Opus audio stream. The client adds the audio stream to the <video> element, rendering the audio in the browser window.

Sample Code

<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8">
  <title>µStreamer H.264 demo</title>
  <script src="https://webrtc.github.io/adapter/adapter-8.1.0.js"></script>
  <!-- janus.js is the JavaScript client library of Janus, as specified above in
       the prerequisites section of the client setup. You might need to change
       the `src` path, depending on where you serve this file from. -->
  <script src="janus.js"></script>
  <style>
    video {
      height: 100%;
      width: 100%;
      object-fit: contain;
    }
  </style>
</head>
<body>
  <video id="webrtc-output" autoplay playsinline muted></video>
  <script type="text/javascript">
    // Initialize Janus library.
    Janus.init({
      // Turn on debug logs in the browser console.
      debug: true,

      // Configure Janus to use standard browser APIs internally.
      dependencies: Janus.useDefaultDependencies(),
    });

    // Establish a WebSockets connection to the server.
    const janus = new Janus({
      // Specify the URL of the Janus server’s WebSockets endpoint.
      server: `ws://${window.location.hostname}:8188/`,

      // Callback function if the client connects successfully.
      success: attachUStreamerPlugin,

      // Callback function if the client fails to connect.
      error: console.error,
    });

    let uStreamerPluginHandle = null;

    function attachUStreamerPlugin() {
      // Instruct the server to attach the µStreamer Janus plugin.
      janus.attach({
        // Qualifier of the plugin.
        plugin: "janus.plugin.ustreamer",

        // Callback function, for when the server attached the plugin
        // successfully.
        success: function (pluginHandle) {
          uStreamerPluginHandle = pluginHandle;
          // Instruct the µStreamer Janus plugin to initiate streaming.
          uStreamerPluginHandle.send({ message: { request: "watch", params: {audio: true} } });
        },

        // Callback function if the server fails to attach the plugin.
        error: console.error,

        // Callback function for processing messages from the Janus server.
        onmessage: function (msg, jsepOffer) {
          // If there is a JSEP offer, respond to it. This starts the WebRTC
          // connection.
          if (jsepOffer) {
            uStreamerPluginHandle.createAnswer({
              jsep: jsepOffer,
              // Prevent the client from sending audio and video, as this would
              // trigger a permission dialog in the browser.
              media: { audioSend: false, videoSend: false },
              success: function (jsepAnswer) {
                uStreamerPluginHandle.send({
                  message: { request: "start" },
                  jsep: jsepAnswer,
                });
              },
              error: console.error,
            });
          }
        },

        // Callback function, for when a media stream arrives.
        onremotetrack: function (mediaStreamTrack, mediaId, isAdded) {
          if (isAdded) {
            // Attach the received media track to the video element. Cloning the
            // mediaStreamTrack creates a new object with a distinct, globally
            // unique stream identifier.
            const videoElement = document.getElementById("webrtc-output");
            if (videoElement.srcObject === null) {
              videoElement.srcObject = new MediaStream();
            }
            videoElement.srcObject.addTrack(mediaStreamTrack.clone());
          }
        },
      });
    }
  </script>
</body>
</html>