Lesson 14-WebRTC Advanced Features and Optimization – TianyaSchool – Science, Technology & Engineering Learning Platform

High-Definition Video Conferencing

Achieving high-definition video conferencing requires optimizing video quality, managing bandwidth, and leveraging hardware acceleration.

// Acquire high-quality video stream
const constraints = {
  audio: true,
  video: {
    width: { ideal: 1920 }, // Or 'max': 1920 to adapt to device capabilities
    height: { ideal: 1080 },
    frameRate: { ideal: 30 },
    facingMode: 'user',
  }
};

navigator.mediaDevices.getUserMedia(constraints)
  .then(stream => {
    // Set video quality
    const videoTracks = stream.getVideoTracks();
    videoTracks.forEach(track => {
      track.applyConstraints({
        advanced: [{ contentType: 'video/mp4; codecs=vp9' }] // Use high-quality encoder
      }).catch(err => console.error('Error applying constraints', err));
    });

    // Add video stream to peer connection
    // Use pc.addStream or pc.addTrack, depending on WebRTC version
  })
  .catch(error => console.error('Error accessing media devices', error));

// Acquire high-quality video stream
const constraints = {
  audio: true,
  video: {
    width: { ideal: 1920 }, // Or 'max': 1920 to adapt to device capabilities
    height: { ideal: 1080 },
    frameRate: { ideal: 30 },
    facingMode: 'user',
  }
};

navigator.mediaDevices.getUserMedia(constraints)
  .then(stream => {
    // Set video quality
    const videoTracks = stream.getVideoTracks();
    videoTracks.forEach(track => {
      track.applyConstraints({
        advanced: [{ contentType: 'video/mp4; codecs=vp9' }] // Use high-quality encoder
      }).catch(err => console.error('Error applying constraints', err));
    });

    // Add video stream to peer connection
    // Use pc.addStream or pc.addTrack, depending on WebRTC version
  })
  .catch(error => console.error('Error accessing media devices', error));

Screen sharing allows users to share their screen or a specific application window with other conference participants.

async function startScreenShare(pc) {
  try {
    // Acquire screen sharing stream
    const screenStream = await navigator.mediaDevices.getDisplayMedia({ video: true, audio: true });

    // Add screen sharing stream to peer connection
    screenStream.getTracks().forEach(track => pc.addTrack(track, screenStream));

    // Optionally remove previous video tracks or handle multiple video streams as needed
    // pc.getSenders().find(sender => sender.track.kind === 'video').replaceTrack(screenStream.getVideoTracks()[0]);

    // Notify remote peer of new track addition
    pc.createOffer().then(offer => pc.setLocalDescription(offer));
  } catch (error) {
    console.error('Error starting screen share', error);
  }
}

async function startScreenShare(pc) {
  try {
    // Acquire screen sharing stream
    const screenStream = await navigator.mediaDevices.getDisplayMedia({ video: true, audio: true });

    // Add screen sharing stream to peer connection
    screenStream.getTracks().forEach(track => pc.addTrack(track, screenStream));

    // Optionally remove previous video tracks or handle multiple video streams as needed
    // pc.getSenders().find(sender => sender.track.kind === 'video').replaceTrack(screenStream.getVideoTracks()[0]);

    // Notify remote peer of new track addition
    pc.createOffer().then(offer => pc.setLocalDescription(offer));
  } catch (error) {
    console.error('Error starting screen share', error);
  }
}

Advanced Techniques

Bandwidth Adaptation: Use the getStats() method of RTCPeerConnection to periodically check network conditions and dynamically adjust video encoding parameters based on bandwidth changes.

pc.getSenders()[0].getStats().then(stats => {
  // Analyze stats and adjust encoding parameters based on RTT, packet loss, etc.
});

pc.getSenders()[0].getStats().then(stats => {
  // Analyze stats and adjust encoding parameters based on RTT, packet loss, etc.
});

Hardware Acceleration: Promote the use of hardware encoders and decoders to reduce CPU load, typically achieved by setting encoding constraints on video tracks.
Multi-Resolution Encoding: In supported browsers, use Simulcast or SVC (Scalable Video Coding) to provide video streams at different resolutions, accommodating varying bandwidth and device capabilities.

Identity Authentication

Identity authentication is critical for ensuring the security and privacy of WebRTC applications. It typically involves user verification, session management, and secure signaling channels.

JWT (JSON Web Tokens) Authentication

JWT is a widely used authentication mechanism for securely transmitting information between client and server. It allows the server to verify client tokens without querying a database.

Code (Server-Side – Node.js):

const jwt = require('jsonwebtoken');
const secret = 'your_secret_key'; // Should be a secure key

// Generate JWT upon successful login
function generateToken(userId) {
    const payload = { userId: userId };
    const token = jwt.sign(payload, secret, { expiresIn: '1h' }); // Set expiration to 1 hour
    return token;
}

// Verify JWT
function verifyToken(req, res, next) {
    const token = req.headers.authorization.split(' ')[1];
    try {
        const decoded = jwt.verify(token, secret);
        req.user = decoded;
        next();
    } catch (err) {
        res.status(401).send('Invalid Token');
    }
}

const jwt = require('jsonwebtoken');
const secret = 'your_secret_key'; // Should be a secure key

// Generate JWT upon successful login
function generateToken(userId) {
    const payload = { userId: userId };
    const token = jwt.sign(payload, secret, { expiresIn: '1h' }); // Set expiration to 1 hour
    return token;
}

// Verify JWT
function verifyToken(req, res, next) {
    const token = req.headers.authorization.split(' ')[1];
    try {
        const decoded = jwt.verify(token, secret);
        req.user = decoded;
        next();
    } catch (err) {
        res.status(401).send('Invalid Token');
    }
}

OAuth2 Integration

OAuth2 is an open standard for authorization, allowing users to provide an access token instead of a username and password to access their data stored with a specific service provider.

Code (Client-Side – JavaScript):

// Use Google OAuth2
const googleAuth = window.gapi.auth2.init({
    client_id: 'your_client_id.apps.googleusercontent.com'
});

googleAuth.signIn().then(user => {
    // Successful login, retrieve access token
    const accessToken = user.getAuthResponse().access_token;
    // Send accessToken to your server for verification
});

// Use Google OAuth2
const googleAuth = window.gapi.auth2.init({
    client_id: 'your_client_id.apps.googleusercontent.com'
});

googleAuth.signIn().then(user => {
    // Successful login, retrieve access token
    const accessToken = user.getAuthResponse().access_token;
    // Send accessToken to your server for verification
});

WebSocket with Token Verification

In WebRTC applications, WebSocket is commonly used as the signaling channel. Ensure WebSocket connections are secure, allowing only authenticated users to establish connections.

Code (WebSocket Server – Node.js + ws library):

const WebSocket = require('ws');
const jwt = require('jsonwebtoken');

const wss = new WebSocket.Server({ port: 8080 });

wss.on('connection', ws => {
    ws.on('message', message => {
        try {
            const data = JSON.parse(message);
            const token = data.token;
            const decoded = jwt.verify(token, 'your_secret_key');
            // Verification successful, process message
            console.log('Authenticated message:', data);
        } catch (err) {
            // Verification failed
            ws.send(JSON.stringify({ error: 'Unauthorized' }));
            ws.close();
        }
    });
});

const WebSocket = require('ws');
const jwt = require('jsonwebtoken');

const wss = new WebSocket.Server({ port: 8080 });

wss.on('connection', ws => {
    ws.on('message', message => {
        try {
            const data = JSON.parse(message);
            const token = data.token;
            const decoded = jwt.verify(token, 'your_secret_key');
            // Verification successful, process message
            console.log('Authenticated message:', data);
        } catch (err) {
            // Verification failed
            ws.send(JSON.stringify({ error: 'Unauthorized' }));
            ws.close();
        }
    });
});

Statistics Model

WebRTC provides rich statistical data to help developers monitor and optimize communication quality, covering network connections, media streams, encoder performance, and more.

Acquiring Statistics

WebRTC offers the getStats() method to retrieve real-time communication statistics, callable from RTCPeerConnection, RTCDataChannel, or MediaStreamTrack objects.

Code Example (JavaScript):

async function getAndLogStats(pc) {
    const stats = await pc.getStats();
    stats.forEach(stat => {
        console.log(stat);
        // Process specific stats based on stat.type (e.g., inbound-rtp, outbound-rtp, candidate-pair)
    });
}

// Periodically call to monitor continuously
setInterval(() => getAndLogStats(peerConnection), 10000); // Fetch stats every 10 seconds

async function getAndLogStats(pc) {
    const stats = await pc.getStats();
    stats.forEach(stat => {
        console.log(stat);
        // Process specific stats based on stat.type (e.g., inbound-rtp, outbound-rtp, candidate-pair)
    });
}

// Periodically call to monitor continuously
setInterval(() => getAndLogStats(peerConnection), 10000); // Fetch stats every 10 seconds

Key Statistical Metrics and Their Uses:

inbound-rtp/outbound-rtp: Represent received and sent media stream statistics. Focus on packetsLost, jitter, roundTripTime (RTT), and bytesReceived/sent to assess media quality and network latency.
candidate-pair: Describes the currently used ICE candidate pair. Monitor currentRoundTripTime, nominated status, and packetsSent/received to diagnose connection issues.
codec: Provides details about encoders/decoders, such as payloadType, mimeType, and clockRate, to understand the codecs in use.
track: Statistics for specific media tracks, including video resolution and frame rate, useful for adjusting video quality.

Practical Application: Bandwidth Adaptation Based on Statistics

Dynamically adjusting encoding parameters based on statistics to handle network fluctuations is a key strategy for enhancing user experience.

async function adjustBandwidthBasedOnStats(pc) {
    const stats = await pc.getStats();
    const outboundRtp = stats.find(stat => stat.type === 'outbound-rtp' && stat.mediaType === 'video');

    if (outboundRtp) {
        const packetsLostFraction = outboundRtp.packetsLost / outboundRtp.packetsSent;
        if (packetsLostFraction > 0.1) { // If packet loss rate exceeds 10%
            // Reduce video quality to handle network congestion
            const sender = pc.getSenders().find(s => s.track.kind === 'video');
            const parameters = sender.getParameters();
            parameters.encodings[0].maxBitrate /= 2; // Halve the maximum bitrate
            await sender.setParameters(parameters);
        }
    }
}

async function adjustBandwidthBasedOnStats(pc) {
    const stats = await pc.getStats();
    const outboundRtp = stats.find(stat => stat.type === 'outbound-rtp' && stat.mediaType === 'video');

    if (outboundRtp) {
        const packetsLostFraction = outboundRtp.packetsLost / outboundRtp.packetsSent;
        if (packetsLostFraction > 0.1) { // If packet loss rate exceeds 10%
            // Reduce video quality to handle network congestion
            const sender = pc.getSenders().find(s => s.track.kind === 'video');
            const parameters = sender.getParameters();
            parameters.encodings[0].maxBitrate /= 2; // Halve the maximum bitrate
            await sender.setParameters(parameters);
        }
    }
}

Performance Monitoring and Alert System

Send statistical data to a backend server to establish a performance monitoring and alerting system, triggering warnings or automatic optimizations when key metrics exceed thresholds.

async function sendStatsToServer(stats) {
    // Send stats to backend API
    fetch('/api/report-stats', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json',
        },
        body: JSON.stringify(stats),
    });
}

async function sendStatsToServer(stats) {
    // Send stats to backend API
    fetch('/api/report-stats', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json',
        },
        body: JSON.stringify(stats),
    });
}

By deeply analyzing and leveraging WebRTC’s statistics model, developers can monitor application performance in real-time and dynamically adjust based on network conditions and metrics, delivering a more stable, high-quality real-time communication experience.

Latency Optimization and Bandwidth Management

Latency optimization and bandwidth management are critical for providing a smooth real-time communication experience in WebRTC. This involves efficient connection establishment, intelligent media stream encoding, and dynamic adjustments to adapt to changing network conditions.

Fast Connection Establishment (Trickle ICE)

Trickle ICE is a strategy that sends ICE candidates as they are collected, rather than waiting for all candidates to be gathered, significantly reducing connection setup time.

Code Example:
When creating an RTCPeerConnection, ensure iceTransportPolicy is set to all to enable Trickle ICE.

const pc = new RTCPeerConnection({
  iceTransportPolicy: 'all', // Enable Trickle ICE
  iceServers: [{ urls: 'stun:stun.l.google.com:19302' }] // STUN server configuration
});

pc.onicecandidate = event => {
  if (event.candidate) {
    // Send candidate to remote peer immediately
    sendCandidateToRemote(event.candidate);
  }
};

const pc = new RTCPeerConnection({
  iceTransportPolicy: 'all', // Enable Trickle ICE
  iceServers: [{ urls: 'stun:stun.l.google.com:19302' }] // STUN server configuration
});

pc.onicecandidate = event => {
  if (event.candidate) {
    // Send candidate to remote peer immediately
    sendCandidateToRemote(event.candidate);
  }
};

Adaptive Encoding and Bandwidth Estimation

VP8/VP9/H.264 Encoding Parameter Adjustment: Dynamically adjust encoding parameters like resolution, frame rate, and bitrate based on network conditions.

// Get video sender
const sender = pc.getSenders().find(s => s.track.kind === 'video');

// Set encoding parameters
sender.getParameters().encodings[0].scaleResolutionDownBy = 2; // Reduce resolution
sender.getParameters().encodings[0].maxBitrate = 500000; // Limit maximum bitrate
sender.setParameters(sender.getParameters()).catch(logError);

// Get video sender
const sender = pc.getSenders().find(s => s.track.kind === 'video');

// Set encoding parameters
sender.getParameters().encodings[0].scaleResolutionDownBy = 2; // Reduce resolution
sender.getParameters().encodings[0].maxBitrate = 500000; // Limit maximum bitrate
sender.setParameters(sender.getParameters()).catch(logError);

RTCP-Based Bandwidth Estimation: Use RTCP feedback (e.g., REMB – Receiver Estimated Maximum Bitrate) to dynamically adjust sending rates.

pc.getSenders()[0].transport.setRembSender((bitrate, ssrcs) => {
  // Update encoding parameters based on received REMB
});

pc.getSenders()[0].transport.setRembSender((bitrate, ssrcs) => {
  // Update encoding parameters based on received REMB
});

Selecting Optimal Transport Protocols and Codecs

UDP vs TCP: WebRTC defaults to UDP for lower latency. In unstable networks, consider TCP as a fallback.
Codec Selection: Prioritize hardware-accelerated codecs like H.264 to reduce CPU usage and improve efficiency.

Audio-Video Synchronization

Ensure synchronization between audio and video streams by adjusting decoding and playback timestamps.

// Assume you have an audio track and a video track
audioTrack.addEventListener('timeupdate', () => {
  // Adjust video playback based on audio time
  videoElement.currentTime = audioTrack.currentTime;
});

// Assume you have an audio track and a video track
audioTrack.addEventListener('timeupdate', () => {
  // Adjust video playback based on audio time
  videoElement.currentTime = audioTrack.currentTime;
});

Packet Buffering and Jitter Buffer

Properly sizing the Jitter Buffer helps smooth out uneven packet arrival due to network jitter.

// Configure during RTCPeerConnection creation
const pc = new RTCPeerConnection({
  // ...other configurations
  sdpSemantics: 'unified-plan', // Unified plan mode supports flexible buffer control
});

// Control Jitter Buffer via SDP negotiation, typically requiring server-side support

// Configure during RTCPeerConnection creation
const pc = new RTCPeerConnection({
  // ...other configurations
  sdpSemantics: 'unified-plan', // Unified plan mode supports flexible buffer control
});

// Control Jitter Buffer via SDP negotiation, typically requiring server-side support

SVC (Scalable Video Coding) and Multi-Stream Processing

SVC (Scalable Video Coding) is a video encoding technology that encodes video content into multiple layers, each representing a different quality or resolution. This allows receivers to select the appropriate layer for decoding based on network conditions and resource constraints, optimizing latency and bandwidth management in real-time communication.

SVC Support

Currently, WebRTC does not natively support SVC encoding, but it can be implemented by integrating SVC-capable codecs (e.g., OpenH264 with SVC extensions). Additionally, commercial solutions like Janus Gateway offer SVC integration support.

Multi-Stream Processing

Although native SVC support is limited, WebRTC can simulate similar effects by sending multiple video streams at different resolutions (known as Simulcast). Simulcast enables the sender to transmit multiple streams of varying quality simultaneously, allowing the receiver to select the most suitable stream based on its conditions.

Code Example (Simulcast):

const pc = new RTCPeerConnection();
const videoTrack = /* Acquire video track */;
const transceiver = pc.addTransceiver(videoTrack, { direction: 'sendonly' });

// Enable Simulcast
if ('setParameters' in transceiver.sender) {
  transceiver.sender.setParameters({
    encodings: [
      { rid: 'f', scaleResolutionDownBy: 4.0 }, // Lower resolution
      { rid: 'h', scaleResolutionDownBy: 2.0 }, // Medium resolution
      { rid: 'q', scaleResolutionDownBy: 1.0 }, // High resolution
    ],
  }).then(() => {
    console.log('Simulcast enabled');
  }).catch(err => {
    console.error('Failed to enable simulcast', err);
  });
}

const pc = new RTCPeerConnection();
const videoTrack = /* Acquire video track */;
const transceiver = pc.addTransceiver(videoTrack, { direction: 'sendonly' });

// Enable Simulcast
if ('setParameters' in transceiver.sender) {
  transceiver.sender.setParameters({
    encodings: [
      { rid: 'f', scaleResolutionDownBy: 4.0 }, // Lower resolution
      { rid: 'h', scaleResolutionDownBy: 2.0 }, // Medium resolution
      { rid: 'q', scaleResolutionDownBy: 1.0 }, // High resolution
    ],
  }).then(() => {
    console.log('Simulcast enabled');
  }).catch(err => {
    console.error('Failed to enable simulcast', err);
  });
}

Receiver Selection of Appropriate Stream

The receiver needs to listen for the onnegotiationneeded event, handle offer/answer SDP negotiation, and identify and select the appropriate video stream.

pc.onnegotiationneeded = async () => {
  try {
    await pc.setLocalDescription(await pc.createAnswer());
    // Send SDP to remote peer...
  } catch (err) {
    console.error('Error creating answer', err);
  }
};

pc.ontrack = (event) => {
  // Process different quality video streams based on event.streams or event.track.id
  const videoElement = document.getElementById('video');
  videoElement.srcObject = event.streams[0]; // Select highest quality stream as an example
};

pc.onnegotiationneeded = async () => {
  try {
    await pc.setLocalDescription(await pc.createAnswer());
    // Send SDP to remote peer...
  } catch (err) {
    console.error('Error creating answer', err);
  }
};

pc.ontrack = (event) => {
  // Process different quality video streams based on event.streams or event.track.id
  const videoElement = document.getElementById('video');
  videoElement.srcObject = event.streams[0]; // Select highest quality stream as an example
};

Considerations

Codec Support: Ensure the chosen codec (e.g., VP9 or H.264) and browser support Simulcast or SVC.
Bandwidth Management: Configure Simulcast or SVC layers appropriately to avoid excessive bandwidth consumption.
Compatibility: While Simulcast is widely supported in modern browsers, SVC integration may require additional libraries or server support.
Performance Monitoring: Continuously monitor network conditions and application performance, adjusting video stream strategies as needed.

Course: WebRTC Advanced Guide：Media, Transport & Architecture

High-Definition Video Conferencing and Screen Sharing