vision/webrtc

WebRTC 实时视频流

低延迟 H.264 硬件推流与驱动面会话信令

基于 AI 手机硬件屏幕推流与 WebRTC 媒体引擎,向上位机直接推送低延迟、60FPS 的实时视频流。网关负责驱动面信令交互。


4 步建立视频流会话

上位机 (Electron / Web)            网关服务 (:58166)            AI 手机
      │                                    │                                    │
      ├─ 1. POST /sessions (创建会话) ────►│── 启动屏幕推流引擎 ────────────────►│
      │◄─ 返回 session_id + ice_servers ───┤                                    │
      │                                    │                                    │
      ├─ 2. 本地创建 RTCPeerConnection     │                                    │
      ├─ 3. POST /sessions/{id}/signal ───►│── 转发 SDP / ICE 信令 ────────────►│
      │◄─ 接收远端 Answer 信令 ────────────┤                                    │
      │                                    │                                    │
      │================ 4. DTLS-SRTP P2P / TURN 媒体流直接直连 (100ms) ========│

会话驱动接口

1. 申请会话 (POST /robot/screen-stream/sessions)

curl -X POST "http://localhost:58166/robot/screen-stream/sessions" \
  -H "Content-Type: application/json" \
  -H "X-AppKey: <your-key>" \
  -d '{
    "deviceId": "848bd13e2f9",
    "clientInstanceId": "client-uuid-001",
    "scale": 0.5,
    "fps": 20
  }'
{
  "code": 200,
  "data": {
    "session_id": "ses-screen-8f7a1b2c",
    "expires_at": 1717172017000,
    "ice_servers": [{ "urls": ["stun:turn.langhua.ai:3478"] }]
  }
}

2. 交换信令 (POST /robot/screen-stream/sessions/{id}/signal)

将客户端生成的 WebRTC Offer 或 Candidate 发送给手机:

curl -X POST "http://localhost:58166/robot/screen-stream/sessions/ses-screen-8f7a1b2c/signal" \
  -H "Content-Type: application/json" \
  -H "X-AppKey: <your-key>" \
  -d '{
    "type": "offer",
    "payload": "{\"type\":\"offer\",\"sdp\":\"...\"}"
  }'

3. 会话续租与释放

  • 续租:每 3~4 分钟调用一次 POST /robot/screen-stream/sessions/{id}/renew(默认租约 5 分钟)。
  • 释放:关闭页面时调用 DELETE /robot/screen-stream/sessions/{id},手机端立即关闭编码器释放资源。

开箱即用单文件播放器 (viewer.html)

将以下代码保存为 viewer.html,填入 appKey 和 deviceId,双击在 Chrome 打开即可直接观看真机画面:

<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8">
  <title>AI 手机实时推流测试</title>
</head>
<body style="background:#111; color:#eee; font-family:sans-serif; text-align:center; padding:20px;">
  <h2>AI 手机实时屏幕 (60 FPS)</h2>
  <video id="phoneVideo" autoplay playsinline muted style="width:360px; height:800px; background:#000; border-radius:12px; border:2px solid #333;"></video>
  <br/><br/>
  <button id="startBtn" onclick="startStream()" style="padding:10px 24px; font-size:16px; cursor:pointer;">启动推流</button>

  <script>
    const GATEWAY = "http://localhost:58166";
    const APP_KEY = "sk-lh-xxxxxx";        // 替换为你的 AppKey
    const DEVICE_ID = "848bd13e2f9";       // 替换为你的 DeviceId

    async function startStream() {
      document.getElementById("startBtn").disabled = true;

      // 1. 创建会话
      const sessRes = await fetch(`${GATEWAY}/robot/screen-stream/sessions`, {
        method: "POST",
        headers: { "Content-Type": "application/json", "X-AppKey": APP_KEY },
        body: JSON.stringify({ deviceId: DEVICE_ID, clientInstanceId: "test-client", scale: 0.5, fps: 20 })
      });
      const { session_id, ice_servers } = (await sessRes.json()).data;

      // 2. 初始化 PeerConnection
      const pc = new RTCPeerConnection({ iceServers: ice_servers });
      pc.ontrack = (e) => {
        document.getElementById("phoneVideo").srcObject = e.streams[0];
      };

      // 3. 生成并发送 Offer
      const offer = await pc.createOffer({ offerToReceiveVideo: true });
      await pc.setLocalDescription(offer);

      await fetch(`${GATEWAY}/robot/screen-stream/sessions/${session_id}/signal`, {
        method: "POST",
        headers: { "Content-Type": "application/json", "X-AppKey": APP_KEY },
        body: JSON.stringify({ type: "offer", payload: JSON.stringify(offer) })
      });
      console.log("WebRTC 握手完成,画面渲染中...");
    }
  </script>
</body>
</html>

并发驱动与抢占机制(开发者必读)

单设备独占策略(Single-Viewer Exclusive):

  1. 同用户平替:若同一用户在不同浏览器标签或新窗口重复打开同一台手机的画面,网关会自动向旧会话下发 BYE (session_replaced) 并优雅关闭旧通道,确保新窗口立即接管画面。
  2. 跨用户防抢占:若其他账号尝试建立视频会话,网关将直接拒绝并返回错误 “该手机已有其他用户的屏幕实况会话”。