Frigate NVR 0.17 on Apple Silicon: Running AI Detection on the M1 Neural Engine

Background

When I first set up Frigate NVR on my M1 Mac Mini, the AI detection worked, but it was running entirely on the CPU. Three Amcrest PoE cameras, each pushing a detect stream, with every frame getting processed through a CPU-based ONNX detector at ~29ms per inference. The M1 handled it, but the machine was running hot, skipping frames, and the recording pipeline kept backing up with warnings every five seconds about "too many unprocessed recording segments." The M1 has this beautiful 16-core Neural Engine sitting right there, purpose-built for exactly this kind of workload, and I wasn't using it at all.

Frigate 0.17 changed that. The release introduced native Apple Silicon support through a ZMQ proxy architecture that lets the Neural Engine handle object detection from outside the Docker container. I had to try it.

The problem with Docker and Apple Silicon

Here's the catch that makes this interesting: Docker Desktop on macOS runs a Linux VM under the hood. Frigate lives inside that VM, and from inside a Linux container, you simply cannot access Apple's CoreML framework or the Neural Engine. It doesn't matter how much RAM you give Docker or how many CPU cores you allocate. The NPU is invisible to the container.

Before I figured this out, I actually thought the problem was resource allocation. I had Docker set to 12GB of RAM on a 16GB machine (leaving macOS gasping for air with 193MB free and 6.5GB in the compressor). I was hoping giving it more memory would somehow offset work to Apple's neural engine. Reducing Docker to 8GB helped the system breathe, but detection was still running on CPU.

How Frigate 0.17 solves it

The architecture is clever. Instead of trying to access the Neural Engine from inside the container, Frigate 0.17 supports a ZMQ detector that runs as a separate process on the Mac host. The flow looks like this:

1. Frigate (inside Docker) sends video frames over TCP via ZMQ
2. The detector client (on the host) receives frames, runs ONNX inference through CoreML
3. CoreML routes the inference to the Neural Engine
4. Results come back to Frigate over the same ZMQ connection

The latency added by this hop is negligible since both processes run on the same machine.

Setting it up

What you need

• Mac with Apple Silicon (M1 or newer)
• Frigate 0.17+ with the standard-arm64 Docker image
• Python 3.11 on the host (I used pyenv)
• The apple-silicon-detector client
• A YOLO model exported to ONNX format

Step 1: Install Python 3.11 via pyenv

My M1 only had Python 3.9.6 from Xcode Command Line Tools. The detector needs 3.11+.

curl https://pyenv.run | bash

Add to your ~/.zshrc:

export PYENV_ROOT="$HOME/.pyenv"
[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"

Then install Python 3.11:

pyenv install 3.11

Step 2: Clone and install the detector

cd ~
git clone https://github.com/frigate-nvr/apple-silicon-detector.git
cd apple-silicon-detector
pyenv local 3.11
make install

Step 3: Export a YOLOv9t model

The detector needs an ONNX model. Frigate doesn't ship one for Apple Silicon, so you need to export it yourself. I used ultralytics to grab YOLOv9t (the tiny variant, perfect for the M1) and export it at 320x320:

# Inside the apple-silicon-detector venv
venv/bin/pip3 install ultralytics onnx onnxslim
venv/bin/python3 -c "
from ultralytics import YOLO
model = YOLO('yolov9t.pt')
model.export(format='onnx', imgsz=320)
"

Then copy it where Frigate can find it:

mkdir -p ~/frigate/config/model_cache
cp yolov9t.onnx ~/frigate/config/model_cache/yolo.onnx

Note: If your pyenv Python was built without lzma support (common if you don't have xz installed), you'll hit a ModuleNotFoundError: No module named '_lzma' from torchvision. You can work around this by mocking the lzma module before importing ultralytics, or just install xz before building Python (brew install xz && pyenv install 3.11).

Step 4: Configure Frigate

Update your config.yml to use the ZMQ detector and the YOLO model:

detectors:
  apple-silicon:
    type: zmq
    endpoint: tcp://host.docker.internal:5555

model:
  model_type: yolo-generic
  width: 320
  height: 320
  input_tensor: nchw
  input_dtype: float
  path: /config/model_cache/yolo.onnx
  labelmap_path: /labelmap/coco-80.txt

The host.docker.internal hostname is how Docker containers reach the Mac host. Port 5555 is where the ZMQ detector listens.

Step 5: Make it persistent with launchd

I ran the detector with nohup for testing, but it wouldn't survive a reboot. And I created a launchd service:

cat > ~/Library/LaunchAgents/com.frigate.apple-silicon-detector.plist << 'EOF'
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
  "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.frigate.apple-silicon-detector</string>
    <key>ProgramArguments</key>
    <array>
        <string>/path/to/apple-silicon-detector/venv/bin/python3</string>
        <string>/path/to/apple-silicon-detector/detector/zmq_onnx_client.py</string>
        <string>--endpoint</string>
        <string>tcp://*:5555</string>
        <string>--providers</string>
        <string>CoreMLExecutionProvider</string>
        <string>CPUExecutionProvider</string>
    </array>
    <key>WorkingDirectory</key>
    <string>/path/to/apple-silicon-detector</string>
    <key>RunAtLoad</key>
    <true/>
    <key>KeepAlive</key>
    <true/>
    <key>StandardOutPath</key>
    <string>/path/to/apple-silicon-detector/detector.log</string>
    <key>StandardErrorPath</key>
    <string>/path/to/apple-silicon-detector/detector.err</string>
</dict>
</plist>
EOF

Load it:

launchctl load ~/Library/LaunchAgents/com.frigate.apple-silicon-detector.plist

The KeepAlive key means launchd will restart the detector if it crashes. RunAtLoad means it starts automatically on login. Between this and Docker Desktop's own auto-start, the whole stack comes up on boot without touching anything.

Step 6: Start everything up

Start the detector first (or let launchd handle it), then start Frigate:

docker start frigate

Check the detector log to confirm CoreML loaded:

INFO - Loading ONNX model with providers: ['CoreMLExecutionProvider']
INFO - Model input: images, shape: [1, 3, 320, 320], type: tensor(float)
INFO - Model ready for inference

The results

I let it run overnight with three cameras. Here's what changed:

After 12 hours of continuous operation

Total detection throughput: 51.9 FPS across all cameras. Zero frames dropped. The "too many unprocessed recording segments" warnings that plagued the CPU detector? Completely gone.

The overall system still runs warm (Frigate container uses ~94% of its allocated 4 CPU cores), but that's ffmpeg doing stream decoding and recording, not detection. The actual AI inference is invisible from a CPU perspective because it's happening on dedicated silicon.

What I learned

The M1's Neural Engine is genuinely underutilized in most homelab setups. Apple doesn't make it easy to access from non-native environments (hence the Docker workaround), but the Frigate team built an elegant solution with the ZMQ proxy pattern. The key insight is that you don't need the detection to happen inside the container. Shipping frames over a local TCP socket adds almost no latency, and you get to use hardware that would otherwise sit completely idle.

If you're running Frigate on Apple Silicon with the CPU detector, this upgrade is worth the 30 minutes it takes to set up. The M1 Mac Mini went from struggling with three cameras to handling them effortlessly, and I have headroom to add more without worrying about the detection pipeline falling behind.

The Mac Mini M1 continues to be one of the best value machines in my homelab. NVR with AI detection, monitoring hub, and it does it all silently with no fan noise and barely any power draw. Not bad for a machine Apple doesn't even sell anymore.