Skip to content

OpenVINO

OpenVINO is an open-source toolkit for optimizing and deploying deep learning models.

  • Compiles models for your hardware.
  • Supports Linux and Windows.
  • Supports CPU, GPU, iGPU, and NPU targets.
  • Supports Intel dGPUs, iGPUs, and NPUs.
  • Supports AMD dGPUs and iGPUs on Windows with FP16 support.
  • Supports NVIDIA GPUs.
  • Supports any GPU with OpenCL support.
  • Supports CPUs with BF16, FP16, or FP32 support.
  • Supports multi-device execution using HETERO mode.

In practice, it is an alternative to TensorRT or Olive that can run on a wide range of hardware.

Installation

Preparations

Note

Do not mix OpenVINO with your old install. Treat OpenVINO as a separate backend.

Running SD.Next with OpenVINO

Open a terminal in the folder where you want to install SD.Next, then clone the repository:

git clone https://github.com/vladmandic/sdnext

Then enter into the sdnext folder:

cd sdnext

Then start WebUI with this command:

Windows:

.\webui.bat --use-openvino

Linux:

./webui.sh --use-openvino

Note

It will install the necessary libraries at the first run so it will take a while depending on your internet.

Running SD.Next with Docker

See the Docker wiki if you want to build a custom image.

To run a prebuilt Docker image:

export SDNEXT_DOCKER_ROOT_FOLDER=~/sdnext
sudo docker run -it \
  --name sdnext-openvino \
  --device /dev/dri \
  -p 7860:7860 \
  -v $SDNEXT_DOCKER_ROOT_FOLDER/app:/app \
  -v $SDNEXT_DOCKER_ROOT_FOLDER/python:/mnt/python \
  -v $SDNEXT_DOCKER_ROOT_FOLDER/data:/mnt/data \
  -v $SDNEXT_DOCKER_ROOT_FOLDER/models:/mnt/models \
  -v $SDNEXT_DOCKER_ROOT_FOLDER/huggingface:/root/.cache/huggingface \
  disty0/sdnext-openvino:latest

Note

It installs required libraries on first run, so startup can take a while depending on your connection. Resulting docker image will use 1.1 GB disk space (uncompressed) for the docker image and 2.5 GB for the venv.

More Info

Limitations

  • Most TensorRT/Olive limitations also apply here.
  • Compilation takes a few minutes and using LoRas will trigger recompilation.
  • Attention Slicing and HyperTile will not work.

Custom Devices

Use the OpenVINO devices option in Backend Settings to select a device. Selecting multiple devices combines them into a single HETERO device.

If no device is specified, OpenVINO auto-selects the default device.

Model Caching

OpenVINO can save compiled models to a cache folder so they do not need to be compiled again.

  • OpenVINO disable model caching in Backend Settings turns caching off.
  • Directory for OpenVINO cache in System Paths sets the cache location.