Skip to content

OpenVINO

OpenVINO is an open-source toolkit for optimizing and deploying deep learning models.
Compiles models for your hardware.
 Supports Linux and Windows
Supports CPU / GPU / iGPU / NPU
 Supports INTEL dGPUs, iGPUs and NPUs.
Supports AMD dGPUs and iGPUs on Windows with FP16 support.
 Supports NVIDIA GPUs.
Supports any GPU with OpenCL support.
 Supports CPUs with BF16, FP16 or FP32 support.
Supports multiple devices at the same time using Hetero Device*.

It is basically a TensorRT / Olive competitor that works with any hardware.

Installation

Preparations

Note

Do not mix OpenVINO with your old install. Treat OpenVINO as a separate backend.

Running SD.Next with OpenVINO

Open CMD / Terminal in a folder you want to install SD.Next and install SD.Next from Github with this command: 

git clone https://github.com/vladmandic/sdnext

Then enter into the sdnext folder:

cd sdnext

Then start WebUI with this command:

Windows:

.\webui.bat --use-openvino

Linux:

./webui.sh --use-openvino

Note

It will install the necessary libraries at the first run so it will take a while depending on your internet.

Running SD.Next with Docker

Check out the Docker wiki if you want to build a custom Docker image.

Using Docker with a prebuilt image: 

export SDNEXT_DOCKER_ROOT_FOLDER=~/sdnext
sudo docker run -it \
  --name sdnext-openvino \
  --device /dev/dri \
  -p 7860:7860 \
  -v $SDNEXT_DOCKER_ROOT_FOLDER/app:/app \
  -v $SDNEXT_DOCKER_ROOT_FOLDER/python:/mnt/python \
  -v $SDNEXT_DOCKER_ROOT_FOLDER/data:/mnt/data \
  -v $SDNEXT_DOCKER_ROOT_FOLDER/models:/mnt/models \
  -v $SDNEXT_DOCKER_ROOT_FOLDER/huggingface:/root/.cache/huggingface \
  disty0/sdnext-openvino:latest

Note

It will install the necessary libraries at the first run so it will take a while depending on your internet.
Resulting docker image will use 1.1 GB disk space (uncompressed) for the docker image and 2.5 GB for the venv.

More Info

Limitations

  • Same limitations with TensorRT / Olive applies here too.
  • Compilation takes a few minutes and using LoRas will trigger recompilation.
  • Attention Slicing and HyperTile will not work.

Custom Devices

Use the OpenVINO devices option in Backend Settings if you want to specify a device.
Selecting multiple devices will use multiple devices as a single HETERO device.

If no device is specified, then the default device will be auto selected by OpenVINO.

Model Caching

OpenVINO can save compiled models to cache folder so you won't have to compile them again.
OpenVINO disable model caching option in Backend Settings will disable caching. Disable this option to enable caching.
Directory for OpenVINO cache option in System Paths will set a new location for saving OpenVINO caches.