Models¶

Models are placed in the project_root/models directory

Requirements¶

• The model must be a raw OpenVINO IR model, containing an openvino_model.xml. The name of the model is the name used to run it. For example if you have project_root/demo-model-int4/openvino_model.xml, you would use ovi run demo-model-int4 or use the menu to select it

• The model must not be named Back or Exit

Pulling models¶

One place you can pull models from is HF hub, pre-converted models in raw IR format are available under the OpenVINO Toolkit organization

Here is an example of downloading qwen2.5-coder-7B-Instruct-Int4

# If you do not already have huggingface-hub installed, install it, if you have it installed, make sure you have the latest version:
# pip install huggingface-hub

# Swap project_root for the path of the project root

cd {project_root}/models

# You may customize the directory name, remember this is the name that will be used for the model menu

hf download OpenVINO/Qwen2.5-Coder-7B-Instruct-int4-ov --local-dir qwen2.5-coder:7B-int4

Ultimately you can download the models from anywhere, as long as they are in OpenVINO IR format

Running models¶

The model can be run via any OpenVINO‑supported targets, such as CPU, NPU and GPU, target selection is explained in available settings

Running via model menu¶

Open the model menu by either running ovi and selecting Launch a model or running ovi run

This will open a menu like the following:

↑/↓ navigate • ← back • enter launch • esc/q quit.
============================================================
 ▸ qwen2.5-coder:7B-int4
   qwen2.5-coder-1.5B-int4

navigate the menu and hit enter on the model you would like to launch, you will then see the following:

Connected to raw model 'qwen2.5-coder:7B-int4'. Type '/exit' to quit.
Use ↑/↓ to browse recent prompts.
------------------------------------------------------------

>>> 

You are now connected to your model!

Running directly¶

Alternatively you can simply run ovi run {model name}, where model name is the name of the directory of the model you want to run.

This achieves the same outcome as running via model menu

Modelfiles¶

In the current release, the modelfile is used only to define the device the model should run on, in later releases they will be used for other settings and parameters.

Syntax¶

Lines that begin with a # will be ignored, the syntax is setting=value, there should be no spaces before or after the =

Available settings¶

Lines that are not any of these will be ignored, currently only the following setting is supported:

• DEVICE, set this setting to either CPU, NPU, GPU, or other OpenVINO-supported targets, eg: DEVICE=GPU. If omitted, ovi defaults to CPU