Compile the model online in EP

[wujiangGitHub]

Hello, some of the backends of onnxruntime require online compilation of models, such as openvino. Some CPUs with poor online compilation of large models may take more than 30 minutes. Is there any solution for this? Or are there any requirements for supporting such backends?

[RyanUnderhill]

Is this an onnxruntime specific issue? Or is there a solution with onnxruntime that genai is not making accessible?

[wujiangGitHub]

Thank you for your answer. 
Because cuda and cpu are different, openvino is the backend engine that needs to compile the model online. The online compilation time of large models may be longer. I would like to ask, if jenAI supports EPs such as openvino and rknpu in the future, can the inference performance of large models meet the requirements?

[RyanUnderhill]

Ah yes, we do plan on supporting OpenVINO and compile based EPs. We will ensure the user experience is usable, as a long compilation time would not be usable. It will probably be through a separate optimized model for these providers.

[wujiangGitHub]

Thank you very much for your answer. Are these optimized models compiled and saved offline? Or can the backend provider optimize the model to improve the online compilation time to make it usable?

[ankitm3k]

Update: ORT GenAI now supports OpenVINO EP, please use the latest commits from https://github.com/microsoft/onnxruntime main branch

[wujiangGitHub]

Hello, how is the performance of the large language generation model?

…

------------------ 原始邮件 ------------------ 发件人: "microsoft/onnxruntime-genai" ***@***.***>; 发送时间: 2025年7月5日(星期六) 下午3:54 ***@***.***>; ***@***.******@***.***>; 主题: Re: [microsoft/onnxruntime-genai] Compile the model online in EP (Discussion #1111) Update: ORT GenAI now supports OpenVINO EP, please use the latest commits from https://github.com/microsoft/onnxruntime main branch — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>