open-qwen2vl-base

[olccihyeon]

Model description

is there any plan to add open-qwen2vl-base model?

Open source status

• The model implementation is available
• The model weights are available

Provide useful links for the implementation

No response

[Uvi-12]

Adding open-qwen2vl-base to Transformers could be valuable, as it provides the base multimodal model for research and fine-tuning. @olccihyeon, if you’re okay with it, I’d be happy to help add it.

[huzaifa-jawad367]

Hi! I’d like to help with open-qwen2vl-base integration. If someone is already leading, I’m happy to take clearly-scoped pieces.
Please let me know how I can assist.

[Uvi-12]

Hi @zucchini-nlp @Rocketknight1 , just wanted to confirm if adding open-qwen2vl-base to Transformers would be okay.

[zucchini-nlp]

@Uvi-12 I am not familiar with the model tbh but from the name sounds like same arch as Qwen2-VL? If that's true and we can re-use Qwen2VLForConditionalGeneration, it would be suuper nice

If not, let's wait until the issue get more community reactions before raising a PR. I see that the model hasn't got many downloads and usage, so it might not be worth the future maintenance cost

[Uvi-12]

Thanks, @zucchini-nlp. I checked the open-qwen2vl-base model and found that it isn’t the same as Qwen2-VL. It uses a different visual pipeline, so we can’t directly reuse Qwen2VLForConditionalGeneration. Given the low adoption, it’s probably best to wait for more community interest before adding it.