Microsoft Phi-4 Multimodal: Hands-on Guide

Microsoft has officially expanded its Phi-4 series with the introduction of Phi-4-mini-instruct (3.8B) and Phi-4-multimodal (5.6B), complementing the previously released Phi-4 (14B) model known for its advanced reasoning capabilities. These additions significantly enhance multilingual support, reasoning, and mathematical skills, and introduce multimodal capabilities.

This lightweight open multimodal model integrates text, vision, and speech processing, offering seamless interaction across different data formats. With 128K token context length and 5.6B parameters, Phi-4 Multimodal stands out as a powerful tool optimized for both on-device execution and low-latency inference.

In this article, we will dig deep into Phi-4-multimodal, a state-of-the-art multimodal small language model (SLM) capable of processing text, vision, and audio inputs. We will also explore practical hands-on implementations, helping developers integrate generative AI into real-world applications.

Phi-4 Multimodal: A Leap in AI Evolution

Key Features of Phi-4 Multimodal

Phi-4-multimodal is a cutting-edge AI model designed to process multiple input types. Here’s what makes it stand out:

• Unified Multimodal Processing: Unlike traditional models that require separate pipelines for different input types, Phi-4 integrates speech, vision, and text into a single processing space, thanks to its mixture-of-LoRAs (Low-Rank Adapters).
• Advanced Training Techniques: The model has undergone supervised fine-tuning, Direct Preference Optimization (DPO), and Reinforcement Learning from Human Feedback (RLHF), ensuring high accuracy and safe outputs.
• Multilingual Support: Text processing supports 22 languages, while vision and audio functionalities enhance understanding in key global languages.
• Optimized for Efficiency: Designed for on-device execution, Phi-4 minimizes computational overhead while maintaining state-of-the-art performance.

Supported Modalities and Languages

Phi-4 Multimodal is built to process text, vision, and audio inputs, making it highly versatile. Here’s a breakdown of language support for each modality:

Modality	Supported Languages
Text	Arabic, Chinese, Czech, Danish, Dutch, English, Finnish, French, German, Hebrew, Hungarian, Italian, Japanese, Korean, Norwegian, Polish, Portuguese, Russian, Spanish, Swedish, Thai, Turkish, Ukrainian
Vision	English
Audio	English, Chinese, German, French, Italian, Japanese, Spanish, Portuguese

Architectural Advancements of Phi-4 Multimodal

1. Unified Representation Space

Phi-4’s mixture-of-LoRAs architecture allows simultaneous processing of speech, vision, and text. Unlike earlier models that required distinct sub-models, Phi-4 treats all inputs within the same framework, significantly improving efficiency and coherence.

2. Scalability and Efficiency

• Optimized for low-latency inference, making it well-suited for mobile and edge computing applications.
• Supports larger vocabulary sets, enhancing language reasoning across multimodal inputs.
• Built with smaller yet powerful parameterization (5.6B parameters), allowing efficient deployment without compromising performance.

3. Improved AI Reasoning

Phi-4 performs exceptionally well in tasks that require chart/table understanding and document reasoning, thanks to its ability to synthesize vision and audio inputs. Benchmarks indicate higher accuracy compared to other state-of-the-art multimodal models, particularly in structured data interpretation.

Vision Processing Pipeline

• Vision Encoder:
  • Processes image inputs and converts them into a sequence of feature representations (tokens).
  • Likely uses a pretrained vision model (e.g., CLIP, Vision Transformer).
• Token Merging:
  • Reduces the number of visual tokens to improve efficiency while preserving information.
• Vision Projector:
  • Converts visual tokens into a format compatible with the tokenizer for further processing.

Audio Processing Pipeline

• Audio Encoder:
  • Processes raw audio and converts it into a sequence of feature tokens.
  • Likely based on a speech-to-text or waveform model (e.g., Wav2Vec2, Whisper).
• Audio Projector:
  • Maps audio embeddings into a compatible token space for integration with the language model.

Tokenization and Fusion

• The Tokenizer integrates information from vision, audio, and text by inserting image and audio placeholders into the token sequence.
• This unified representation is then sent to the language model.

The Phi-4 Mini Model

The core Phi-4 Mini model is responsible for reasoning, generating responses, and fusing multimodal information.

• Stacked Transformer Layers:
  • It follows a transformer-based architecture for processing multimodal input.
• LoRA Adaptation (Low-Rank Adaptation):
  • The model is fine-tuned using LoRA (Low-Rank Adaptation) for both vision (LoRAᵥ) and audio (LoRAₐ).
  • LoRA helps efficiently adapt pretrained weights without significantly increasing model size.

How Phi-4 Architecture Works?

1. Image and audio inputs are separately processed by their respective encoders.
2. Encoded representations pass through projection layers to align with the language model’s token space.
3. The tokenizer fuses the information, preparing it for processing by the Phi-4 Mini model.
4. The Phi-4 Mini model, enhanced with LoRA, generates text-based outputs based on multimodal context.

Comparison of Phi-4 Multimodal on Different Benchmarks

Phi-4 Multimodal Audio and Visual Benchmarks

The benchmarks likely assess the models’ capabilities in AI2D, ChartQA, DocVQA, and InfoVQA, which are standard datasets for evaluating multimodal models, particularly in visual question-answering (VQA) and document understanding.

1. s_AI2D (AI2D Benchmark)
   • Evaluates reasoning over diagrams and images.
   • Phi-4-multimodal-instruct (68.9) performs better than InternOmni-7B (53.9) and Gemini-2.0-Flash-Lite (62).
   • Gemini-2.0-Flash (69.4) slightly outperforms Phi-4, while Gemini-1.5-Pro (67.7) is slightly lower.
2. s_ChartQA (Chart Question Answering)
   • Focuses on interpreting charts and graphs.
   • Phi-4-multimodal-instruct (69) outperforms all other models.
   • The next closest competitor is InternOmni-7B (56.1), but Gemini-2.0-Flash (51.3) and Gemini-1.5-Pro (46.9) perform significantly worse.
3. s_DocVQA (Document VQA – Reading Documents and Extracting Information)
   • Evaluates how well a model understands and answers questions about documents.
   • Phi-4-multimodal-instruct (87.3) leads the pack.
   • Gemini-2.0-Flash (80.3) and Gemini-1.5-Pro (78.2) perform well but remain behind Phi-4.
4. s_InfoVQA (Information-based Visual Question Answering)
   • Tests the model’s ability to extract and reason over information presented in images.
   • Phi-4-multimodal-instruct (63.7) is again the top-performing model.
   • Gemini-1.5-Pro (66.1) is slightly ahead, but the other Gemini models underperform.

Phi-4 Multimodal Speech Benchmarks

1. Phi-4-Multimodal-Instruct excels in Speech Recognition, beating all competitors in FLEURS, OpenASR, and CommonVoice.
2. Phi-4 struggles in Speech Translation, performing worse than WhisperV3, Qwen2-Audio, and Gemini models.
3. Speech QA is a weakness, with Gemini-2.0-Flash and GPT-4o-RT far ahead.
4. Phi-4 is competitive in Audio Understanding, but Gemini-2.0-Flash slightly outperforms it.
5. Speech Summarization is average, with GPT-4o-RT performing slightly better.

Phi-4 Multimodal Vision Benchmarks

• Phi-4 is a top performer in OCR, document intelligence, and science reasoning.
• It is solid in multimodal tasks but lags behind in video perception and some math-related benchmarks.
• It competes well with models like Gemini-2.0-Flash and GPT-4o but has room for improvement in multi-image and object presence tasks.

Phi-4 Multimodal Vision Quality Radar Chart

Key Takeaways from the Radar Chart

1. Phi-4-Multimodal-Instruct’s Strengths

• Excels in Visual Science Reasoning: Phi-4 achieves one of the highest scores in this category, outperforming most competitors.
• Strong in Popular Aggregated Benchmark: It ranks among the top models, suggesting robust overall performance across multimodal tasks.
• Competitive in Object Visual Presence Verification: It performs similarly to high-ranking models in verifying object presence in images.
• Decent in Chart & Table Reasoning: While not the best, Phi-4 maintains a competitive edge in this domain.

2. Phi-4’s Weaknesses

• Underperforms in Visual Math Reasoning: It is not a leader in this area, with Gemini-2.0-Flash and GPT-4o outperforming it.
• Lags in Multi-Image Perception: Phi-4 is weaker in handling multi-image or video-based perception compared to models like GPT-4o and Gemini-2.0-Flash.
• Average in Document Intelligence: While it performs well, it is not the best in this category compared to some competitors.

Hands-On Experience: Implementing Phi-4 Multimodal

Microsoft provides open-source resources that allow developers to explore Phi-4-multimodal’s capabilities. Below, we explore practical applications using Phi-4 multimodal.

Required Packages

!pip flash_attn==2.7.4.post1 torch==2.6.0 transformers==4.48.2 accelerate==1.3.0 soundfile==0.13.1 pillow==11.1.0 scipy==1.15.2 torchvision==0.21.0 backoff==2.2.1 peft==0.13.2

Required Imports

import requests
import torch
import os
import io
from PIL import Image
import soundfile as sf
from transformers import AutoModelForCausalLM, AutoProcessor, GenerationConfig
from urllib.request import urlopen

Define Model Path

model_path = "microsoft/Phi-4-multimodal-instruct"

# Load model and processor
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_path, 
    device_map="cuda", 
    torch_dtype="auto", 
    trust_remote_code=True, 
    attn_implementation='flash_attention_2',
).cuda()

Load Generation Config

generation_config = GenerationConfig.from_pretrained(model_path)

Define Prompt Structure

user_prompt = '<|user|>'
assistant_prompt = '<|assistant|>'
prompt_suffix = '<|end|>'

Image Processing

print("\n--- IMAGE PROCESSING ---")
image_url = 'https://www.ilankelman.org/stopsigns/australia.jpg'
prompt = f'{user_prompt}<|image_1|>What is shown in this image?{prompt_suffix}{assistant_prompt}'
print(f'>>> Prompt\n{prompt}')

Download and Open the Image

image = Image.open(requests.get(image_url, stream=True).raw)
inputs = processor(text=prompt, images=image, return_tensors='pt').to('cuda:0')

Generate Response

generate_ids = model.generate(
    **inputs,
    max_new_tokens=1000,
    generation_config=generation_config,
)
generate_ids = generate_ids[:, inputs['input_ids'].shape[1]:]
response = processor.batch_decode(
    generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False
)[0]
print(f'>>> Response\n{response}')

Input Image

Output

The image shows a street scene with a red stop sign in the foreground. The
 stop sign is mounted on a pole with a decorative top. Behind the stop sign,
 there is a traditional Chinese building with red and green colors and
 Chinese characters on the signboard. The building has a tiled roof and is
 adorned with red lanterns hanging from the eaves. There are several people
 walking on the sidewalk in front of the building. A black SUV is parked on
 the street, and there are two trash cans on the sidewalk. The street is
 lined with various shops and signs, including one for 'Optus' and another
 for 'Kuo'. The overall scene appears to be in an urban area with a mix of
 modern and traditional elements.

similarly, you can also for audio processing

print("\n--- AUDIO PROCESSING ---")
audio_url = "https://upload.wikimedia.org/wikipedia/commons/b/b0/Barbara_Sahakian_BBC_Radio4_The_Life_Scientific_29_May_2012_b01j5j24.flac"
speech_prompt = "Transcribe the audio to text, and then translate the audio to French. Use <sep> as a separator between the original transcript and the translation."
prompt = f'{user_prompt}<|audio_1|>{speech_prompt}{prompt_suffix}{assistant_prompt}'
print(f'>>> Prompt\n{prompt}')

# Downlowd and open audio file
audio, samplerate = sf.read(io.BytesIO(urlopen(audio_url).read()))

# Process with the model
inputs = processor(text=prompt, audios=[(audio, samplerate)], return_tensors='pt').to('cuda:0')

generate_ids = model.generate(
    **inputs,
    max_new_tokens=1000,
    generation_config=generation_config,
)
generate_ids = generate_ids[:, inputs['input_ids'].shape[1]:]
response = processor.batch_decode(
    generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False
)[0]
print(f'>>> Response\n{response}')

Use Case:

• AI-powered news reporting through real-time speech transcription.
• Voice-controlled virtual assistants with intelligent interaction.
• Real-time multilingual audio translation for global communication.

Some More Outputs by Phi-4 Multimodal

1. Image Analysis by Phi-4 Multimodal

2. Mathematics Image Analysis by Phi-4 Multimodal

Future of Multimodal AI and Edge Applications

One of the standout aspects of Phi-4-multimodal is its ability to operate on edge devices, making it an ideal solution for IoT applications and environments with limited computing resources.

Potential Edge Deployments:

• Smart Home Assistants: Integrate into IoT devices for advanced home automation.
• Healthcare Applications: Improve diagnostics and patient monitoring through multimodal analysis.
• Industrial Automation: Enable AI-driven monitoring and anomaly detection in manufacturing.

Conclusion

Microsoft’s Phi-4 Multimodal is a breakthrough in AI, seamlessly integrating text, vision, and speech processing in a compact, high-performance model. Ideal for AI assistants, document processing, and multilingual applications, it unlocks new possibilities in smart, intuitive AI solutions.

For developers and researchers, hands-on access to Phi-4 enables cutting-edge innovation—from code generation to real-time voice translation and IoT applications—pushing the boundaries of multimodal AI.