Multimodal: CV, ASR, TTS

Same Transformer idea — predict next unit over patches, frames, or latents

Modality comparison

ViT: Vision Transformer

Inference code (HuggingFace)

from transformers import AutoImageProcessor, AutoModelForImageClassification
from PIL import Image

processor = AutoImageProcessor.from_pretrained("google/vit-base-patch16-224")
model = AutoModelForImageClassification.from_pretrained("google/vit-base-patch16-224")

image = Image.open("cat.jpg")
inputs = processor(images=image, return_tensors="pt")
logits = model(**inputs).logits
label = model.config.id2label[logits.argmax(-1).item()]
print(label)  # e.g. "Egyptian cat"

Eval: ImageNet top-1 / top-5 accuracy. Fine-tune on your dataset if domain shift (medical, industrial).

SAM & Stable Diffusion

SAM: promptable segmentation

# pip install segment-anything
from segment_anything import sam_model_registry, SamPredictor
import numpy as np

sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h.pth")
predictor = SamPredictor(sam)
predictor.set_image(image_rgb)

# Click prompt (x, y) on object
masks, scores, _ = predictor.predict(
    point_coords=np.array([[520, 375]]),
    point_labels=np.array([1]),
    multimask_output=True,
)
best_mask = masks[scores.argmax()]  # pixel-level segmentation

Eval: IoU (Intersection over Union) on held-out objects; zero-shot on new categories.

Stable Diffusion: text-to-image

from diffusers import StableDiffusionPipeline
import torch

pipe = StableDiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5", torch_dtype=torch.float16)
pipe = pipe.to("cuda")

image = pipe(
    prompt="a cat sitting on the moon, watercolor",
    num_inference_steps=30,
    guidance_scale=7.5,
).images[0]

Eval: FID (distribution vs real images), CLIP score (text-image alignment), human preference A/B.

Whisper: speech to text

import whisper

model = whisper.load_model("base")  # tiny/base/small/medium/large
result = model.transcribe("meeting.wav", language="en")
print(result["text"])

# With word timestamps
for seg in result["segments"]:
    print(seg["start"], seg["end"], seg["text"])

WER calculation

$$\text{WER} = \frac{S + D + I}{N}$$

S = substitutions, D = deletions, I = insertions, N = words in reference transcript.

import jiwer

reference = "the cat sat on the mat"
hypothesis = "the cat sit on mat"
wer = jiwer.wer(reference, hypothesis)
print(f"WER = {wer:.2%}")  # lower is better; <5% = strong ASR

Preprocessing pipeline

1. Resample to 16 kHz mono
2. Compute 80-bin log-mel spectrogram
3. Encoder (bidirectional over time) → decoder (autoregressive text tokens)

Text-to-Speech pipeline

# OpenAI TTS API (simplest production path)
from openai import OpenAI
client = OpenAI()

response = client.audio.speech.create(
    model="tts-1-hd",
    voice="nova",
    input="Hello, welcome to the multimodal course.",
)
response.stream_to_file("welcome.mp3")

# Open-source: ChatTTS / Coqui TTS / edge-tts for local dev

Neural TTS stages

Eval: MOS 1–5 (human listeners), speaker similarity (SIM), character error rate for intelligibility tests.

Vision-Language Models (GPT-4o, LLaVA)

Fuse image encoder + LLM via projection layer. User sends image + text; model predicts text tokens.

from openai import OpenAI
import base64

client = OpenAI()
with open("chart.png", "rb") as f:
    b64 = base64.standard_b64encode(f.read()).decode()

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "What trend does this chart show?"},
            {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}},
        ],
    }],
)
print(response.choices[0].message.content)

Multimodal evaluation

Cross-course eval matrix: Evaluation Guide.

Hands-on lab: multimodal mini pipeline

pip install openai-whisper jiwer transformers diffusers pillow openai

1. CV: Run ViT on 10 images; record top-1 labels; manually score 10/10 plausibility.
2. ASR: Record 5 short clips (or use LibriSpeech samples); transcribe with Whisper; compute WER vs reference.
3. TTS: Generate 3 sentences; rate MOS 1–5 yourself (naturalness + intelligibility).
4. VLM: Send 3 images + questions to GPT-4o or LLaVA; check answer faithfulness to image content.
5. Agent hook: Wrap Whisper + ViT as tools in Agent Core lab; one query: "Describe this image and read aloud."