Summary 13 — Multimedia Information Retrieval, Part II (Audio Features, Video & MPEG-7)

Source: 09_MIR_Part-II.pdf

1. Table of Contents (Topics Covered)

Gist (2–3 paragraphs)

Continuing MIR, this lecture details audio features and video retrieval. An audio waveform has three dimensions — loudness (usually ignored for IR), pitch (the musical note / lowest harmonic frequency), and timbre (everything else; identifies the instrument and carries word/emotional content). Key representations: the spectrogram (frequency over time), the chromagram (spectrogram folded into a 12-D note/chroma vector, used in music IR), and MFCC (Mel-Frequency Cepstral Coefficients — a ~13-D vector capturing the broad spectral shape, the standard timbre representation for speech).

Video retrieval relies on video abstracts/summaries that capture content concisely. Static (frame-based) summaries use key-frames assembled into storyboards (thumbnail arrays in chronological order; sophisticated variants like Video Manga use variable-size thumbnails reflecting importance), and mosaics/salient stills that aggregate camera/object motion (pan, zoom, timeprint) into a single panoramic image. Dynamic (video-based) summaries add time and audio: slide shows, moving storyboards (synced to audio), and movie trailers (e.g., the MoCA project’s 3-step segment→select→assemble pipeline). Interactive tools like movieDNA visualize whole video collections as a DNA-like fingerprint. Summary evaluation remains largely subjective (can users do tasks with the summary instead of the full video?).

The lecture closes with MPEG-7, a standard for describing multimedia at both low and high levels via Descriptors (D) and Description Schemes (DS). It standardizes color (dominant/scalable/layout/structure), texture (Gabor-filter homogeneous texture, edge histograms), shape (region-based ART, contour-based CSS, 3D), motion (camera motion, trajectories, motion activity), localization, audio (17 temporal/spectral parameters + high-level tools for sound/spoken-content/instrument-timbre/melody), navigation/access summaries, content organization (collections, models), and user-interaction preferences.

2. Key Terminologies — Meaning & Use Cases

Term Meaning Use Case
Pitch The musical note (lowest harmonic frequency) Melody, emotion cues
Timbre Sound quality minus loudness & pitch Instrument ID, word content
Spectrogram Frequency content over time Audio analysis
Chromagram 12-D note vector (octaves folded) Music IR
MFCC ~13-D spectral-shape vector (mel-scaled) Speech/timbre representation
Video abstract/summary Concise content representation Quick browsing
Key-frame Representative still from a shot Static summaries
Storyboard Chronological thumbnail array Video navigation
Salient still / mosaic Panoramic image aggregating motion Pan/zoom shot summary
Dynamic summary Time+audio short video (trailer, slideshow) Audio-heavy videos
movieDNA DNA-like visualization of video collections Collection overview
MPEG-7 Standard for multimedia content description Interoperable metadata
Descriptor (D) / Description Scheme (DS) Feature representation / structure of relations MPEG-7 building blocks
Gabor filters Orientation/scale-tuned texture filters Homogeneous texture descriptor
ART / CSS Angular Radial Transform / Curvature Scale Space Region/contour shape descriptors

3. Process Workflow Diagram

flowchart TD
    subgraph Audio[Audio Feature Extraction]
        A[Waveform] --> B[Spectrogram]
        B --> C[Chromagram - notes]
        B --> D[MFCC - timbre]
    end
    subgraph Video[Video Summarization]
        E[Source Video] --> F[Segment into shots]
        F --> G[Extract key-frames]
        G --> H{Summary type}
        H -->|Static| I[Storyboard / Mosaic]
        H -->|Dynamic| J[Trailer / Moving storyboard]
        H -->|Interactive| K[movieDNA]
    end
    C --> L[MPEG-7 Descriptors / Schemes]
    D --> L
    I --> L
    J --> L
    L --> M[Indexed, searchable multimedia]

4. ELI5 — Complex Terms Explained Simply