Computational Visual Music / Algorithmic Abstract Animation

Image as music, generated by code.

What it is

A school of digital image-making where abstract form, color and motion are computed and structured like music — sometimes synced to a soundtrack, sometimes built as silent visual rhythm. The craft treats geometry, motion curves and harmonic ratios as the raw material, generating non-representational imagery from algorithms rather than recording the world with a camera. It runs from the first machine-computed films of the 1960s–70s to today's real-time, audio-reactive and live-coded visuals. The deep claim is that moving image and music share a grammar of pattern, repetition and transformation that a program can author directly.

How it is made

Motion as parameterized math

Form and movement are defined as equations and rules — harmonic series, sine sweeps, rotations, point-by-point transformations — so a few numeric parameters drive an entire sequence. Whitney called this 'harmonic progression' across visual elements; Cuba choreographed sixteen 100-point objects by writing the math that moved each one. Tuning a coefficient reshapes the whole piece.

Sound and image from one system

Audio drives the picture (FFT/spectral analysis maps frequency bands onto particle charge, color or scale, as in Hodgin's Magnetosphere) or both are generated from a shared data source and clock (Ikeda converts data and waveforms into synchronized barcode patterns). The link is structural, not decorative — the visual is an analogue of the sound's structure.

Real-time vs. frame-computed

Early work was rendered offline on institutional hardware (vector displays photographed frame by frame). Modern tools run live: a GLSL shader, a Processing/openFrameworks sketch or a Hydra patch recomputes every frame at performance speed, so the artist improvises and the image responds to incoming audio or live-edited code.

Where it comes from

Analog ancestors (1920s–1960s)

The optical era set the goal but used no computation: Walther Ruttmann's and Oskar Fischinger's hand-painted abstract films, Mary Ellen Bute's oscilloscope-assisted 'seeing sound' shorts, Jordan Belson's photographed light works, and the color-organ tradition (Wilfred's Clavilux). These pioneers defined 'visual music' but predate the machine that this atlas tracks — they are origins, not part of the digital roster.

The machine enters (1958–1975)

John Whitney Sr. built a mechanical analog computer from a WWII M-5/M-7 antiaircraft gun director and used it for Catalog (1961) and Permutations, then moved to digital processes by the early 1970s. IBM made him its first artist-in-residence in 1966; Arabesque (1975), programmed by Larry Cuba, is his digital landmark. His 1980 book Digital Harmony argued music and visual art share one mathematics.

Pure code and the second generation (1975–1990)

Larry Cuba — Whitney's programmer on Arabesque — went fully algorithmic: 3/78 (Objects and Transformations) (1978) and Two Space used Tom DeFanti's real-time vector language GRASS at the University of Illinois at Chicago; Calculated Movements (1985) used the Datamax UV-1 and Zgrass. He also programmed the Death Star briefing wireframe in Star Wars (1977). This established abstract animation written, not drawn.

Artists

  • John Whitney Sr.

    Catalog (1961, mechanical analog computer); Permutations; Arabesque (1975, digital, programmed by Larry Cuba); book Digital Harmony (1980); IBM's first artist-in-residence (1966)

    Widely called a father of computer animation; converted a WWII gun-director into an analog computer, then moved to digital, treating motion as harmonic progression.

  • Larry Cuba

    3/78 (Objects and Transformations) (1978, GRASS); Two Space; Calculated Movements (1985, Zgrass/Datamax UV-1); Star Wars (1977) Death Star briefing graphics; programmer on Whitney's Arabesque

    Pure algorithmic animator of the 'second generation' who choreographed point-of-light forms by writing the math, using DeFanti's real-time vector languages GRASS and Zgrass.

  • Robert Hodgin

    Magnetosphere (iTunes default visualizer, 2007; FFT-driven particle system, Processing → C++); Into the Trees (realtime A/V performance); Solar; flight404 audio-reactive works

    Generative coder whose FFT-driven particle systems turn audio into real-time abstract motion; his Magnetosphere shipped as the iTunes visualizer.

  • Memo Akten

    Forms (2013, Prix Ars Electronica Golden Nica); Simple Harmonic Motion series; Body Paint; Deep Meditations; Superradiance

    Artist and computer scientist who builds audiovisual works and rhythmic 'simple harmonic motion' systems from code, data and machine learning.

  • Ryoji Ikeda

    datamatics; test pattern (data/sound converted to barcode and binary patterns in real time); cyclo.; supersymmetry; spectra

    Composer-artist who generates tightly synchronized sound and stark binary/barcode visuals from data, with mathematics shaping music, time and space.

  • Casey Reas

    Co-creator of Processing; Process series and Process Compendium 2004–2010; Software Structures; In Silico; Compressed Cinema

    Co-author of Processing whose Process works define abstract motion as short English rule-texts that software turns into endlessly varying generative imagery.

  • Olivia Jack

    Hydra (open-source browser live-coding video synth, WebGL); PIXELSYNTH; Live Lab

    Created Hydra, a browser-based modular video synthesizer for live-coding audio-reactive visuals, carrying the algorithmic-image lineage into real-time performance.

  • Tarik Barri

    Versum (3D audiovisual programming environment); live A/V with Thom Yorke and Nicolas Jaar; 27 (with Robert Henke)

    Dutch audiovisual artist who built Versum, a navigable 3D space where sound and image are one system, performed live as real-time visual music.

Frames

John Whitney Sr.

  • Catalog (1961) by John Whitney Sr.
    Catalog (1961) · John Whitney Sr. · Source
  • Permutations (1968) by John Whitney Sr.
    Permutations (1968) · John Whitney Sr. · Source
  • Arabesque (1975) by John Whitney Sr.
    Arabesque (1975) · John Whitney Sr. · Source

Larry Cuba

  • 3/78 (Objects and Transformations) (1978) by Larry Cuba
    3/78 (Objects and Transformations) (1978) · Larry Cuba · Source
  • Calculated Movements (1985) by Larry Cuba
    Calculated Movements (1985) · Larry Cuba · Source

Robert Hodgin

  • Magnetosphere (2007) by Robert Hodgin
    Magnetosphere (2007) · Robert Hodgin · Source
  • Magnetosphere — iTunes visualizer (2007) by Robert Hodgin
    Magnetosphere — iTunes visualizer (2007) · Robert Hodgin · Source
  • Magnetosphere — early Processing prototype (2004) by Robert Hodgin
    Magnetosphere — early Processing prototype (2004) · Robert Hodgin · Source
  • Solar by Robert Hodgin
    Solar · Robert Hodgin · Source

Memo Akten

  • Forms (2013) by Memo Akten
    Forms (2013) · Memo Akten · Source
  • Forms (2013) by Memo Akten
    Forms (2013) · Memo Akten · Source
  • Simple Harmonic Motion #12 (2015) by Memo Akten
    Simple Harmonic Motion #12 (2015) · Memo Akten · Source
  • Simple Harmonic Motion #11 (2014) by Memo Akten
    Simple Harmonic Motion #11 (2014) · Memo Akten · Source

Ryoji Ikeda

  • datatron / test pattern (Transmediale 2010) by Ryoji Ikeda
    datatron / test pattern (Transmediale 2010) · Ryoji Ikeda · Source
  • data-verse (exhibition) by Ryoji Ikeda
    data-verse (exhibition) · Ryoji Ikeda · Source
  • data-verse (projection detail) by Ryoji Ikeda
    data-verse (projection detail) · Ryoji Ikeda · Source

Casey Reas

  • Process 18 (Process series) by Casey Reas
    Process 18 (Process series) · Casey Reas · Source
  • Compressed Cinema by Casey Reas
    Compressed Cinema · Casey Reas · Source
  • MicroImage by Casey Reas
    MicroImage · Casey Reas · Source
  • In Silico by Casey Reas
    In Silico · Casey Reas · Source

Olivia Jack

  • Hydra (live-coding video synth) by Olivia Jack
    Hydra (live-coding video synth) · Olivia Jack · Source
  • Hydra (live-coding video synth) by Olivia Jack
    Hydra (live-coding video synth) · Olivia Jack · Source

Techniques

  • harmonic-progression-of-motion
  • fft-audio-reactive-mapping
  • point-and-vector-choreography
  • real-time-shader-rendering
  • live-coding-performance
  • parameter-driven-variation
  • data-to-pattern-conversion
  • frame-by-frame-vector-capture
  • particle-force-systems
  • rule-text-generative-systems

Tools

  • mechanical-analog-computer
  • GRASS / Zgrass (DeFanti)
  • vector-display-hardware
  • Processing / p5.js
  • openFrameworks
  • GLSL
  • Hydra
  • TouchDesigner
  • Max/MSP
  • custom-code

What to take from it

Bind the picture to the math of the sound

Don't cut to the beat by hand — analyze the audio (FFT bands, amplitude envelope) and route those numbers into scale, color, charge or rotation. The sync reads as structural because it literally is the sound's structure driving the form.

Compose with harmonic ratios, not keyframes

Whitney's principle: drive many elements with related frequencies/phase offsets so they drift in and out of alignment like a chord resolving. A handful of ratio parameters yields motion that feels musical rather than arbitrary.

Few primitives, many transformations

Cuba's 100-point objects and Reas's Elements show that a tiny vocabulary (points, a form, a behavior) plus rules of transformation generates rich sequences. Start from the simplest mark and let the algorithm vary it.

Make it live to find it

Run the system in real time (Hydra, a shader, a sketch) and improvise against audio or edit the code mid-performance. Watching parameters respond instantly surfaces the good states a frame-by-frame render would never stumble onto.

Further reading