Towards a conceptually endorsed synesthesia

To remove the barriers between sight and sound,
between the seen world and the heard world!
To bring about a unity and a harmonious
relationship between these two opposite spheres.
What an absorbing task!
The Greeks and Diderot, Wagner and Scriabin –
who has not dreamt of this ideal?
Is there anyone who has made no attempt
to realise this dream?
- Sergei Eisenstein [1]

The word synesthesia stems from the ancient Greek words σύν (syn), "together," and αἴσθησις (aisthēsis), ‘sensation’. Traditionally, (in the field of neuropsychology) synesthesia refers to an anomaly in the human brain (estimates vary between 1 in 20 and 1 in 20000) that results into the transference of an experience by one sense into another. This means that people with synesthesia can, involuntary, for instance ‘see’ sounds, ‘taste’ shapes, and so on.[2] Although synesthesia has been the subject of intense scientific investigation during many centuries, modern researchers have only recently rediscovered the concept. Today, both artists and scientists research synesthesia and the rules that “should be applied to transform sound into image, spatial movement into timbre or harmony into color."[3]
Because synesthesia is a highly subjective matter that only relies upon what is referred to as an anomaly, scientists and artists have not been able to trigger the effect purposefully. Consequently, in the arts the concept of synesthesia has evolved into a metaphor that relates to the act of crossing boundaries between different disciplines or fusing different media into one work.[4] An example of this is the integrated, ‘complete artwork’, which originates from 1849, when the German opera composer Richard Wagner first described his Gesamtkunstwerk. For Wagner, the individual arts of painting, dance, music and poetry had progressed as far as they could. He described that the future of the arts lied within the Gesamtkunstwerk, a synthesis of the poetic, visual, musical and dramatic arts. This total work of art would speak to all the senses at once and result in total immersion.[5]

A Color Box (Len Lye, 1935)
After 40 years of more or less successful attempts to record sound and moving image synchronised, it was Len Lye who finally set out to prove once and for all that music and video together could be part of the language of art. Lye is known to have stated: "All of a sudden it hit me. If there was such a thing as composing music, there could be such a thing as composing motion."[6]
Driven by a lack of money, Lye started off inventing new, controversial ways to create film without a camera. In 1935, he made the experimental movie A Colour Box. In this movie he color-painted and scratched the celluloid, connecting the visuals to the tunes of a Caribbean jazz piece. In 1937 A Colour Box was acquired by the General Post Office, who reissued it with the addition of a 'cheaper parcel post' message. The video continued its existence as a commercial.[7] Today, Len Lye is known as one of the pioneers of the music video genre.
Another telling example of cross modal practice finds its origin in the 1960s, when the artist Dick Higgins, a founder of the Fluxus movement, became well known for promoting interdisciplinary art practice. Following the example of Marcel Duchamp, Higgins calls for unusual combinations of genres. He popularises the term intermedia, which stands for media that investigate the border of multiple art practices. According to Higgins, in the realm of intermedia, any available object or experience can and should be incorporated into the artwork by mapping one structure onto another (in the digital sometimes referred to as transcoding). As a result, these intermedia artworks act in unpredictable ways, producing a new, more complex whole than its original parts.[8]
Variations V (John Cage and Higgins, 1965)
In 1965 John Cage answered to Higgins’ call for intermedia by composing the performance Variations V (with the help of David Tudor, Billy Klüver and others). As most of Cage’s works, the piece revolved around a central concept of chance. The play was generated on the spot, from thirty-five randomly equipped ‘remarks’ that outlined its structure, components, and methodology. The performance’ score was created within two different sound systems. The first sound system was dependent on directional photocells (light sensors) that were connected to an ‘orchestra’ of tape-recorders and record players. As the dancers moved around, they interrupted the light falling on the photocells, which switched the sound on and off. The second system used a series of antennas. When a dancer came within four feet of an antenna, this would initiate sound stemming from short-wave radios. In the background of the performance, screens would show a movie by Stan VanDerBeek and manipulated film footage from Nam June Paik.[9] Cage’s performance Variations V united technology, dance, video and sound, mapping their structures on top of each other, while leaving them dependent on chance. In doing so he created a noisy, multilayered synesthetic whole, that incorporated a cacophony sounds and visuals that stretched the limits of what was understood by the audience.

Today, only 50 years after Cage’s answer to Higgins plea for intermedia art practice, the combination of music and video technologies has become almost ubiquitous. On television there are music videos and jingles, while in discos it is the VJs that have taken the task upon themselves to reconstruct an atmosphere inherent to the music, to let the image expand the aural sensations.

During the last 50 years, throughout which we have seen the rise of digital technologies, not only the frequency with which we come across intermedia works has changed. It is important to realise that the means by which these works have been created, the technologies, also evolved considerably. Today the computer can be used to map the structures of music, like rhythm and pitch, upon a video, often even in real time. Because of its ability to create mathematical correspondences between for instance visual and sonic data, the computer has become an ideal tool to cut and paste layers of different media upon each other.
The development of this kind of digital equipment and the creation of such audiovisual works did not receive an undivided warm welcome. Critics argue that since the commodification of digital techniques that map video and sonic signals seamless on top of each other, the synesthetic audiovisual work of art is no longer extraordinary. What once was a complex study of audiovisual trans-coding, is now all to often reduced to a glossy, instant, generative software-show.
But not only the wide availability of proprietary digital technologies has stimulated the growth of this kind of audiovisual work. With the popularisation of open source platforms like Processing and Pure Data, and the everyday development of new custom software and plugins, it seems we have entered an era in which even custom development of audiovisual software has become an omnipresent standard. We have undeniably arrived at a time where art is fused together with a ready-made customisation culture, and where the plugin-VJ creates industrial wallpaper that morphs to the beat.

Advanced Beauty, Sample clip (flight404, 2007).
(one the most beautiful audiovisual transcodings I know, yet missing some kind of conceptual relation)
With the rise of digital technologies, the process of the artist working in audiovisual media has become all to often both in a lack of meaning or opaque. At present there seem to be over a 1000-plus-one options to transcode the sonic on to the visual. Every day new technical approximations of synesthesia are created, either by rearranging modular structures or from scratch. Even non proprietary software seems to have become just another type of commodity.
The results of this trend became clear to me during my visits to Mapping Festival (Geneva, April-May 2007) and Cimatics Festival (Brussels, 27-30 November 2008), where it struck me how many performances are centered around the presentation of new software that can map the properties of the music better, clearer or differently onto the visuals. As a result some shows were full of interactive wallpapers; the pictures were mostly very pretty, filling up the otherwise terrifyingly empty white walls.
I am of the opinion that these kinds of audiovisual performances often lack a story, symbolic content or concept. We have arrived at a point where the connections between the sonic and the visual have become as beautifully constructed as arbitrary, empty and hollow. “Digital technology enables us to synthesize and synchronize events in diverse media, but it completely bypasses the problem of symbolic representations.”[10] It has become increasingly clear that the development of digital technologies forces us to ask ourselves (as music video producers, VJs or even as an audience) how we can change the fusion of digital video and sound into a less arbitrary and more conceptually synesthetic, technological and meaningful whole.
Although I can imagine that non-modular mechanical and analogue devices would most probably force the artist to explore conceptual synesthesia more fundamentally, since their processes are not just a matter of trans-coding inputs into outputs, my starting point doesn’t lie within a choice of technique, or in a suggestion for a completely new, different set of technologies. I think it is most interesting to explore alternative strategies and concepts within the realm of the digital. In analogue works like A Colour Box and Variations V, the artists researched the technologies of their time and the perception of connections between optic and sonic waves. The works changed the world in the sense that they not only expanded the technologies, but also the human realities, knowledge and experience of the visual and the audible; noise entered the realm of what was accepted. With the help of technology these works created a new space for dialogue between the artist, the participant and the observer.
Although art and communication are often considered to be separate domains, Gerda Lampalzer claims that during the sixties, it was “usually the communication process itself that was central to intermedia artworks.”[11] Essentialist views of art have since then lost much of their prevalence – it is now common to say that a piece of art does not even exist without the human perception of it. Although this might be a philosophically biased statement, we can at least claim that artworks like Len Lye’s and Cage’s involve a form of communication through a technological medium.
In the sixties (and before), the merit of intermedia technologies was the creation of new forms of synesthetic art, which in its turn generated new spaces for dialogue. With the rise of digital technologies, most of the spaces for dialogue are now dominated by plugins that transcode data into other data, leaving the generated audiovisual works often empty and in a lack of symbolic meaning.
Tok Tek and Karl Klomp (Visuals) (New Dutch Electronics, 2007)
As VJs and music video creators, we have to try to get the meaning back into our works and not just focus on new techniques. By realising that the visual and the sonic are both realms of communication, we might be able to find correlations between the two, that can be used to open up a new realm for dialogue. Moreover, if we search for possibilities to break with the now formalised act of communication in both realms, we can create new forms of dialogue and create a new conceptually endorsed form of synesthesia

[1] Eisenstein, Sergei. Film Form [and] The Film Sense; Two Complete Unabridged Works. New York: Meridian Books, 1968. p. 87
[2] Ox, Jack. “Introduction: Synesthetic Fusion in the Digital Age.” Leonardo, vol. 32, no. 5 (1999): p. 391-392. 1 May 2009.
[3] Föllmer, Golo. “Audio Art.” Media Art Net | Overview of Media Art | Audio. 15 Feb 2007. 1 May 2009.
[4] Ox, Jack. “Intersenses/Intermedia: A Theoretical Perspective.” Leonardo, vol. 34, no. 1 (2001): p. 47-48. 1 May 2009.
[5] Packer, Randall, and Ken Jordan. “Overture.” In: Multimedia: From Wagner to Virtual Reality. New York: Norton, 2001: p. xiii-xxxi.
[6] Boucher, Marc. “Kinetic Synaesthesia: Experiencing Dance in Multimedia Scenographies.” Kinetic Synaesthesia: Experiencing Dance in Multimedia Scenographies. Contemporary Aesthetics 2 (2004). 1 May 2009.
[7] A Colour Box (Uk: Len Lye, 1937). 1 May 2009.
[8] Ox, 1999
[9] Media Art Net. “Media Art Net | Cage, John.” John Cage «Variations V» 1 May 2009.
[10] Hertz, Paul. “Synesthetic Art-An Imaginary Number?.” Leonardo, vol. 32, no. 5 (1999): p. 399-404. 1 May 2009.
[11] Helfert, Heike. “Technological Constructions of Space–Time Aspects of perception.” Media Art Net | Overview of Media Art | Perception 15 Feb 2007. 1 May 2009.