Neurocognitive mechanisms for perception-action coupling
Music is an exceptionally rich model to explore the interplay between sensory (auditory and visual) and motor brain systems in the context of social interactions such as imitation or inter-personal coordination.
We address the coupling of perception and action at two levels: within and between human brains. Intra-cortical coupling (within brain) is related to predictive brain processes that support adaptation and anticipation of events in the surrounding environment. Inter-cortical coupling (between
brains) is related to the relationship between ongoing neural processes in separate brains while they share sensorimotor information. Intra- and inter-cortical coupling accounts for how multiple individuals entrain via reciprocal prediction, adaptation and shared representations to achieve shared goals.
To explore these phenomena, we use two neurophysiological approaches: brain recording and brain stimulation. These methods are complementary in that brain recording techniques—e.g., electroencephalography (EEG)—are used to map neural processes and to identify neural signatures that
are behaviourally relevant, while brain stimulation techniques—transcranial magnetic stimulation (TMS) and alternating current stimulation (tACS)—are used to interfere with these processes to infer their causal effects upon behaviour. At MARCS Institute, we employ these techniques in parallel
(e.g. dual-EEG) while pairs of individuals are involved in real-time social interactions.
We aim to understand the biological signatures of social cognition by combining these neurophysiological approaches with sensorimotor (i.e. behavioural) measures (see Sensorimotor synchronisation, entrainment,
and rhythm perception and production) as well as assessments of individual differences related to personality or ensemble skills (see Developing and modelling ensemble skills and joint action).
Sensorimotor synchronisation, entrainment, and rhythm perception and production
The ability to synchronise actions with rhythms in the environment is a fundamental human skill that is expressed exquisitely in the way in which humans interact in musical
contexts. Ensemble musicians synchronise the sounds of co-performers. Orchestral musicians synchronise with the gestures of a conductor. Dancers synchronise their body movements with respect to one another and the musical accompaniment. All these situations are characterised by highly refined forms of
coupling between rhythmic movement and external rhythms via a process of entrainment.
Entrainment in musical contexts is underpinned by psychological processes that allow an individual to perceive the rhythm of an external sequence of events, to anticipate the timing of upcoming events based on this rhythm, to produce rhythmic movement, and to coordinate these produced movements with
events in the external sequence.
At MARCS Institute, we study these processes using a variety of experimental tasks, ranging from naturalistic interpersonal coordination tasks to controlled laboratory tasks. The naturalistic tasks include real musical ensemble performance (e.g., piano duos) and rudimentary forms of dyadic sensorimotor
synchronization (joint finger-tapping). The controlled laboratory tasks require an individual to synchronize simple movements (finger taps, drum strikes, or limb oscillations) with repetitive events in auditory, visual, or multimodal (e.g., audio-visual) pacing sequences. Synchrony is assessed by a variety
of techniques applied to data from discrete events such as finger taps and continuous measures of body movements recorded with motion capture systems.
We are also studying the processes by which individuals from one culture learn the rhythms that characterise the music of another culture. The specific focus of this stream of research is on how adults develop the capacity to perceive temporal relations and to generate expectations in the context of
new, culturally unfamiliar music. The implications of this work extend beyond gaining knowledge about basic psychological mechanisms, as learning to appreciate the music of other cultures may facilitate intercultural respect and understanding by enabling individuals from different cultures to interact
with one another through musical activities.
Developing and modelling ensemble skills and joint action
Musical ensemble performance is a social art form that places exceptional demands upon the mental and physical capacities of co-performers. A particularly remarkable feature of
ensemble performance is the balance that individuals are able to achieve between temporal precision and flexibility in interpersonal coordination. Such coordination is typically a highly creative affair involving different body parts, complementary roles played by different individuals, and the need
to accommodate changing cognitive, motor, affective, and social demands during performance. Yet competent co-performers are able synchronize their actions to within several tens of milliseconds. When considered as a microcosm of human interaction, ensemble performance is therefore a fruitful domain in
which to study the dynamics of interpersonal coordination and nonverbal communication under controlled conditions.
Our research on ensemble performance addresses strategies that are used to prepare for performance, psychological mechanisms that underpin interpersonal coordination during performance, and intrinsic (personal) and extrinsic (contextual) factors that affect the implementation of these strategies and
The theoretical framework guiding this research is centred on three core cognitive-motor skills that allow a performer to anticipate, attend, and adapt to the actions of co-performers in real time. The framework also permits us to assess how these three ensemble skills interact with one another, as
well as with social-psychological factors (e.g., variables related to personality) and the performer's knowledge about the music and familiarity with co-performers.
Research based on this framework is being conducted with performers in diverse ensembles, including piano duos, choral groups, jazz combos, and gamelan musicians. Interpersonal coordination within such groups is quantified by analysing acoustic recordings, as well as body movements recorded with motion
capture systems. Computational modelling techniques are used to investigate the relationship between these measures of interpersonal coordination and basic ensemble skills assessed in laboratory tasks.
Goals of this program of research include understanding the dynamics of leader-follower relations and individual differences in ensemble skills. Elucidating the causes of these individual differences will potentially inform the assessment and treatment of clinical conditions that affect movement timing
and social interaction.
Perceiving loudness and its relation to affect
We investigate fundamental dimensions of sound and music — loudness, pitch, timbre and contour, duration and tempo. We want to understand how these features function as dynamic
cues to motion and danger. And we also investigate how they elicit emotional reactions in people communicating through music and speech.
For example, the Music Cognition and Action group interrogates the implications of the observations that sonic intensity increases signal the approach of everyday objects, and that speech involves controlled intensity changes. Our work suggests that in music world-wide, intensity control is the main
Currently we are specifically investigating the influence of musical acoustic intensity on perception of loudness, change, affect, and of performers' effort level.
We also analyse patterns of acoustic intensity in a range of music, and speech and environmental sounds. In the light of the perceptual studies, such patterns will enhance understanding of biologically and musically important perception mechanisms. They will also aid development of effective sonic symbols
for information transfer, and commercial applications in music information retrieval and auditory data mining.
Improvising and creating dance and music
Real-time music and dance creation and the psychological, communicative and social interactions they entail are under investigation. We understand improvisation as a
highly cultivated (that is, expertise-based) process realized within performance in most societies. But we also understand it as a process much more widely (for example, by choreographers, contact dancers, and composers) as part of longer term creative methods. Members of the Music Cognition and Action
group have worked extensively in the past on the musicology of improvisation, and on the nature of dance interactions during production and performance. The Music Cognition and Action group now studies cognitive, distributed and neurophysiological processes in improvisers in action.
In relation to music creation, this includes studies on solo and duo improvisers and their interactions in terms of event timing, leadership, cognitive processes, and neurophysiological responses including skin conductance and brain coupling. In relation to dance and more broadly, it particularly concerns
learning and memory for sequences, and co-construction of novel ideas.
The studies use information from musical interface digital flow, acoustic and video data, performance timing and other outputs, together with psychophysiological data. These data are subjected coordinately to computational analysis and modelling.
Associative learning including action and dance sequences
One of the fundamental ways in which we make sense of the world and how we infer causation and meaning is through the reliable association of events and stimuli in time and space.
For example, if we regularly get a static electrical shock from the door handle of our car we may come to feel anxious as we reach for the door handle and learn to avoid touching the door handle with our bare hand. Learning by association is ubiquitous; it has been shown in honey bees, sea snails, dogs
and humans. It is also wide reaching in that it can influence perception, reflexive responses, emotional reactions as well as voluntary actions.
Currently we are exploring the role of cognition in associative learning of reflexive responses and emotional reactions to determine the extent to which how we interpret stimulus pairings influences how we learn about those stimulus pairings.
The flip side of learning is memory. Many studies have scrutinised human memory for static, visual stimuli such as digits, words, sentences, pictures and spatial locations. Relatively little is known about memory for dynamic, complex and generally non-verbal sequences of rhythmic bodily actions such
as those that characterise dance and gymnastics.
The way in which such material is learned and memorised, and the aptitude for such learning are our targets at MARCS Institute. This is important not only for training and development in sport and the performing arts, but also to enhance the explanatory power of existing models of human memory.