Music and Gesture

My thesis investigates the relationships between visual and auditory modalities through the medium of North Indian Classical Music. Intermodal binding and unitization can be studied via natural body gestures and hand movements in improvised music. In this thesis, we present experiments to analyze the dependance relationships between musical parameters and body gestures through data from pitch analysis, motion capture and video processing. We provide evidence for a cognitive model for music-motion binding. We investigate the typology and categorization of these gestures and perception of monodic musical shapes amongst musicians and non musicians. We also implement models to convert these gestures back into sound, using rule based grammars derived from raga models.

Analyzing Gestures and Motion Metaphors


Publication: Kelkar, T., & Indurkhya, B. (2014, July). Typology of gestures and motion metaphors in improvisation of Hindustani vocal music. In Proceedings of the Sixth Conference of the Society for Gesture Studies: Gesture in Interaction. University of California, San Diego.

Abstract: Singers of Hindustani Classical Music (HCM) are known to teach and sing with elaborate hand gestures displaying melodic contours, rhythmic stresses and other features. These gestures embody a mapping of musical phrases to space, motion and movement. Although the associations between these hand gestures and mental imagery are documented in various ancient and modern treatises, the mapping between the gesture types and their musical content has not been explicitly studied. We explore the chhota-khyal musical form by analysing gestural language in each of its temporal phases. A chhota-khyal is a set of two couplets (sthyayi, antara) involving a fixed set of musical and poetic rules. It can be composed and sung within any raga. We conducted a study of three professional singers of HCM, each performing 5-8 minute khyals in four selected ragas. The rendition of khyal involves three phases: elaborating the raga by singing on vowels (alaap); tala-bound composition (ciz, bolalap), improvised with theme and variations; and a fast section spanning the whole range of raga (taan, layakari). The transition between different phases of khyal is gradual, and is marked by fixed musical events that a singer must include during the performance.

Performer Gestures
Video Analysis
Audio Analysis

Each musical event in khyal was analysed for all renditions to observe the gestural commonalities pertaining to these events. We analysed the music accompanying gestures in two ways: 1) computer-based motion detection and pitch data extraction, and 2) observational study of palm shapes and their association with motion- metaphors for imaginary sound objects. By analysing pivotal points in khyal across different performers, we propose a typology of gestures used for specific ornamentations in khyal singing. We notated palm shapes and identified gestures in the typology using the schemes described. We interviewed the singers regarding their gesturing of musical contours to understand the cognitive construction and awareness-level of these gestures. We observed that semantic content of khyal texts plays a critical role in guiding the mental imagery of the performers. We did a tripartite analysis of these gestures by comparing these results with insights from other controlled experiments on a) prosodic content and gestural stresses in spoken khyal texts; b) abstract presenting gestures in speech; and c) granularity of spatial representations of purely musical melodic contours in listeners.

We compared the spatial coordinates of hand movements with the pitch data for each musical event in khyal. As each phrase is a unique gesture unit, the gesticulation continues throughout the duration of the musical phrase, displaying different components of a gesture phrase. The body only comes to rest once the musical phrase is complete. This neutral position suggests phrase level information in the music, illustrating grouping mechanisms for melodies in HCM to be understood via gestural phrases. Despite the differences in gestural languages of gharanas, performers and individual renditions, we have illustrated the features of motion metaphors of melody in HCM. The results of this study can be used to generate clearer mappings of intuitive gesture-sound relationships in Hindustani Classical singing, which can be applied to improve pedagogical methods.

Sound Tracings

pdf      Try Sound Tracing:

Publication: Kelkar, T., & Indurkhya, B. (2014 June). A motion-based approach to analysis, notation and creation of Hindustani Music. Proceedings of xCoAx2014: Computation Communication Aesthetics and X. Universidade do Porto, Porto.

Abstract: Performers of Hindustani Classical Music depend heavily on complex models of motion and movement to elaborate melodic ideas through hand gestures and motion metaphors. Despite advances in computational modeling of grammars that govern the elaboration of a raga, these systems run into difficulties because of the nature of pitch systems in computer programs and in performance. We elaborate the problems with trying to obtain the ideas in a flexible-pitch scheme like HCM through the means of a fixed-pitch scheme-like notation and computer music generation. In this paper, we present some experiments to study the effectiveness of a graphical notation scheme in HCM, sound tracing study and an analysis of the terminology used for ornaments – through which to understand motion in HCM. We plan to analyze them computationally to develop a formalism that would be more suitable to the nuances of HCM than the present schemes.

Interfaces to draw music


Publication: Roy, U., Kelkar, T., & Indurkhya, B. (2014, June). TrAP: An Interactive System to Generate Valid Raga Phrases from Sound-Tracings. In Proceedings of the 2014 conference on New interfaces for musical expression. Goldsmiths, University of London, London.

Abstract: We propose a new musical interface, TrAP (TRace-A-Phrase) for generating phrases of Hindustani Classical Music (HCM). In this system the user traces melodic phrases on a tablet interface to create phrases in a raga. We begin by analyzing tracings drawn by 28 participants, and train a classifier to categorize them into one of four melodic categories from the theory of Hindustani Music. Then we create a model based on note transitions from the raga grammar for the notes used in the singable octaves in HCM. Upon being given a new tracing, the system segments the tracing and computes a final phrase that best approximates the tracing.

Tonal Graphs and Music Generation


Publication (Accepted): Kelkar, T. (2015, January). Colour Wheel for Music: Graphical visualization of analogous relationships of Raags. In Proceedings of International Conference of New Musical Concepts. Treviso, Italy.

Abstract: A raag is a melodic structure with grammatical rules for improvised phrases. Raags define tonal relationships between various notes. There are hundreds of raags in number, all having unique descriptors. In this paper, we visualize tonal spaces of raag by creating a graph with a force directed layout, and a propose mapping of colour to this tonal space. We derive the graph for the visualization by parameterization of raags as described in the theory of HCM. We compare a radial layout for these tonal spaces to a colour harmony profile and explain some cross raag relationships using the methods used to derive colour schemes. We discuss the affective implications and empirical verifications of this model.
This model has potential applications in sonification and tone-color mapping. The implications of a layout for tonal music is also useful for deriving implicit harmonic relationships. Navigate the graph as below:

Music Education Applications

Virtual Lab for Music

Publication (Under Review): Kelkar, T., Ray, A., Choppella, V. (2015, January). SangeetKosh: An Open Web Platform for Hindustani Music Education. In Proceedings of International Conference on Advanced Learning Technologies. Hualien, Taiwan.

Abstract: In this paper, we present the construction and architecture of SangeetKosh: an open platform for studying North Indian or Hindustani Classical Music (HCM) on the web. This platform works on a three faceted architecture - providing musical experiments, quizzes and drills; a platform to host and find repositories of music; and a way to semantically link and annotate pre-existing musical content on the web. We present this architecture as an open framework that can be adapted to other music learning environments. We describe the ways in which this application caters to the cultural and social problems of music students of HCM. This application is also structured to be sensitive to local context, with easy ways to translate into local languages, and make the platform accessible to a larger demographic. We present some user experiments about the usage and benefits of SangeetKosh.