Recommended
More Related Content
Recently uploaded
Recently uploaded (20)
Featured
Featured (20)
Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction
- 1. 29/09/2011 Rhythmic Blueprints Design and Evaluation of Rhythmic Interaction Cumhur Erkut and Antti Jylhä Aalto University, School of Electrical Engineering Department of Signal Processing and Acoustics MindTrek 2011 29/09/2011 Tampere, Finland Overview • Getting know each other • Sensitizing: interactive rhythms in various scales – Musical rhythms – Social rhythms • Elements of rhythm – Pulse, beat, meter, and tempo – Resonance, synchronization, entrainment • Design and evaluation models, and how we use them • Conclusions: Directions and Guidelines • Remember: http://blogs.aalto.fi/rhythmicity/ 29/09/2011 2 1
- 2. 29/09/2011 Michael H. Thaut, in Rhythm, Music, and the Brain, pp. 16-17 If we return briefly to the importance of temporal regulation for all our higher cognitive and motor functions, we may have very good reason to believe that rhythm in music, the element of temporal order, has a unique and profound influence on our perceptual processes related to cognition, affect, and motor function. Rhythm may enhance our brain operations through providing structure and anticipation in time. Rhythm may be one of the central processors to optimize our gestalt formation in the basic process of learning and perception. 29/09/2011 3 Rhythmic interaction • Time is of the essence • A natural human capability – Walking, hammering, talking… – Anticipation, mutual coordination • Examples – Personal Orchestra {Borchers:2004hx} – Virtual interactive humanoids {Nijholt:2008ty} – Percussion robot Haile {Weinberg:2006wl} – B-keeper {Robertson:2007tf} – Hand clap interface {Jylha:2009gq} 29/09/2011 4 2
- 3. 29/09/2011 Demo: Sonically Augmented Table and Rhythmic Interaction {Pesonen:2010ur} 29/09/2011 5 Elements of Musical Rhythms • Pulses: events in a “pulse train” with regular temporal spacing – The inter-pulse interval always the same – The basis of rhythm perception {Thaut:2005te}. • Beats: audible pulse markings – Sequenced events – May deviate from exact pulse timings in slight shifts 29/09/2011 6 3
- 4. 29/09/2011 Elements of Musical Rhythms • Tempo – Metric of (musical) rhythmic “speed” (rate of pulses/beats) – Inversely proportional to inter-pulse interval – Often measured as beats per minute (BPM) – In music, never completely stable • Meter – Defines the rhythmic structure in music (and vice versa) – Often expressed as beats per measure • E.g., 3/4, 4/4 – Relates to accentuation 29/09/2011 7 Related concepts • Vibration: Mechanical response of a body to an external stimuli • Frequency: Vibrations per second • Resonance: Tendency to vibrate at a certain frequency • Entrainment: A process between multiple bodies to align their rhythmic resonances • Synchronization: The quasi-stable state of entrainment 29/09/2011 9 4
- 5. 29/09/2011 Time-sensitive crowded scenes in movies 29/09/2011 10 Laws are simple: use in media? 29/09/2011 11 5
- 6. 29/09/2011 Sound synthesis, control, and hierarchical events 29/09/2011 12 ClaPD 29/09/2011 13 6
- 7. 29/09/2011 29/09/2011 14 Tutorial in a Nutshell … 29/09/2011 15 7
- 8. 29/09/2011 Hand clapping interface for sonic interactions (AM ’08 / CHI ’09) 29/09/2011 16 Applications • Potential in – Games and entertainment – Sound design – New HCI schemes • Three example cases 1. Hand-clap driven sampler 2. Controlling music tempo 3. Synchronizing a virtual audience 29/09/2011 17 8
- 9. 29/09/2011 System dataflow Tempo estimation Sound Hand clap Application User input detection Clap type identification Sonic feedback 29/09/2011 18 Music tempo control • The user claps to control the tempo of music – BPM of the user s clapping is mapped to that of the music 29/09/2011 19 9
- 10. 29/09/2011 Virtual audience • The user claps to synchronize a virtual audience with her clapping • The user can sync the audience with or without reference music • Interaction is immediate – The user is part of the clapping crowd 29/09/2011 20 Demo video • http://www.youtube.com/watch?v=7HLYGkayAGA 29/09/2011 21 10
- 11. 29/09/2011 Interface implementation with PD • Clap detection: [bonk~] (Puckette98)! • Tempo estimation: [rhythm_estimator] (Seppänen01)! • Combination: [clap_tracker]! 29/09/2011 22 Evaluation • Informal evaluation – 2 subjects tested the example applications – Interface was found easy to use – Both subjects found that the tempo of the virtual audience or the music drove their clapping • Some latency appears in the system – Mostly from buffering and computations – Not reported as disturbing by the subjects 29/09/2011 23 11
- 12. 29/09/2011 Negotiation • Perceived tempo affected subjects clapping – The user negotiates with the computer to set the tempo • Mutual coordination, ”dual-drive” – Process analogue: musical ensemble 29/09/2011 24 Explorations of rhythmic interaction with dancers {Erkut:2009wu} 29/09/2011 25 12
- 13. 29/09/2011 Design Blueprints: our version Based on Z. Obrenovic, J. Abascal, and D. Starcevic, “Universal cite{Erkut:2011ta} accessibility as a multimodal design issue,” Communications of the ACM, 29/09/2011 vol. 50, no. 5, pp. 83–88, 2007. 28 Rhythmic musical interface 29/09/2011 29 13
- 14. 29/09/2011 29/09/2011 30 iPalmas (AM ‘09) • An interactive Flamenco rhythm tutor for hand clapping cite{Jylha:2009uc} 29/09/2011 31 14
- 15. 29/09/2011 Flamenco 29/09/2011 32 Rhythm in Flamenco • Compas = meter in Flamenco • Usually 12-beat cycles – Accentuation on certain beats depending on style, e.g. bulerias: X - - X--X-X-X- • Lots of percussion – Footwork, hand clapping, instruments… • Focal performer leads – Improvisation requires communication 29/09/2011 33 15
- 16. 29/09/2011 Palmas • Supporting the compas • Palmero/-a • Two types – Hard and soft • Follow the focal performer http://www.youtube.com/watch?v=Yo38h7Wdc88 29/09/2011 34 Palmas skill requirements 1. Sordas and fuertes 2. Steady and accurate basic accompaniment 3. Decorative clapping 4. Starting to clap after silence 5. Reacting to tempo changes 6. Reacting to rhythmical cues 29/09/2011 35 16
- 17. 29/09/2011 System (1) • Flamenco palmas synthesis and tutor – Teaches the skills – Provides synthetic examples and accompaniment • User’s clapping as input • Auditory and visual feedback 29/09/2011 36 System (2) Clap User analysis Virtual GUI tutor Clap synthesis Visual feedback 29/09/2011 37 17
- 18. 29/09/2011 System (3) 29/09/2011 38 Synthetic palmas • ClaPD hand clap synthesis engine (Peltola et al. 2007) – Enveloped noise burst – Band-pass filter defines clap type – Coupled oscillator model for rhythmic interaction • Pre-defined palmas patterns set synthesis parameters • User and/or tutor set the tempo 29/09/2011 39 18
- 19. 29/09/2011 Analysis of clapping • Clap type – Classification by template matching ( [bonk~] ) • Accent detection – Loudness-based • Tempo – Probabilistic estimate based on inter-onset intervals • Tempo steadiness – Temporal variance • Correct accentuation – Comparison of user’s and tutor’s accent patterns 29/09/2011 40 Feedback (1) • Clapping sounds from the system – Synchrony, accents, tempo changes • Numeric feedback and sliders – Performance metrics • Visualization – Dancing circles – Transcription of the pattern 29/09/2011 41 19
- 20. 29/09/2011 Feedback (2) 29/09/2011 42 Prototype • Built on Pure Data (Pd) • Synthesis of virtual palmeros • Clap type training • Basic compas training – Tutor speeds up when the user gets better -> Tempo changes trained as well 29/09/2011 43 20
- 21. 29/09/2011 Interlude: evaluation Human System 29/09/2011 44 Interlude: evaluation Human Interaction System 29/09/2011 45 21
- 22. 29/09/2011 What can/should be evaluated • Human factors – Rhythmic capabilities • Perception • Production • System factors – I/O latency – Computational complexity • Interaction – Fluency, naturalness – Etc. • The whole triptych 29/09/2011 46 Methods of evaluation • Qualitative – Characterization of the capabilities/properties/phenomena • Understanding – Interviews, observations, etc. • Quantitative – Measurable quantities • For quantifying the qualitative attributes • Also can be used to derive qualitative results 29/09/2011 47 22
- 23. 29/09/2011 Human factors • Sensorimotor synchronization (SMS {Repp:2005tb}) – Synchronization between sensory stimulus and motor response – Often metronome-based evaluation or simple rhythmic tasks • “tapping to the beat” – Also, movement-to-music evaluation • How movement trajectories correlate with the rhythmic elements • Rhythmic capabilities – Production: e.g., rhythmic stability, accentuation – Perception: modal acuity – Some people can be actually “rhythm-deaf” (Phillips-Silver et al. 2011) 29/09/2011 48 System factors • I/O latency – Plays a role in rhythmic applications • In principle a constraint, but can be overcome in cyclic interactions • I/O modalities – Auditory, visual, haptic – Streaming (continous) vs. event-based (discrete) • Computational properties – Complexity, required processing power, memory, … 29/09/2011 49 23
- 24. 29/09/2011 Evaluating interaction • Qualitative studies • User tests – Often based on simple tasks • In-performance evaluation • Hybrid methods 29/09/2011 50 However… • … evaluation of individual components is valuable, but does not necessarily explain everything Wholesome evaluation needed 29/09/2011 51 24
- 25. 29/09/2011 Case example: iPalmas evaluation (CMJ ’11) • Subjective experiment AND objective description – Both qualitative and quantitative measures cite{Jylha2011:CMJ} 29/09/2011 52 iPalmas revisited Audio recording Metrics: - Tempo Detection and - Accent Text file log Hand claps analysis - Deviation User - Time stamp Audio Synthetic feedback Tempo Post-analysis hand claps, Visual Transcription, feedback circles, Tutor numeric feedback Audio Mostly 2011 Jylhä & Erkut 29/09/2011 53 25
- 26. 29/09/2011 Subjective experiment • To evaluate 1. Human factors 2. System 3. Interaction • Rhythmic tasks with realistic rhythmic patterns – Training, testing • Logging of measurable quantities • Observations throughout the experiment • Interviews, verbal comments • Post-experiment questionnaire 29/09/2011 54 Evaluation of the first iPalmas system • Performed to evaluate rhythmic interaction and the system (Jylhä et al. 2011) • 16 subjects • 4 patterns • 4 tutors (audio and audiovisual, adaptive and fixed tempo) • Training phase and test phase, recall phase Audio Mostly 2011 Jylhä & Erkut 29/09/2011 55 26
- 27. 29/09/2011 Evaluation of the first iPalmas system Audio Mostly 2011 Jylhä & Erkut 29/09/2011 56 Evaluation of the first iPalmas system • Auditory information (clapping) the key to learning and performing – The reverb disturbing to some • Transcription helpful • Circles pretty but not very helpful • Numeric FB useful only to some – Helped in “tuning in” to clapping the accents • Temporal variation in clapping – Tutor stops subject speeds up • Adaptive mode helps performance Audio Mostly 2011 Jylhä & Erkut 29/09/2011 57 27
- 28. 29/09/2011 Tempo speed-up and fluctuation Audio only tutor (steady) Audio only tutor (adaptive) 200 200 BPM BPM 180 180 Subject11 160 Subject1 160 0 20 40 60 0 20 40 60 Time (sec) Time (sec) Audiovisual tutor (steady) Audiovisual tutor (adaptive) 200 200 BPM BPM 180 180 160 160 0 20 40 60 0 20 40 60 Time (sec) Time (sec) cite{Jylha2011:CMJ} 29/09/2011 58 Quantitative findings • Average time to start clapping: 10 cycles (40 s) • Accentuation correctness – Audio-only: 68.1 % – Audiovisual: 73.7 % – Fixed tempo: 67.6 % – Adaptive tempo: 74.2 % • Indication: for accentuated beats, IOI slightly longer (344.7 ms) than for non-accentuated beats (343 ms) – More prominent with the adaptive tutor cite{Jylha2011:CMJ} 29/09/2011 59 28
- 29. 29/09/2011 Evaluation Blueprints: Summarize Qualitative, quantitative, and metrics cite{Erkut:2011ta} 29/09/2011 60 Direction1: SID http://en.wikipedia.org/wiki/Sonic_interaction_design 29/09/2011 61 29
- 30. 29/09/2011 Direction 2: Interaction Gestalts and Attributes User Interactive experience artifact • User experience • Artifact qualities properties Interaction gestalt {Lim:2009tw} 29/09/2011 62 Directions 1+2 Combined: New basic sonic interaction design http://www.room50.org/stefanodellemonache See, cite{Rocchesso:2009wi}, 29/09/2011 and cite{Franinovic:2009wl} 63 30
- 31. 29/09/2011 Other directions • Social rhythms: H. Lefebvre, “Rhythmanalysis: Space, Time and Everyday Life,” Book, pp. 1–129, 2004 => CSCW => Social games • Rhythms and emotions: Thaut’05, but also recent CHI papers, e.g. cite{Epp:2011hz} • Gamification! • What else? 29/09/2011 64 References • C. Epp, M. Lippold, and R. L. Mandryk, “Identifying emotional states using keystroke dynamics,” in CHI’11, Vancouver, BC, Canada, 2011, pp. 715–724. • C. Erkut, A. Jylhä, and R. Discioglu, “A structured design and evaluation model with application to rhythmic interaction displays,” in Proc. New Interfaces for Musical Expression (NIME), Oslo, Norway, 2011, pp. 477–480. • C. Erkut, A. Jylhä, and I. Ekman, “Recent advances in exploring self-induced sonic interactions in the context of performing arts,” in Intl. Workshop on Haptic and Audio Interaction Design, 2009, pp. 1–2. • K. Franinovic, “Toward Basic Interaction Design,” available online at http://tdd.elisava.net/coleccion/25/franinovic-en • A. Jylhä and C. Erkut, “A hand clap interface for sonic interaction with the computer,” CHI-EA, Apr. 2009. • A. Jylhä, C. Erkut, I. Ekman, and K. Tahiroglu, “iPalmas - An interactive flamenco rhythm machine,” Proc. Audio Mostly, pp. 1–2, May. 2009. • A. Jylhä, I. Ekman, C. Erkut, and K. Tahiroglu, “Design and Evaluation of Rhythmic Interaction with an Interactive Tutoring System,” Computer Music Journal, vol. 35, no. 2, pp. 36–48. 2011. 29/09/2011 68 31
- 32. 29/09/2011 References • Y.-K. Lim, E. Stolterman, H. Jung, and J. Donaldson, “Interaction gestalt and the design of aesthetic interactions,” Proc.. Conf. Designing Pleasurable Products and Interfaces, pp. 239–254, 2007. • M. Pesonen, Sonically Augmented Table and Rhythmic Interaction, Master’s thesis, Aalto Univerity, School of Electrical Engineering, 2010. • B. Repp, “Sensorimotor synchronization: A review of the tapping literature,” Psychonomic Bulletin and Review, vol. 12, no. 6, pp. 969–992, 2005. • A. Robertson and M. Plumbley, “B-Keeper: A Beat-Tracker for Live Performance,” Proc. NIME, pp. 234–237, 2007. • D. Rocchesso, P. Polotti, and S. D. Delle Monache, “Designing Continuous Sonic Interaction,” Intl. J. Design, vol. 3, no. 3, pp. 13–25, May. 2009. • M. Thaut, Rhythm, Music, and the Brain. New York, NY, USA: Routledge, 2005. • G. Weinberg and S. Driscoll, “Robot-human interaction with an anthropomorphic percussionist,” CHI '06, Apr. 2006. 29/09/2011 69 Discussion • (Edward T) Hall's studies (on cultural rhythms) and (Saul) Greenberg's work on social rhythms (awareness) • Should the tutorial be called "interaction with rhythms" instead of rhythmic interaction? • Synesthesia as a resource? • Behavioral change: negotiation already used in game design (transfer of adaptation) • Adaptive coaching • Language learning and prononciation 29/09/2011 70 32