Heart Response to Harmonic Music Interval Stimuli Via Deep Learning Structures

Ennio Idrobo-Ávila, Humberto Loaiza-Correa, Flavio Muñoz-Bolaños, Leon van Noorden, Rubiel Vargas-Cañas

Abstract


The effect of music on the heart is reflected in variables such as heart rate and electrocardiographic (ECG) signals. ECG is a record of heart electrical activity and is a useful tool in diagnosing various cardiopathies. Artificial intelligence techniques have recently been implemented to analyze ECG and RR-interval data and are used thus in the present study to examine the influence on the heart of harmonic musical intervals and colored noise. Harmonic intervals were chosen because of their emotional response, while noise has been linked to positive responses such as improved sleep quality. A deep learning system was implemented, employing the ResNet-18 and GoogLeNet pre-trained networks to discriminate 31 different classes of ECG and RR-interval responses to the sound stimuli. Following an exploratory approach, deep learning was selected as an alternative to traditional analysis with the expectation that it could be incorporated into future music perception research. Classification revealed the ability of the implemented system to demonstrate heart response to the stimuli. ECG signals performed best, with 97% accuracy and Matthew’s coefficient of 0.97, while RR-interval achieved a 93% accuracy and Matthews coefficient of 0.93, suggesting that the considered stimuli of harmonic musical intervals and noise produced different responses in the heart. Moreover, the Matthews coefficient values above 0.7 and close to 1 imply a correlation between the two types of stimuli and the heart response, as measured by ECG and RR-interval signals.

Keywords


Electrocardiographic signals; GoogLeNet; music; noise; ResNet-18; transfer learning.

Full Text:

PDF

References


A. Capucci, New Concepts in ECG Interpretation. 2019.

S. Koelsch and L. Jäncke, “Music and the heart,†Eur. Heart J., p. ehv430, 2015.

H.-J. Trappe, “Music and heart: What is verified, what is not, what’s new? ,†Kardiologe, vol. 11, no. 6, pp. 486–496, 2017, doi: 10.1007/s12181-017-0192-7.

K. Kantono, N. Hamid, D. Shepherd, Y. H. T. Lin, S. Skiredj, and B. T. Carr, “Emotional and electrophysiological measures correlate to flavour perception in the presence of music,†Physiol. Behav., vol. 199, pp. 154–164, 2019, doi: 10.1016/j.physbeh.2018.11.012.

J. K. Jain and R. Maheshwari, “Effect of indian classical music and pop music on heart rate variability: A comparative study,†Indian J. Community Heal., vol. 31, no. 4, pp. 556–560, 2019.

M. Erfanian, A. J. Mitchell, J. Kang, and F. Aletta, “The psychophysiological implications of soundscape: A systematic review of empirical literature and a research agenda,†Int. J. Environ. Res. Public Health, vol. 16, no. 19, Sep. 2019, doi: 10.3390/ijerph16193533.

A. W. C. Yuen and J. W. Sander, “Can natural ways to stimulate the vagus nerve improve seizure control?,†Epilepsy Behav., vol. 67, pp. 105–110, 2017, doi: 10.1016/j.yebeh.2016.10.039.

M. Warth, J. Kessler, T. K. Hillecke, and H. J. Bardenheuer, “Trajectories of Terminally Ill Patients’ Cardiovascular Response to Receptive Music Therapy in Palliative Care,†J. Pain Symptom Manage., vol. 52, no. 2, pp. 196–204, 2016, doi: 10.1016/j.jpainsymman.2016.01.008.

S. Palma, M. Keilani, T. Hasenoehrl, and R. Crevenna, “Impact of supportive therapy modalities on heart rate variability in cancer patients–a systematic review,†Disabil. Rehabil., vol. 42, no. 1, pp. 36–43, Jan. 2020, doi: 10.1080/09638288.2018.1514664.

X.-Y. Zhang, J.-J. Li, H.-T. Lu, W.-J. Teng, and S.-H. Liu, “Positive effects of music therapist’s selected auditory stimulation on the autonomic nervous system of patients with disorder of consciousness: a randomized controlled trial,†Neural Regen. Res., vol. 16, no. 7, pp. 1266–1272, 2021, doi: 10.4103/1673-5374.301021.

S. Sieciński and P. Kostka, “Influence of Music on HRV Indices Derived from ECG and SCG,†International Scientific Conference Advances in Applied Biomechanics, AAB 2020, vol. 1223. Springer Science and Business Media Deutschland GmbH, Faculty of Biomedical Engineering, Department of Biosensors and Processing of Biomedical Signals, Silesian University of Technology, Roosevelta 40, Zabrze, 41-800, Poland, pp. 381–389, 2021, doi: 10.1007/978-3-030-52180-6_39.

J. Sa de Almeida et al., “Music enhances structural maturation of emotional processing neural pathways in very preterm infants,†Neuroimage, vol. 207, 2020, doi: 10.1016/j.neuroimage.2019.116391.

C.-C. Hsu, S.-R. Chen, P.-H. Lee, and P.-C. Lin, “The Effect of Music Listening on Pain, Heart Rate Variability, and Range of Motion in Older Adults After Total Knee Replacement,†Clin. Nurs. Res., vol. 28, no. 5, pp. 529–547, 2019, doi: 10.1177/1054773817749108.

T. McPherson, D. Berger, S. Alagapan, and F. Fröhlich, “Active and Passive Rhythmic Music Therapy Interventions Differentially Modulate Sympathetic Autonomic Nervous System Activity,†J. Music Ther., vol. 56, no. 3, pp. 240–264, 2019, doi: 10.1093/jmt/thz007.

V. Moaiyed, M. Firoozabadi, and M. Khezri, “Recognition of Music-Induced Emotions Based on Heart-Brain Connectivity,†2018, doi: 10.1109/ICBME.2017.8430259.

D. Najumnissa, P. Alagumariappan, A. Bakiya, and M. S. Ali, “Analysis on the effect of ECG signals while listening to different genres of music,†2019, doi: 10.1109/ICACCP.2019.8882925.

G. Yadu, D. Panigrahi, S. K. Nayak, A. Dey, and K. Pal, “Wavelet Packet Analysis of ECG signals to Understand the Effect of a Motivating Song on Heart of Indian Male Volunteers,†in Expert System Techniques in Biomedical Science Practice, National Institute of Technology, Rourkela, India: IGI Global, 2018, pp. 168–192.

S. Paul, G. Yadu, S. K. Nayak, A. Dey, and K. Pal, “Recurrence quantification analysis of electrocardiogram signals to recognize the effect of a motivational song on the cardiac electrophysiology,†vol. 575. Springer Verlag, Department of Biotechnology and Medical Engineering, National Institute of Technology, Rourkela, 769008, India, pp. 165–172, 2020, doi: 10.1007/978-981-13-8687-9_16.

K. Wang, W. Wen, and G.-Y. Liu, “The autonomic nervous mechanism of music therapy for dental anxiety,†2017, pp. 289–292, doi: 10.1109/ICCWAMTIP.2016.8079858.

A. Mincholé, J. Camps, A. Lyon, and B. Rodríguez, “Machine learning in the electrocardiogram,†J. Electrocardiol., 2019, doi: 10.1016/j.jelectrocard.2019.08.008.

S. Parvaneh, J. Rubin, S. Babaeizadeh, and M. Xu-Wilson, “Cardiac arrhythmia detection using deep learning: A review,†J. Electrocardiol., 2019, doi: 10.1016/j.jelectrocard.2019.08.004.

T.-H. Nguyen, T.-N. Nguyen, and T.-T. Nguyen, “A deep learning framework for heart disease classification in an IoTs-based system,†Intelligent Systems Reference Library, vol. 165. Springer Science and Business Media Deutschland GmbH, Department of Industrial Electronic-Biomedical Engineering, Ho Chi Minh City University of Technology and Education, Ho Chi Minh City, Viet Nam, pp. 217–244, 2020, doi: 10.1007/978-3-030-23983-1_9.

M. Salem, S. Taheri, and J. Yuan, “ECG arrhythmia classification using transfer learning from 2-dimensional deep CNN features,†in 2018 IEEE Biomedical Circuits and Systems Conference (BioCAS), 2018, pp. 1–4.

O. Faust, Y. Hagiwara, T. J. Hong, O. S. Lih, and U. R. Acharya, “Deep learning for healthcare applications based on physiological signals: A review,†Comput. Methods Programs Biomed., vol. 161, pp. 1–13, 2018, doi: 10.1016/j.cmpb.2018.04.005.

L. Chen, G. Xu, S. Zhang, J. Kuang, and L. Hao, “Transfer Learning for Electrocardiogram Classification Under Small Dataset BT - Machine Learning and Medical Engineering for Cardiovascular Health and Intravascular Imaging and Computer Assisted Stenting,†2019, pp. 45–54.

J. L. Hagad, K. ichi Fukui, and M. Numao, “Modelling Naturalistic Work Stress Using Spectral HRV Representations and Deep Learning,†in Advances in Intelligent Systems and Computing, 2020, vol. 1128 AISC, pp. 267–277, doi: 10.1007/978-3-030-39878-1_24.

S.-H. Huang, B.-L. Chuang, Y.-H. Lin, C.-S. Hung, and H.-P. Ma, “A congestive heart failure detection system via multi-input deep learning networks,†2019, doi: 10.1109/GLOBECOM38437.2019.9013460.

L. Wang and X. Zhou, “Detection of congestive heart failure based on LSTM-based deep network via short-term RR intervals,†Sensors (Switzerland), vol. 19, no. 7, 2019, doi: 10.3390/s19071502.

Y. Li et al., “Combining Convolutional Neural Network and Distance Distribution Matrix for Identification of Congestive Heart Failure,†IEEE Access, vol. 6, pp. 39734–39744, 2018, doi: 10.1109/ACCESS.2018.2855420.

M. Merino-Monge, I. M. Gómez-González, J. A. Castro-García, A. J. Molina-Cantero, and R. Quesada-Tabares, “A preliminary study about the music influence on EEG and ECG signals,†2018, pp. 100–106.

Y.-L. Hsu, J.-S. Wang, W.-C. Chiang, and C.-H. Hung, “Automatic ECG-Based Emotion Recognition in Music Listening,†IEEE Trans. Affect. Comput., vol. 11, no. 1, pp. 85–99, 2020, doi: 10.1109/TAFFC.2017.2781732.

M. A. Alves, D. M. Garner, J. A. T. do Amaral, F. R. Oliveira, and V. E. Valenti, “The effects of musical auditory stimulation on heart rate autonomic responses to driving: A prospective randomized case-control pilot study,†Complement. Ther. Med., vol. 46, pp. 158–164, 2019, doi: 10.1016/j.ctim.2019.08.006.

A. Ranger et al., “Physiological and emotional effects of pentatonic live music played for preterm neonates and their mothers in the Newborn Intensive Care Unit: A randomized controlled trial,†Complement. Ther. Med., vol. 41, pp. 240–246, 2018, doi: 10.1016/j.ctim.2018.07.009.

C. Gäbel, N. Garrido, J. Koenig, T. K. Hillecke, and M. Warth, “Effects of Monochord Music on Heart Rate Variability and Self-Reports of Relaxation in Healthy Adults,†Complement. Med. Res., vol. 24, no. 2, pp. 97–103, 2017, doi: 10.1159/000455133.

M. J. Mollakazemi, D. Biswal, J. Evans, and A. Patwardhan, “Eigen Decomposition of Cardiac Synchronous EEGs for Investigation of Neural Effects of Tempo and Cognition of Songs,†2018, vol. 2018-July, pp. 2402–2405, doi: 10.1109/EMBC.2018.8512806.

J. Kim, C. A. Strohbach, and D. H. Wedell, “Effects of manipulating the tempo of popular songs on behavioral and physiological responses,†Psychol. Music, vol. 47, no. 3, pp. 392–406, 2019, doi: 10.1177/0305735618754688.

B. Bretherton, J. Deuchars, and W. L. Windsor, “The Effects of Controlled Tempo Manipulations on Cardiovascular Autonomic Function,†Music Sci., vol. 2, p. 205920431985828, Jan. 2019, doi: 10.1177/2059204319858281.

D. Biswal, M. J. Mollakazemi, S. Thyagarajan, J. Evans, and A. Patwardhan, “Baroreflex Sensitivity during Listening to Music Computed from Time Domain Sequences and Frequency Domain Transfer Function,†2018, vol. 2018-July, pp. 2776–2779, doi: 10.1109/EMBC.2018.8512861.

S. Sharma et al., “Indian classical music with incremental variation in tempo and octave promotes better anxiety reduction and controlled mind wandering—A randomised controlled EEG study,†Explore, 2020, doi: 10.1016/j.explore.2020.02.013.

M. Orini, F. Al-Amodi, S. Koelsch, and R. Bailón, “The Effect of Emotional Valence on Ventricular Repolarization Dynamics Is Mediated by Heart Rate Variability: A Study of QT Variability and Music-Induced Emotions,†Front. Physiol., vol. 10, 2019, doi: 10.3389/fphys.2019.01465.

S. Ishimitsu, K. Oue, A. Yamamoto, and Y. Date, “Sound quality evaluation using heart rate variability analysis,†2017.

B. Benward and M. Saker, Music in theory and practice, Volume I, 8th ed. New York: McGraw-Hill, 2009.

C. J. Murrock and P. A. Higgins, “The theory of music, mood and movement to improve health outcomes: Discussion paper,†J. Adv. Nurs., 2009, doi: 10.1111/j.1365-2648.2009.05108.x.

M. R. Ebben, P. Yan, and A. C. Krieger, “The effects of white noise on sleep and duration in individuals living in a high noise environment in New York City,†Sleep Med., vol. 83, pp. 256–259, 2021, doi: 10.1016/j.sleep.2021.03.031.

T. A. Pickens, S. P. Khan, and D. J. Berlau, “White noise as a possible therapeutic option for children with ADHD,†Complement. Ther. Med., vol. 42, pp. 151–155, 2019, doi: 10.1016/j.ctim.2018.11.012.

Y.-A. Chiou et al., “Electrocardiogram lead selection for intelligent screening of patients with systolic heart failure,†Sci. Rep., vol. 11, no. 1, 2021, doi: 10.1038/s41598-021-81374-6.

M. Marei, S. E. Zaatari, and W. Li, “Transfer learning enabled convolutional neural networks for estimating health state of cutting tools,†Robot. Comput. Integr. Manuf., vol. 71, 2021, doi: 10.1016/j.rcim.2021.102145.

A. Yildiz, H. Zan, and S. Said, “Classification and analysis of epileptic EEG recordings using convolutional neural network and class activation mapping,†Biomed. Signal Process. Control, vol. 68, 2021, doi: 10.1016/j.bspc.2021.102720.

A. Canziani, A. Paszke, and E. Culurciello, “An analysis of deep neural network models for practical applications,†arXiv Prepr. arXiv1605.07678, 2016.

T. Li and M. Zhou, “ECG Classification Using Wavelet Packet Entropy and Random Forests,†Entropy , vol. 18, no. 8. 2016, doi: 10.3390/e18080285.

M. Takahashi and Y. Takai, “Low-temperature growth of YBa2Cu3Ox by pulsed laser deposition,†Supercond. Sci. Technol., vol. 11, no. 3, pp. 265–269, 1998, doi: 10.1088/0953-2048/11/3/002.

H. Moeinzadeh et al., “Wctecgdb: A 12-lead electrocardiography dataset recorded simultaneously with raw exploring electrodes’ potential directly referred to the right leg,†Sensors (Switzerland), vol. 20, no. 11, pp. 1–12, 2020, doi: 10.3390/s20113275.

A. Azizi and P. Ghafoorpoor Yazdi, “Introduction to Noise and its Applications BT - Computer-Based Analysis of the Stochastic Stability of Mechanical Structures Driven by White and Colored Noise,†A. Azizi and P. Ghafoorpoor Yazdi, Eds. Singapore: Springer Singapore, 2019, pp. 13–23.

D. Self, “Small signal audio design.†Focal Press, Oxford, U.K.; Burlington, Mass., 2010.

R. Kher, “Signal Processing Techniques for Removing Noise from ECG Signals,†J. Biomed. Eng. Res., vol. 3, 2019.

C. Wang, S. Yang, X. Tang, and B. Li, “A 12-Lead ECG Arrhythmia Classification Method Based on 1D Densely Connected CNN,†Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11794 LNCS. Springer, Chengdu Spaceon Electronics Co., Ltd., Chengdu, China, pp. 72–79, 2019, doi: 10.1007/978-3-030-33327-0_9.

M. Chala, B. Nsiri, M. H. El yousfi Alaoui, A. Soulaymani, A. Mokhtari, and B. Benaji, “An automatic retinal vessel segmentation approach based on Convolutional Neural Networks,†Expert Syst. Appl., vol. 184, 2021, doi: 10.1016/j.eswa.2021.115459.

O. Archangelidi, M. Pujades-Rodriguez, A. Timmis, X. Jouven, S. Denaxas, and H. Hemingway, “Clinically recorded heart rate and incidence of 12 coronary, cardiac, cerebrovascular and peripheral arterial diseases in 233,970 men and women: A linked electronic health record study,†Eur. J. Prev. Cardiol., vol. 25, no. 14, pp. 1485–1495, 2018, doi: 10.1177/2047487318785228.

S. Kuila, N. Dhanda, and S. Joardar, “ECG signal classification for arrhythmia detection using DEA and ELM,†J. Theor. Appl. Inf. Technol., vol. 99, no. 14, pp. 3403–3416, 2021.

F. Shaffer and J. P. Ginsberg, “An Overview of Heart Rate Variability Metrics and Norms,†Front. Public Heal., vol. 5, p. 258, Sep. 2017, doi: 10.3389/fpubh.2017.00258.

G. Zimatore et al., “Recurrence quantification analysis of heart rate variability during continuous incremental exercise test in obese subjects,†Chaos, vol. 30, no. 3, 2020, doi: 10.1063/1.5140455.

W. Guo, C. Xu, J. Tan, and Y. Li, “Review and implementation of driving fatigue evaluation methods based on RR interval,†vol. 503. Springer Verlag, Beijing Key Lab of Urban Intelligent Traffic Control Technology, North China University of Technology, Shijingshan, Beijing 100144, China, pp. 833–843, 2019, doi: 10.1007/978-981-13-0302-9_81.

R. He et al., “Automatic Detection of Atrial Fibrillation Based on Continuous Wavelet Transform and 2D Convolutional Neural Networks ,†Frontiers in Physiology , vol. 9. p. 1206, 2018.

M. Guo and Y. Du, “Classification of Thyroid Ultrasound Standard Plane Images using ResNet-18 Networks,†in 13th IEEE International Conference on Anti-Counterfeiting, Security, and Identification, ASID 2019, 2019, vol. 2019-Octob, pp. 324–328, doi: 10.1109/ICASID.2019.8925267.

J. M. Rathod and H. S. Salehi, “Vision system with deep learning classifiers for automatic quality inspection,†in Pattern Recognition and Tracking XXXI 2020, 2020, vol. 11400, doi: 10.1117/12.2555032.

Olga Russakovsky*, Jia Deng*, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg and Li Fei-Fei. (* = equal contribution) ImageNet Large Scale Visual Recognition Challenge. IJCV, 2015.

Ü. Çavuşoğlu, “A new hybrid approach for intrusion detection using machine learning methods,†Appl. Intell., vol. 49, no. 7, pp. 2735–2761, 2019, doi: 10.1007/s10489-018-01408-x.

D. Chicco and G. Jurman, “The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation,†BMC Genomics, vol. 21, no. 1, 2020, doi: 10.1186/s12864-019-6413-7.

L. Frunzo, R. Garra, A. Giusti, and V. Luongo, “Modeling biological systems with an improved fractional Gompertz law,†Commun. Nonlinear Sci. Numer. Simul., vol. 74, pp. 260–267, 2019, doi: 10.1016/j.cnsns.2019.03.024.

N. W. Alt and S. Jochum, “Sound Design Under the Aspects of Musical Harmonic Theory.†SAE International , 2003, doi: 10.4271/2003-01-1508.

J. Yang, Y. Pan, and Y. Luo, “Investigation of brain-heart network during sleep,†in 42nd Annual International Conferences of the IEEE Engineering in Medicine and Biology Society, EMBC 2020, 2020, vol. 2020-July, pp. 3343–3346, doi: 10.1109/EMBC44109.2020.9175305.

K. Schiecke, A. Schumann, F. Benninger, M. Feucht, K.-J. Baer, and P. Schlattmann, “Brain-heart interactions considering complex physiological data: Processing schemes for time-variant, frequency-dependent, topographical and statistical examination of directed interactions by convergent cross mapping,†Physiol. Meas., vol. 40, no. 11, 2019, doi: 10.1088/1361-6579/ab5050.

E. Bigand, R. Parncutt, and F. Lerdahl, “Perception of musical tension in short chord sequences: The influence of harmonic function, sensory dissonance, horizontal motion, and musical training,†Percept. Psychophys., 1996, doi: 10.3758/BF03205482.




DOI: http://dx.doi.org/10.18517/ijaseit.12.4.16033

Refbacks

  • There are currently no refbacks.


Published by INSIGHT - Indonesian Society for Knowledge and Human Development