Robust multi-stream keyword and non-linguistic vocalization detection for computationally intelligent virtual agents

Martin Wöllmer; Erik Marchi; Stefano Squartini; Björn Schuller

doi:10.1007/978-3-642-21090-7_58

Robust multi-stream keyword and non-linguistic vocalization detection for computationally intelligent virtual agents

Martin Wöllmer, Erik Marchi, Stefano Squartini, Björn Schuller

Chair of Health Informatics

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

7 Scopus citations

Abstract

Systems for keyword and non-linguistic vocalization detection in conversational agent applications need to be robust with respect to background noise and different speaking styles. Focussing on the Sensitive Artificial Listener (SAL) scenario which involves spontaneous, emotionally colored speech, this paper proposes a multi-stream model that applies the principle of Long Short-Term Memory to generate context-sensitive phoneme predictions which can be used for keyword detection. Further, we investigate the incorporation of noisy training material in order to create noise robust acoustic models. We show that both strategies can improve recognition performance when evaluated on spontaneous human-machine conversations as contained in the SEMAINE database.

Original language	English
Title of host publication	Advances in Neural Networks - 8th International Symposium on Neural Networks, ISNN 2011
Pages	496-505
Number of pages	10
Edition	PART 2
DOIs	https://doi.org/10.1007/978-3-642-21090-7_58
State	Published - 2011
Event	8th International Symposium on Neural Networks, ISNN 2011 - Guilin, China Duration: 29 May 2011 → 1 Jun 2011

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Number	PART 2
Volume	6676 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	8th International Symposium on Neural Networks, ISNN 2011
Country/Territory	China
City	Guilin
Period	29/05/11 → 1/06/11

Keywords

Conversational agents
keyword spotting
long short-term memory
multi-condition training

Access to Document

10.1007/978-3-642-21090-7_58

Cite this

Wöllmer, M., Marchi, E., Squartini, S., & Schuller, B. (2011). Robust multi-stream keyword and non-linguistic vocalization detection for computationally intelligent virtual agents. In Advances in Neural Networks - 8th International Symposium on Neural Networks, ISNN 2011 (PART 2 ed., pp. 496-505). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 6676 LNCS, No. PART 2). https://doi.org/10.1007/978-3-642-21090-7_58

Wöllmer, Martin ; Marchi, Erik ; Squartini, Stefano et al. / Robust multi-stream keyword and non-linguistic vocalization detection for computationally intelligent virtual agents. Advances in Neural Networks - 8th International Symposium on Neural Networks, ISNN 2011. PART 2. ed. 2011. pp. 496-505 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); PART 2).

@inproceedings{9f336559b8da4454b1e4a594b9648d53,

title = "Robust multi-stream keyword and non-linguistic vocalization detection for computationally intelligent virtual agents",

abstract = "Systems for keyword and non-linguistic vocalization detection in conversational agent applications need to be robust with respect to background noise and different speaking styles. Focussing on the Sensitive Artificial Listener (SAL) scenario which involves spontaneous, emotionally colored speech, this paper proposes a multi-stream model that applies the principle of Long Short-Term Memory to generate context-sensitive phoneme predictions which can be used for keyword detection. Further, we investigate the incorporation of noisy training material in order to create noise robust acoustic models. We show that both strategies can improve recognition performance when evaluated on spontaneous human-machine conversations as contained in the SEMAINE database.",

keywords = "Conversational agents, keyword spotting, long short-term memory, multi-condition training",

author = "Martin W{\"o}llmer and Erik Marchi and Stefano Squartini and Bj{\"o}rn Schuller",

year = "2011",

doi = "10.1007/978-3-642-21090-7_58",

language = "English",

isbn = "9783642210891",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

number = "PART 2",

pages = "496--505",

booktitle = "Advances in Neural Networks - 8th International Symposium on Neural Networks, ISNN 2011",

edition = "PART 2",

note = "8th International Symposium on Neural Networks, ISNN 2011 ; Conference date: 29-05-2011 Through 01-06-2011",

}

Wöllmer, M, Marchi, E, Squartini, S & Schuller, B 2011, Robust multi-stream keyword and non-linguistic vocalization detection for computationally intelligent virtual agents. in Advances in Neural Networks - 8th International Symposium on Neural Networks, ISNN 2011. PART 2 edn, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), no. PART 2, vol. 6676 LNCS, pp. 496-505, 8th International Symposium on Neural Networks, ISNN 2011, Guilin, China, 29/05/11. https://doi.org/10.1007/978-3-642-21090-7_58

Robust multi-stream keyword and non-linguistic vocalization detection for computationally intelligent virtual agents. / Wöllmer, Martin; Marchi, Erik; Squartini, Stefano et al.
Advances in Neural Networks - 8th International Symposium on Neural Networks, ISNN 2011. PART 2. ed. 2011. p. 496-505 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 6676 LNCS, No. PART 2).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Robust multi-stream keyword and non-linguistic vocalization detection for computationally intelligent virtual agents

AU - Wöllmer, Martin

AU - Marchi, Erik

AU - Squartini, Stefano

AU - Schuller, Björn

PY - 2011

Y1 - 2011

N2 - Systems for keyword and non-linguistic vocalization detection in conversational agent applications need to be robust with respect to background noise and different speaking styles. Focussing on the Sensitive Artificial Listener (SAL) scenario which involves spontaneous, emotionally colored speech, this paper proposes a multi-stream model that applies the principle of Long Short-Term Memory to generate context-sensitive phoneme predictions which can be used for keyword detection. Further, we investigate the incorporation of noisy training material in order to create noise robust acoustic models. We show that both strategies can improve recognition performance when evaluated on spontaneous human-machine conversations as contained in the SEMAINE database.

AB - Systems for keyword and non-linguistic vocalization detection in conversational agent applications need to be robust with respect to background noise and different speaking styles. Focussing on the Sensitive Artificial Listener (SAL) scenario which involves spontaneous, emotionally colored speech, this paper proposes a multi-stream model that applies the principle of Long Short-Term Memory to generate context-sensitive phoneme predictions which can be used for keyword detection. Further, we investigate the incorporation of noisy training material in order to create noise robust acoustic models. We show that both strategies can improve recognition performance when evaluated on spontaneous human-machine conversations as contained in the SEMAINE database.

KW - Conversational agents

KW - keyword spotting

KW - long short-term memory

KW - multi-condition training

UR - http://www.scopus.com/inward/record.url?scp=79957788667&partnerID=8YFLogxK

U2 - 10.1007/978-3-642-21090-7_58

DO - 10.1007/978-3-642-21090-7_58

M3 - Conference contribution

AN - SCOPUS:79957788667

SN - 9783642210891

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 496

EP - 505

BT - Advances in Neural Networks - 8th International Symposium on Neural Networks, ISNN 2011

T2 - 8th International Symposium on Neural Networks, ISNN 2011

Y2 - 29 May 2011 through 1 June 2011

ER -

Wöllmer M, Marchi E, Squartini S, Schuller B. Robust multi-stream keyword and non-linguistic vocalization detection for computationally intelligent virtual agents. In Advances in Neural Networks - 8th International Symposium on Neural Networks, ISNN 2011. PART 2 ed. 2011. p. 496-505. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); PART 2). doi: 10.1007/978-3-642-21090-7_58

Robust multi-stream keyword and non-linguistic vocalization detection for computationally intelligent virtual agents

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this