↓Skip to main content

Speech Recognition with Sphinx

STT Java Mary TTS CMU Sphinx
Table of Contents

After I’d been fooling around with speech synthesis enough, I wanted to go a bit further in the opposite direction and try speech recognition. The goal here wasn’t to transcribe sentences, but rather to recognize some defined commands. Again, everything should run purely locally and be reasonably resource-efficient, and that’s where Sphinx comes in. A small tutorial.

CMU Sphinx is a speech recognition project from Carnegie Mellon University in Pittsburgh, Pennsylvania. The project has been around for over 20 years and is still being developed. Version 5.0.0 of Sphinx4 was released on October 13, 2022.

Sphinx Listens #

The small Java program should listen to the microphone and transcribe the user’s speech in real-time in the console.

build.gradle #

For this small demo, I’m using version 5prealpha-SNAPSHOT, hosted on sonatype.org.

repositories {
    mavenLocal()
    mavenCentral()
    maven { url "https://oss.sonatype.org/content/repositories/snapshots/" }
}

dependencies {
    implementation group: 'edu.cmu.sphinx', name: 'sphinx4-core', version: '5prealpha-SNAPSHOT'
    implementation group: 'edu.cmu.sphinx', name: 'sphinx4-data', version: '5prealpha-SNAPSHOT'
}

SpeechRecognizer.java #

The SpeechRecognizer should later run in its own thread. For the demo, we use English speech recognition. I need to define an AcousticModel, a LanguageModel, and a Dictionary here. The LiveSpeechRecognizer listens to the microphone and transcribes live.

import java.io.IOException;

import edu.cmu.sphinx.api.Configuration;
import edu.cmu.sphinx.api.LiveSpeechRecognizer;
import edu.cmu.sphinx.api.SpeechResult;

public class SpeechRecognizer implements Runnable {
	
	String path = "resource:/edu/cmu/sphinx/models/en-us/";
	LiveSpeechRecognizer recognizer;
	
	public SpeechRecognizer() throws IOException {
		
		final Configuration configuration = new Configuration();
		configuration.setAcousticModelPath(path + "en-us");
		configuration.setDictionaryPath(path + "cmudict-en-us.dict");
		configuration.setLanguageModelPath(path + "en-us.lm.bin");
		recognizer = new LiveSpeechRecognizer(configuration);
	}
	
	@Override
	public void run() {
		recognizer.startRecognition(true);
		SpeechResult result;
		while ((result = this.recognizer.getResult()) != null) {
			String hypothesis = result.getHypothesis();
			if (hypothesis != null) {
				System.out.println(hypothesis);
			}
		}
	}
}

Main.java #

The “public static void main” is manageable again:

import java.io.IOException;

public class Main {
	
	public static void main(String[] args) throws IOException {
		SpeechRecognizer speechRecognizer = new SpeechRecognizer();
		new Thread(speechRecognizer).start();
	}
}

A First Test: Disenchantment #

For a first small test, I read Sphinx a sentence from https://www.esa.int/Science_Exploration/Space_Science/Webb:

“The James Webb Space Telescope is the next great space science observatory following Hubble.”

My attempts result in the following disappointing interpretations:

  • “chains wet spit the pope is the best with this i’m salsa people republic”
  • “the shameful say to school that gripped the size of the republic”
  • “and and wes a look when i eat besides we going on”

To preempt objections that my English pronunciation might be “not the best”, let me mention that the Google Assistant was able to transcribe the text without errors in a short test with my smartphone. Besides the shockingly useless results, the program feels very sluggish, and the built-in Memory Tracker reports it needs about 2 GB.

However, that wasn’t the intended use case anyway. Instead of full transcription, the program should only be able to interpret a few short commands. So I try something simpler. I say: “Computer, please open the browser.” At least Sphinx now recognizes the word “computer” in about half the cases (other results: “ultra”, “contract”, “corbett”, and “couch”), but “open the browser” also becomes “open a bottle”, “open the problem”, or simply “and girls up”. In this state, Sphinx is hardly usable.

A Hack #

Most of the previous results consist of words I wouldn’t expect as part of my commands. What if we simply drastically reduce the number of possibilities?

Based on the original cmu-en-us.dict file, I put together a reduced cmu-reduced-en-us.dict and adjust the path with setDictionaryPath. The new dictionary contains only a few words:

computer K AH M P Y UW T ER
open OW P AH N
close K L OW S
please P L IY Z
browser B R AW Z ER
editor EH D AH T ER
settings S EH T IH NG Z
the DH AH
player P L EY ER
console K AA N S OW L
chat CH AE T

Unfortunately, the program now takes very long to start: every missing word is logged as a warning. But then Sphinx surprises with very fast and nearly flawless speech recognition – as long as you stick to this restricted list of words. Memory usage is also down to 200 MB, just one-tenth of before.

Grammar #

The thousands of warnings already suggest this probably isn’t the intended way to optimize Sphinx. Instead, one should define a grammar. For that, I create a file command-grammar.gram:

#JSGF V1.0
grammar grammar;
public <basicCmd> = computer <command> (please)*;

<command> = <action> <object>;
<action> = /2/ open | /1/ close;
<object> = [the | a] (browser | settings | player | chat | console);

This is then configured in the SpeechRecognizer:

configuration.setGrammarName("command-grammar");
configuration.setGrammarPath("resource:/");
configuration.setUseGrammar(true);

And voilà, the results are just as good as in the previous hack, but the program now starts without warnings. Additionally, you can define things here, for example, that all commands must start with “Computer”, then mention the action, and finally the object. To showcase the features of JSGF (Java Speech Grammar Format), the example assumes that “open” occurs twice as often as “close”, and that politeness towards computers is optional (hence the asterisk after “please”).

And a Bit More Nonsense: Sphinx vs Mary #

Finally, a small experiment: What if we let Sphinx interpret sentences spoken by MaryTTS? Keep playing until input and output match. How many iterations does it take until Sphinx and Mary agree?

Monologue #1 #

With MaryTTS’s default voice “cmu-slt-hsmm”, very quickly only nonsense comes back from Sphinx. Notable is that somewhere along the way, a sequence of interpretations seems to repeat: After the second “ah”, “prove it”, “war ah sh home”, I gave up.

> The James Webb Space Telescope is the next great space science observatory following Hubble.
> the chrome or flu sh
> hannah old who knew old from home if shaq
> home room
> take him long
> hi ah fang moon
> area who showroom all
> le old high sheen
> mm mm mm mm
> ah a high
> this one
> ash who
> huh how
> in
> home
> e.
> ah
> prove it
> war ah sh home
> moon old hatch ah one
> in ah hum
> ah for
> for lol
> louis room
> mm mm flash
> home home home hung from roof
> all for one higher for food wall
> proof old moon half old fool mm mm mm
> huh sure mm mm
> who come home
> ha more
> says who
> sherman shirt
> how whole womb flu
> mm mm mm mm mm mm mm hole
> ah
> prove it
> war ah sh home

Monologue #2 #

In the next attempt I use the “Prudence” voice, which sounds a bit more natural. And indeed, the results are significantly better, and the loop ends after just six iterations:

> The James Webb Space Telescope is the next great space science observatory following Hubble
> the chains like space telescope is the next great space science observatory fawning harpo
> she leans likes a statistic is the next great space science observatory full mean how can
> he means like the statistic is the next great space science itself into a full mean how can
> he means like a statistic is the next great space science itself into a full mean how can
> he means like a statistic is the next great space science itself into a full mean how can
> he means like a statistic is the next great space science itself into a full mean how can
> he means like a statistic is the next great space science itself into a full mean how come

Still Good Enough? #

If you use the grammar feature, Sphinx is quite usable for my use case – recognizing a few defined commands. If you want to do more with it, there are still options like Acoustic Model Adaptation that could presumably squeeze out even more accuracy.