Welcome, Guest
Username Password: Remember me

TTS-bookmark-events and blocking TTS-events
(1 viewing) (1) Guest
  • Page:
  • 1

TOPIC: TTS-bookmark-events and blocking TTS-events

TTS-bookmark-events and blocking TTS-events 5 years ago #178

  • ap
  • OFFLINE
Dear Reeti-team,

currently I am working on an university project. The goal of this project is to simulate complex dialogue behavior with the help of Reeti. Therefore, we use an own written Java-based tool from our department, which uses a state-machine-like notation for modeling dialogues. Every state can contain for example a dialog structure with utterances:

reeti: Hello, my name is Reeti.
other_agent: Thank you Reeti. And my name is…

“reeti” is in this case one of our supported agents. Every agent has got an IP-address, which is bound to a specific port, so that we can use a socket connection between our modeling tool and the agent. Now, the utterance of every reeti-agent-section should be sent to a Reeti-server and executed with its TTS-engine by the URBI-client. As a main requirement, the Reeti server must be implemented as a Java-application. In addition to that, an utterance can also contain internal modeling commands, for improving the user experience between the users and the agents:

reeti: Hello, [led color=’red’] my name is Reeti.

Looking to the example above, we want to achieve following effect: after saying “Hello, ”, the URBI-client of our Java-server should execute a Global.servo.changeColor(“red”)-command asynchronously and continue with the rest of the TTS-command without interruption.

Summarized, we use the following steps:
  1. Establish a TCP-socket-connection between our Java-based modeling tool and Java-based Reeti server.
  2. Visit one state
  3. Go through the utterances of each state
  4. If the agent-name is “reeti”, then send the utterance to the Reeti-server.
  5. Apply server-side URBI-command-transformation rules to the received utterance. For example:
    • [led color=’red’] --> Global.servo.changeColor(“red”);
    • Some text --> Global.tts.sayWithSynchro(“Some text”);
  6. Store the transformed URBI-commands inside the data structure ExecutionBlock.
  7. Execute the TTS command inside the data structure and additional commands at the right position inside the TTS-text, if needed.
  8. Send a notification back to our modeling tool, if the TTS-engine has finished speaking.
  9. Visit the next state or proceed with the next utterance.


The utterance “Hello, [led color=’red’] my name is Reeti.” will be for example transformed to Global.tts.sayWithSynchro(“Hello, \\k=0 my name is Reeti.”);. The tag \\k=0 refers to a non-TTS bookmark event, which is stored inside our ExecutionBlock-object in a HashMap. The command should be executed asynchronously after speaking “Hallo, “. According to some official Loquendo-TTS-documentations, some implementations support bookmark-events with using the \\k=<INTEGER>-tag.

The main question is: Why does not Reeti support bookmark-events? Or: how we can use bookmark events or some sort of callback events inside our Java-based server application to get the events at the right position?

The next big problem concerns the execution of TTS-commands and the outdated API-documentation. In contrast to the documentation, TTS-commands are non-blocking. This is a serious problem in our case, because we want to send only a notification back to our modeling-tool, if the TTS-engine has finished with speaking. Currently, TTS-commands are just fired as asynchronous fire-and-forget events, which immediately return -1 or 0. This is not applicable in our situation. By using the USyncClient, we found some sort of hack to solve this problem. But on the other side, this solution does not satisfy us, because it does not work properly with sayWithSynchro-commands:

 
USyncClient client = new USyncClient();
UMessage message = client.syncGet(Global.tts.say(\\”Hello, my name is Reeti\\”);Global.tts.say(\\”\\”)); // insert empty TTS-command at the end. But: this does not work properly with sayWithSynchro-commands
 


How we can use blocking TTS-commands like in earlier versions, without any hacks and the loss of the sayWithSynchro-functionality?

Finally, we would like to point out, that these features are necessary for us:
  • Bookmark events like in other Loquendo-TTS-implementations
  • Blocking TTS-commands


Many thanks in advance for your help.

Alex.

Re: TTS-bookmark-events and blocking TTS-events 4 years, 12 months ago #179

Hello,

Answer to your first question : "Why does not Reeti support bookmark-events ?"
If you update your reeti, you will download a new UTTS module. This module allow you tu use Bookmark events. There is a bound variable in UTTS called "bookmarkReached" which reflect the state of the current bookmark. If the reeti is not speeking, or no bookmark had been reached, this variable is equal to an empty string (""). Else, this variable is equal to the content of the bookmark.
For example, if you send this command to the server :
Global.tts.say("Hi, I am a \\book=sad sad REETI. Ho no, wait, I am \\book=happy happy");
At the begining, bookmarkReached="", just before Reeti said "sad", bookmarkReached="sad". And just before reeti said "happy", bookmarkReached="happy". When reeti finish its sentence (after saying "happy"), then bookmarkReached="".

Answer to your second question : "How to be have a blocking TTS Command".
It is possible to create a simple urbi script like this :
 
function Global.TTSSayBlocking(phrase)
{
Global.tts.say(phrase);
waituntil(Global.tts.isSpeaking==1);
};
 


Then, you can add this function to ~/reetiPrograms/load/functions.u and then this function will exists in your REETI.

Best regards,
  • Page:
  • 1
Time to create page: 0.27 seconds