Messages

Thanks! I hope it turns out to be useful to someone.

As for figuring out how to animate mouths, if you look in the second spoiler-hidden section of the post, there are three references which might help.

The "This board" and "This topic" search filters for the search box in the top-right corner of the forum pages don't work: it always searches through the whole forums. Similarly, the filters on the search result page don't work: it ignores any changes you make, e.g. if you tell it to only look for posts by a certain user, or "match any words". So you always have to go to the main search page to configure the search.

[IPA]

A couple more things:

Pamela and Annosoft SAPI 5.1 LipSync use almost exactly the same phoneme sets, with only minor variation (TotalLipSync is not case sensitive). Pamela can tag vowels with three levels of stress (0-2), e.g. AY0, UW1. This is not particularly useful in AGS, and should be ignored (by setting TotalLipSync.Init(ePamelaIgnoreStress)) unless there's good reason not to. Anyway, here's the full list and what they represent (ones where Annosoft differs emphasized):

Spoiler

IPA	Example	Pamela	Anno
[silence]	-	None	x
ɑ:, ɒ	father, box (AmEng)	AA	AA
æ	at, snack	AE	AE
ʌ, ə	hut, alone	AH	AH
ɑ, ɔ	thaw, dog	AO	AO
aʊ	cow, out	AW	AW
aɪ	hide, guy	AY	AY
b	bang	B	b
tʃ	cheese	CH	CH
d	damn	D	d
ð	these, bathe	DH	DH
ɛ, ɛə	bed, bear	EH	EH
ɜ:, ɚ	hurt, butter	ER	ER
eɪ	ate, bait	EY	EY
f	fine	F	f
g	good	G	g
h	house	HH	h
ɪ, ɪə	it, fear	IH	IH
i:	eat, free	IY	IY
dʒ	gee, jaw	JH	j
k	key, crush	K	k
l	lip	L	l
m	monkey	M	m
n	no	N	n
ŋ	ping, pong	NG	NG
əʊ	oak, slow	OW	OW
ɔɪ	toy	OY	OY
p	put	P	p
r	read	R	r
s	sap	S	s
ʃ	sharp	SH	SH
t	total	T	t
θ	thin	TH	TH
ʊ, ʊə	good, poor	UH	UH
u:	you	UW	UW
v	viking	V	v
w	we, question	W	w
j	yield	Y	y
z	zoo	Z	z
ʒ	seizure, genre	ZH	ZH

(Based on the in-app list in PAMELA and the Annosoft list here.)

[close]

Also, if you use Rhubarb to do the lip syncing, this could be helpful:

You don't need to compile Rhubarb, just get the latest release for Windows or OSX: https://github.com/DanielSWolf/rhubarb-lip-sync/releases

The script doesn't call Rhubarb, you'll have to do all of that. Take all the speech clips, convert them to .wav if necessary, copy them into the Rhubarb directory, and for each one, call "rhubarb.exe myclip.wav > myclip.tsv" (where "myclip" is the name of the clip). You can also put the text corresponding to each clip in individual .txt files to assist with the speech recognition, and then you'd call "rhubarb.exe myclip.wav -d myclip.txt > myclip.tsv". Then once that's done, copy all the .tsv files over into the directory of your compiled AGS game, and the module will read them.

Obviously this process is tedious, and also Rhubarb takes quite a while to process each clip, so if you have more than a couple of dozen clips you'll definitely want to automate it (you could write a batch file to go through and process each .wav file in the directory)

In fact, I wrote a very simple version of such a batch file:

Code: bash

for %%F in (clips/*.wav) do (
    rhubarb.exe clips/%%~nxF -d guide/%%~nF.txt > sync/%%~nF.tsv
)

This assumes that the voice clips are in a folder called "clips/" inside the Rhubarb directory, that the text files are in a folder called "guide/", and that there is a folder called "sync/" where the .tsv files will be written. It also requires a .txt file for each .wav file. So there are a lot of possible improvements. Save this in a text file in the Rhubarb directory and name it something like agsbatch.bat, and you can run it to process all the speech clips in one go (which might take a while!).

[TotalLipSync]

TotalLipSync

(Character head/animation by Preston Blair. King Graham sprite ripped from King's Quest II VGA by AGDI. Background from AGS Awards Ceremony by Ali.)

Download TotalLipSync Module (v0.5)

TotalLipSync is a module for voice-based lip sync. It allows you to play back speech animations that have been synchronized with voice clips. TotalLipSync has all the same capabilities as the voice-based lip sync that is built into AGS, and offers the following additional advantages:

• It works with LucasArts-style speech (as well as with Sierra-style and full-screen speech modes).
• In addition to the Pamela (.pam) and Papagayo/Moho Switch (.dat) formats supported by AGS, it also reads Annosoft SAPI 5.1 (.anno) and Rhubarb (.tsv) lip sync files.
• In particular, Rhubarb support means that lip syncing can be 100% automated (with decent results): no manual tracking of the speech clips is required.
• It is more flexible: You can switch speech-styles mid-game, change the phoneme mapping, use files with different data formats, etc.
• You don't have to do the phonemes-to-frames mapping manually: The module comes with a default auto-mapping.

How to use

• Create the lip sync data files for the speech clips. You can use one of these tools (personally I would recommend Papagayo for manual tracking, and Rhubarb for automatic lip syncing, but the Lip Sync Manager plugin is good too):
  
  • PAMELA
  • AGS Lip Sync Manager plugin
  • Papagayo
  • Annosoft SAPI 5.1 Lipsync (requires Windows Speech Recognition 5.1 exactly - may not work on newer systems)
  • Rhubarb
  The filename of each sync file should be the same as the speech clip except for the extension, and you need to place them in your compiled game folder (by default, in a folder names "sync/" inside the game folder).
• Create the speech animation for your character(s), with different animation frames for the different phonemes (see below), and set it as their speech view.
• Download and import the TotalLipSync module into your AGS project.
• Make sure your game settings are correct: the AGS built-in lip sync (in the project tree under "Lip sync") should be set to "disabled".
  
• If you are going to use Sierra-style (or full-screen) speech for your lip sync animations, you must create a dummy view. Make sure to give it exactly one loop and one frame. If you name the view TLS_DUMMY it will automatically be used by the module. Otherwise you can set the view to use with TotalLipSync.SetSierraDummyView().

You are now ready to use the module. Add the code to initialize TotalLipSync on startup:

Code: ags

function game_start() 
{
  TotalLipSync.Init(eLipSyncRhubarb);    // Or whatever lip sync format you're using
  TotalLipSync.AutoMapPhonemes();
}

Or if you want a custom phonemes-to-frames mapping:

Code: ags

function game_start() 
{
  TotalLipSync.Init(eLipSyncPamelaIgnoreStress);

  TotalLipSync.AddPhonemeMappings("None",0);
  TotalLipSync.AddPhonemeMappings("B/M/P",1);
  TotalLipSync.AddPhonemeMappings("S/Z/IH/IY/SH/T/TH/D/DH/JH/N/NG/ZH",2);
  TotalLipSync.AddPhonemeMappings("EH/CH/ER/EY/G/K/R/Y/HH",3);
  TotalLipSync.AddPhonemeMappings("AY/AA/AH/AE",4);
  TotalLipSync.AddPhonemeMappings("AO/AW/UH",5);
  TotalLipSync.AddPhonemeMappings("W/OW/OY/UW",6);
  // Frame 7 unassigned to match default Moho mapping
  TotalLipSync.AddPhonemeMappings("F/V",8);
  TotalLipSync.AddPhonemeMappings("L",9);
}

To speak a line with lip syncing, you simply call the extender functions Character.SaySync() or Character.SayAtSync(), using a speech clip prefix:

Code: ags

  cGraham.SaySync("&1 This line will be animated with lip sync");
  cGraham.SayAtSync(320, 100, 240, "&2 ... and so will this");    // x_left, y_top, width, message

And that's all there is to it! (If you don't use a speech clip prefix, or if there is no matching sync file, the speech animation won't play at all.)

Phoneme-to-frame mappings
The principle of lip syncing is that different sounds (phonemes) correspond to different mouth shapes. If we display an animation frame with the right mouth shape at the same time as that sound appears in the audio being played, the animation will seem to match the speech. The first step, then, is to identify the phonemes and timing of them in the speech (that's what the tools listed above are for), and the second step is to choose an appropriate animation frame for each phoneme. We usually don't use different animation frames for all the different phonemes, so we combine phonemes into groups that are all mapped to a single frame. The different tools have different sets of phonemes (or phoneme groups), so we have to define different mappings from phonemes to frames.

So here is the default mapping for each data format used by TotalLipSync. It has been set up for a speech animation with ten different frames, each representing a different mouth position. (This is a fairly standard setup.) If you stick to these frames and these mappings, you can use the same speech view no matter what lip sync tool or data format you use:

Spoiler

Frame	Description	Preston Blair	Rhubarb	AGD2	Rhubarb phoneme ID	Moho phoneme	Pamela phonemes
0	Mouth closed (or slack)	[slack or same as 1]			X	rest	None
1	M, B, P				A	MBP	M/B/P
2	Various consonants, (Rhubarb: Ee-type sounds)				B	etc	K/S/T/D/G/DH/ TH/R/HH/CH/Y/N/ NG/SH/Z/ZH/JH
3	Eh-type sounds, (Non-Rhubarb: Ee-type sounds)				C	E	IH/IY/EH/AH/ EY/AW/ER
4	Ah-type and I-type sounds				D	AI	AA/AE/AY
5	Aww-type sounds, Ow-type sounds (can also go in 6)				E	O	AO/OW
6	U-type and Oo-type sounds (Non-Moho: W)				F	U	UW/OY/UH
7	Moho: W		[same as 6]	[same as 6]	[same as 6]	WQ	W
8	F, V				G	FV	F/V
9	L (Th-type sounds can also go here, rather than in 2)				H	L	L

[close]

Where to get it
TotalLipSync is hosted on Github (mainly just as a way for me to learn about how Github works):
https://github.com/messengerbag/TotalLipSync

You can download the current release from there:

Download TotalLipSync Module (v0.5)

Known bugs
None

Change log
0.5
-Added APIs to get the currently lip syncing character, the current phoneme and current frame.

0.4
-Fixed support for Sierra-style speech
-Minor bug fixes for edge-cases
-Documentation

0.2 (pre-release)
-Added support for Papagayo/Moho Switch (.dat), Annosoft SAPI 5.1 LipSync (.anno) and Rhubarb (.tsv)

0.1 (pre-release)
-Pamela support for LucasArts-style speech

Originally based on code by Calin Leafshade (though very little of it remains in the current version).
Thanks to Grundislav for providing a speech view used in development and testing of the module!

I've seen it, but don't remember what it is.

The Indy 4 movie?

Yes!

Most people probably only know the movie (if at all) from the off-putting promotional shot that turned into a "murderbear" meme, but the film is sweet and surprisingly hilarious. From the enthusiastically old-timey voice over about "Darkest Peru!" that opens the movie, to Peter Capaldi and Nicole Kidman as the slapstick villains, I pretty much had a grin on my face the whole way through. The first shot has one of my favorite gags, as the girl listens to a Chinese language lesson with the phrase "I have been accused of insider trading and require legal representation."

Well, then I'll post another screencap, and maybe someone else will identify it before you overcome your scruples



(Of course, if you're pretty sure you'd have gotten it from this one, feel free to claim it.)

No x2

Is it City of God? It looks similar to the shots from the last time we had it in this thread.

The Sphere?

The movie that warned us of the dangers of kids getting high on telepathic links with drunk aliens!

It was Walton Goggins in the background, BTW.
And Chris Cooper in the foreground, of course.

Another in the list of "reminds me of, but can't very well be" guesses: Margin Call. (The monitors are clearly the wrong era.)

After that guess, I was curious about whether the guy in the background was who I thought it was, and ended up spoiling myself on the movie.

The films that come to mind are Bridge of Spies and Sliding Doors, but I'm reasonably certain it's neither of those.

Yes, it's Spartan. The film didn't really hold up on re-watch, but I can highly recommend the DVD commentary with Kilmer.

You're reminding me that I've had The Salton Sea on my to-watch list for ages.

No

No guesses? Is it that hard, or are people just tired of this game?

No one has been close so far.