MODULE: TotalLipSync v0.5

[Olleh19]

I have two suggestions:

1. Don't have significant silences in your speech files. Divide the lines whenever there is a long pause, and trim the audio files to contain near-constant dialog. Forget about lipsync and just use regular AGS talk animations.
2. As you have tried to do, edit the sync-files to only distinguish silent periods (frame X in Rhubarb terms) from speaking periods, and then fill the speaking periods with a standard sequence of frames. Since this is tedious work, it would be best to automate it unless the number of lines to sync is very small.

If you are getting errors with the edited sync files, you have made some error in the editing. Without seeing the files or the error message there is no way to tell what that might be, but one possibility is that your text-editing app uses the wrong format for the line endings (one way to test this would be to take a sync file that is confirmed to work, and delete and reinsert a line ending and see if it breaks).

Notepad ++ is the editor of choice, btw. Sounds like some great advice there! Thanks as always, i will try some more!.
The instagram link works on my Laptop (Secondary Computer) that is NOT logged into Facebook or Instagram. However if going thru https://www.instagram.com/andreasblack1/ you can't hit the play button it seems and need to login. I don't have the mp4 left on my system since it crashed some month ago, and i'm back with a new HDD. But anyway, it doesn't matter.

It's funny you'd say that tho. Cause that's exactly what i've tried earlier today. But you get this feeling that you still want to manually be able to control the overall speech animation delay sometimes in sections, i guess in a way it could still be done, just like you say with AGS "own" lipsync support. If i type "SpeechAnimationDelay" out just before "a fast" line" or a word is said, in code. However that means i could only have "that word" in that line, or else all the others words get affected which may not be fast in nature, which could in theory i believe be better done with the Total Lipsync in the actual syncfile manually . I should take a look at your chart and line them up in the syncfile and experiment more, and try add rows of 01.10, 01.20, 01.30 etc etc.

I'm after the ability to change speed of the animation on different words (so in a way i guess that could be called "Animation syncing", but different then lipsyncs original feature, *frame syncinc* (?)). Cause at the moment it get's stuck a lot of times on "one frame", which looks aweful. I hate to swear in Church, but Thimbleweed Park alltho a great game, had the same fate. Sometimes the mouth shapes are stuck for too long, and it looks down right aweful.  Just because this worked in Sony Vegas tho, doesn't mean it would in an AGS editor. . But what i've just explained was exactly what i did in Sony Vegas. The ability to sometimes speed up the 10 frame sequence mp4, sometimes slow it down, and then *insert closed mouth* at silent places automatically with Total Lipsync (i had to do that manually in Vegas ofc). Would be a totally killer feature!

Maybe the easiest way to describe it should have been to say this: Bypass one frame function for words, and just have an overall speed animationdelay setting for chosen sections of the text (if text + audiosync will be available). Forexample you could type something like, "break" in the syncfile and it would "recognise" that OK that was that line/or word, and then the next line or word that comes get's a new potential animationspeed, or keeps the same unless the user choose to change it. To the users liking, and when there is no sound under -40-50db *insert noisegate* = closed mouth frame or a animation for closing the mouth, how cool would that be!. I know i'm talking out of my ass here . I'm no coder! I just enjoy AGS so much. Had a lot of fun lately, and even getting the lips to move "somewhat" is an epic win it itself Snarky! Thanks to your hardwork that was possible! But then you start dreaming "wouldn't it be possible to do this", and so on, so forth. What i've done so far with the Total Lipsync is probably good enough for the majority of people. It's just that my perfectionist spirit see's other possibilities, and now i've gotten them out of my system. Feels better now. Thank you!

[Dave Gilbert]

Hello all! I've been using this for awhile in conjunction with Rhubarb, and it seems to be working *perfectly* aside from one big issue.

If I reload an earlier saved game, or press F9, it somehow loses the lip sync data. The voiceover plays but characters' mouths stop moving entirely. Any idea why this would happen?

Any light-shedding appreciated!

edit: To clarify, if I start a fresh game the lip sync plays perfectly with mouth movement. But if I reload a save, or press F9 to restart, then the mouths stop moving (although the VO plays).

[Dave Gilbert]

Sorry to double post, but this seems to be caused by a bug in the engine rather than the module. The work around was to add these lines of code to an eEventRestoreGame in addition to game_start.

  TotalLipSync.Init(eLipSyncRhubarb);   
  TotalLipSync.AutoMapPhonemes();

The engine bug is being fixed so this info may be redundant soon!

[eri0o]

Hey, I know no one asked, but just thought about mentioning. Since this module can leverage files in the game installation directory, if someone doesn't want to do that, and instead want to have the files packaged with the game, since AGS 3.6.0 it's possible. Simply, create a directory at the root of your project, name it say "lipsync". Then note in the property below in General Settings. In the game code, when asked for the path for the files, use "$DATA$/lipsync", to find the files inside the AGS Game package.

Spoiler

[close]

[Snarky]

That's really great to avoid a ****ton of tiny files in your distribution. It's also great because it means I don't have to code the pack/unpack feature I've had on my TODO list for years.

[newwaveburritos]

I had a bit of trouble getting this to work in AGS 3.6.0.35 and I can't really explain it so I thought I would mention it.  I think the problem was basically that it was not loading the files correctly.  Eventually, I discovered that it was writing in the warnings.log what was going on:

Game : (room:2)[R 369] FileOpen: FAILED: c:/Brick Sundown 3.6/Brick Sundown/Compiled/Windows/sync/Bric1.dat

When I had the files name 1.DAT and 1.WAV it wasn't working but when I renamed them BRIC1.DAT and BRIC1.WAV it started working.  The name of the character is "Brick" so it sort of makes sense and seems to work fine now once I determined that maybe I should rename the file to the one that it's looking for.

I'm using Papagayo for the lip-syncing file.  The .DAT file is in Compiled>Windows>sync and the .WAV is in Compiled>Windows>Data

Anyway, it's working great now and I really like the module so thanks for making it!

[Snarky]

Yes, AGS requires voice clip file names to be the first four letters of the character script name (excluding the first letter if it starts with "c"), followed by a number. Since it isn't something specific to this module, but a standard part of AGS, I didn't explain it, but perhaps I should mention it.

There is also a complication that on some platforms (but not Windows) the file names are case-sensitive. I should add some better way to deal with this.

[newwaveburritos]

Newwaveburritos: Very mysterious behavior.
Snarky: Pretty standard stuff.  Did you read the documentation?
Newwaveburritos: Very mysterious indeed...

[Snarky]

Well, it was a failure of imagination on my part not to consider that users might be unfamiliar with how voice speech works in AGS. It should be mentioned/referenced in the documentation. Thanks for pointing it out!

[Crimson Wizard]

There is also a complication that on some platforms (but not Windows) the file names are case-sensitive. I should add some better way to deal with this.

IIRC for every asset name engine should try both case-sensitive and case-insensitive search. At least that worked for the files stored on disk.
But it may be skipped for the files stored inside the package, which frankly may be considered inconsistent, and something to be fixed.

[Snarky]

This is the problem I was referring to. Later versions of the engine may have been changed to be case-insensitive, but it should still be handled by the module in case it's used on older AGS versions. (And yes, if it doesn't find files case-insensitively in the packaged assets, that's probably going to be an issue since sync files are ideally suited for that kind of packaging.)

[Crimson Wizard]

Weird, I thought that problem was already addressed. I must check what the current situation is.

EDIT: I double checked, and the engine itself looks up for the *assets* (anything inside the package) using case-insensitive comparisons, always. This was true at least since AGS 3.3.0, and possibly even earlier in the initial Linux ports.

I suppose the problem may exist in File.Open and similar *script commands* when they look in the dir.

I don't know why this was not addressed earlier, when the above thread was created...

[newwaveburritos]

Now that I've learned how it works I think it's a fantastic module and really you get a lot of character out of some relatively easy steps.  I do have a question, though.  Is it possible to have two characters talking over top of each other at once?  I was thinking you could make the speech line a single WAV file but the characters seem to block one another when called in tandem.  It's not a big deal but I was trying to script an argument lol.  Thanks for the module all the same!

[AndreasBlack]

I'm not sure what's going on in my syncing, help! I'm exporting wrong perhaps? Since AGS wanted a dat file as i understood the instructions. I push the export Anime Studio button in Papagayo. Problem is the sync doesn't look nothing like in Pagagayo once it's in AGS. It looks more like 10fps everything is slowed down. Both AGS and Papagayo is set at 60fps. What's wrong? 9 Frames in the sprite view, perhaps i should have showed those in the video..Edit: Hmm, maybe there's too many closed mouths in there, nope doesn't seem to change anything. Is it the Thumbleweed Template perhaps that's causing something?

If i take away the sync file the character doesn't move his mouth at all, so obviously it has some form of function, but not the correct one.

Code: ags

TotalLipSync.Init(eLipSyncMoho); //Papagayo setting, right? i have 9 mouth movements in the speech view..
TotalLipSync.AutoMapPhonemes();

[Snarky]

Check out https://github.com/messengerbag/TotalLipSync#void-totallipsyncsetdatafileframerateint-framerate

[AndreasBlack]

Trying to SaySync the Standard Replies like "I can't use that" inside the Thumbleweed TemplateSettings.asc that works for multiply characters, and not just Character.SaySync. However i can't seem to find an option for it. I'm guessing it's doable, it's just my none coder brain is clueless

[Dave Gilbert]

Hi! I had a question about the way Rhubarb is set up in this module:

The rhubarb mapping is written like this:

Code: AGS

void _autoMapPhonemesRhubarb()
{
 TotalLipSync.AddPhonemeMapping("X",0);
  TotalLipSync.AddPhonemeMapping("A",1);  // mbp
  TotalLipSync.AddPhonemeMapping("B",2);  // other consonants
  TotalLipSync.AddPhonemeMapping("C",3);  // EH/AH/EY etc. (bed, hut, bait)
  TotalLipSync.AddPhonemeMapping("D",4);  // AA/AE/AY (father, bat, like)
  TotalLipSync.AddPhonemeMapping("E",5);  // AO/OW (thaw, slow)
  TotalLipSync.AddPhonemeMapping("F",6);  // UW/OY/UH/OW (you, toy, poor)
  // Frame 7 unassigned to match Moho mapping
  TotalLipSync.AddPhonemeMapping("G",8);  // F/V (fine, very)
  TotalLipSync.AddPhonemeMapping("H",9);  // L (letter)
}

When my talking sprites are drawn and saved to individual files, they are labelled with the appropriate letter name. So they are called A.png, B.png, C.png, etc, all the way to X.png.

Thus, when they are imported, they are imported in alphabetical order. A first, and X last. When I assign the sprites to the loops, I just assign them in order. Like so:



Since the sprite order is different than what the code above expects, I changed the code to this:

Code: AGS

void _autoMapPhonemesRhubarb()
{
  TotalLipSync.AddPhonemeMapping("X",8);
  TotalLipSync.AddPhonemeMapping("A",0);  // mbp
  TotalLipSync.AddPhonemeMapping("B",1);  // other consonants
  TotalLipSync.AddPhonemeMapping("C",2);  // EH/AH/EY etc. (bed, hut, bait)
  TotalLipSync.AddPhonemeMapping("D",3);  // AA/AE/AY (father, bat, like)
  TotalLipSync.AddPhonemeMapping("E",4);  // AO/OW (thaw, slow)
  TotalLipSync.AddPhonemeMapping("F",5);  // UW/OY/UH/OW (you, toy, poor)
  TotalLipSync.AddPhonemeMapping("G",6);  // F/V (fine, very)
  TotalLipSync.AddPhonemeMapping("H",7);  // L (letter)
}

This SEEMS like it should work, but I am noticing that the final result for lipsyncing isn't as good as it could be. So I'm wondering if the problem lies with the rhubarb end, or if me changing the code like this messed things up. For example, what was the reason that frame 7 was originally unassigned? Was it essential to keep that?

Anyway, light shedding appreciated!

-Dave

[Snarky]

So I'm wondering if the problem lies with the rhubarb end, or if me changing the code like this messed things up. For example, what was the reason that frame 7 was originally unassigned? Was it essential to keep that?

Hmm, changing the mapping should be fine (but see below), though I would strongly recommend that instead of changing the AutoMapPhonemes implementation in the module code, you set up a manual mapping outside of the module when you initialize it—so, instead of calling AutoMapPhonemes, you use those exact AddPhonemeMapping calls yourself to set it up the way you want it.

The reason why the auto-setup is arranged the way it is (skipping frame 7) is because the other lipsync data formats, Moho and Pamela/Annosoft, distinguish between "ooh" sounds and "w" sounds, but Rhubarb uses the same phoneme/mouth shape for both. The auto-setup maps the frames the same way for all the formats, allowing you to set up the animation view once, and then use whichever lipsync data format you like. You can even switch formats along the way, if for example you want to hand-sync some scenes in Pamela format and do others automatically in Rhubarb (though you'd need to reset the module with the new format and mappings whenever you switch). I don't remember precisely why it's frame 7 that is the "optional" one rather than the last frame, but I would guess I based the order on some standard or convention for lipsync setups.

Anyway, to get to the point of your question: looking over the code, it does assume that Frame 0 is the non-speaking frame, as in the AGS convention for NormalView and SpeechView; this frame is set when the animation file does not specify a frame. (This is necessary because some of the data formats can have gaps, but I don't remember if this applies to Rhubarb.) You could try to change it here:

Code: ags

void _playFrame(int frame)
{
  // Look up the frame based on phoneme, unless argument is -1, in which case use frame 0
  if(frame == -1)
  {
    _currentPhoneme = "";
    _currentFrame = 0; // <-- Change this to 8
  }
  // ...

If that doesn't help, I would be inclined to believe the problem is with Rhubarb, but another potential problem could be that the animation and the audio drift out of sync because AGS isn't running at its set game speed (which is pretty likely with a high-res game like Old Skies, if that's where you're using it). If so, that would probably be fixable fairly easily by basing the calculation of the current time code on the system clock rather than on counting game frames.

[eri0o]

I don't think you can get system clock from the API today, but like, does getting the elapsed time in milliseconds from when the game started running something that would help there?

[Snarky]

I don't think you can get system clock from the API today

You can: DateTime.Now

Incidentally, this is one of those cases that demonstrates how much harder things get when you go from a basic module to a more general one: If using the system clock fixes any desync problems related to game slowdown, and is simple to do (it is), doing so should be fine, right?

Well, not quite. Because what if the game, including audio, is paused in the middle of the animation? (For example by tabbing away from the window, if not set to run in the background?) Then the system clock progress will not match the audio progress. And if the module is extended to support syncing during background speech, we get another complication, because then the game can be saved and restored in the middle of a sync animation.

Probably the most robust solution would be to use the system clock unless there is a major discrepancy with the last frame (say, more than a second), and in that case use the game speed as a fallback.