[Feature Request] Videos non-blocking playback with GUIs

[eri0o]

Being able to playback videos in AGS while using GUIs at the same time could enable building FMV games in AGS. Bonus points for having the video just run in any Overlay (regular overlays or room overlays), which would allow something similar to using videos for things in the background or some specifics, like say, a communication device in a GUI that receives a message.

From what I remember, when a video is playing AGS enters in a special state, so the screen is exclusively for the video and it can't process other things beyond the key to skip the video.

This as other special states are distributed throughout the code (see: Refactor AGS for 1 game loop and game state managing). It looks like some states could be instead managed through a Finite State Machine, but others are things we kinda want to just happen at the same frame (so not exactly a different state). This refactoring is not necessary, but if there was some similar way to arrange this it could maybe work. Video is a bit tricky that the sound and image has to be synchronous, so not sure that could workout.

This thread is both to discuss use cases in game and to see what would be needed in ags details, and also figuring out things like the API for this.

• Thread about GUI elements on top of video

[Vincent]

I think that since this engine is mainly based on a certain type of video games (point and click) and since FMV games somehow fall into this category, the function of playing a video should work a bit more broadly, such as running a video without blocks or having more resources in manipulating a video. I think it would be nice if videos can have more resources of manipulation like the audio channels do. For example having something like this for videos too:

Video.Pause
Video.Resume
Video.Seek
Video.SeekMs
Video.Speed
Video.Stop
Video.ID
Video.IsPaused
Video.IsPlaying
Video.LengthMs
Video.Position
Video.PositionMs
Video.Volume

But of course also keep the things that it already does because they are useful as VideoSkipStyle and Flags.

[Crimson Wizard]

My opinion on this, shortly:

1. I believe there should be a sort of VideoPlayback type (name is an example), which is returned from PlayVideo, and lets you control the video, similar to how AudioChannel lets you control the audio playback.
On a side note, as we discussed this previously, we should consider introducing AudioPlayback type, representing a playing audio instance, which is returned from AudioClip.Play, instead of returning AudioChannel. Maybe AudioPlayback and VideoPlayback could have a parent type (MediaPlayback?) which shares common properties and functions too. This will make both video and audio handled similarly.

2. The video should be updated on the main game loop, regardless of whether it's blocking or non-blocking. The difference between these would be simply that some of the game parts are not run during blocking video, similarly to how some parts are paused during blocking Walk or Say command.
NOTE: the video decoding may be performed on a separate thread even. By "updated on the main loop" I mean that the playback state and displayed image should be updated when the rest of the game updates.

3. The video should be decoded onto a dynamically created sprite, which may then be applied to any object. This will allow to play video on anything, and sort among other game elements on screen. VideoPlayback will have a Graphic property, or similar, to let reference this sprite. The sprite likely should be "owned" by the video until it is stopped, after which the sprite is disposed automatically.

4. The video's audio should be played through standard audio system, VideoPlayback should have a reference to AudioChannel it plays its audio on.
Maybe AGS will let configure which channel to play on, but that's a secondary question.
There are other details to decide here, like, should the video's audio have its own configurable "audio type", and so on.

[Vincent]

I think all CW opinions exposed are very pleasant and well organized which overall should work everything fine.
On a side note I was thinking of the easiest way for users to play a video as simply as playing an audio file in ags, in my mind I picture it something like this:

Maybe have a separate category for videos:


So you can simply add a video as you do with an audio file:


So every single video has his own properties (Regarding point 4 I think the video should have his own audio type too):


And inside the script we might eventually do something like this:

Code: ags

Video*video;
video = vIntro.Play(audioPriority, repeatStyle, blockingStyle, videoSkipStyle, flags);

     // vIntro.Play(eAudioPriorityNormal, eOnce, eNoBlock, eVideoSkipNotAllowed, 10);

[eri0o]

The video should be decoded onto a dynamically created sprite, which may then be applied to any object. This will allow to play video on anything, and sort among other game elements on screen. VideoPlayback will have a Graphic property, or similar, to let reference this sprite. The sprite likely should be "owned" by the video until it is stopped, after which the sprite is disposed automatically.

This sounds like a good idea, but just to try to figure out how is the "video sprite" size specified, does it uses the video resolution or is the dynamic sprite something we would create separately with whatever size and pass the dynamic sprite pointer to it? (VideoPlayback* vp = vcIntro.Play(dyn_spr);)

About the decoding, somehow the frame has to be set at the beginning or at the end of the game frame, so the contents of the dynamic sprite is predictable so people can assign it to things.

[Crimson Wizard]

This sounds like a good idea, but just to try to figure out how is the "video sprite" size specified, does it uses the video resolution or is the dynamic sprite something we would create separately with whatever size and pass the dynamic sprite pointer to it? (VideoPlayback* vp = vcIntro.Play(dyn_spr);)

There is also a question of ownership. If user passes a dynamic sprite to the Play function, that means that the user also owns the sprite. This means that user may delete the sprite or edit it anytime by accessing its DrawingSurface. That may cause conflicts with what video is trying to do. For that reason I'd consider an alternative method of passing width and height into Play function, and let video object create its own sprite.

About the decoding, somehow the frame has to be set at the beginning or at the end of the game frame, so the contents of the dynamic sprite is predictable so people can assign it to things.

The decoding must happen on a separate thread to not unnecessarily load up or block the game's thread. Also, the video playback may work with its own fps setting, not necessarily matching the game's fps.

The solution I could come up with at this point is to implement a back-buffer mechanism for this sprite. That means that instead of having 1 bitmap (or texture), the sprite has 2 associated bitmaps (or textures). One of them is considered to be sync with the game, and another is the one that video decoder writes to. Upon entering a game frame update, these bitmaps switch: a previous back-buffer becomes an active sprite's surface, and the previous active surface becomes back-buffer, that a video decoder will write to when necessary.

[Crimson Wizard]

I was too lazy, but finally opened a ticket with this task:
https://github.com/adventuregamestudio/ags/issues/2155

[eri0o]

I have been just getting a general feel on what is the status of video codecs today - this is only tangentially related, but still.

So I found the following:

pl_mpeg: this is mpeg-1 codec, which is very outdated, but has expired patents,  and the code for it is also very small and builds to few kB.

dav1d: this is an AV1 codec written mostly in assembly with a bit of C, it's cross platform and requires nasm and meson to build. The built lib dav1d.dll is around 3MB and using strip gets it down to 2MB. License is not GPL, so we can use on iOS. Plus, AV1 has no patent license requirements or royalties!

libgav1: this is the AV1 library that is available in Android, but it can be built cross platform too as it's used in Google Chrome too. It's not as fast as dav1d - 1920x1080@60 is definitely heavy for it, even on my desktop. But it uses CMake and doesn't require nasm, can be built with any compiler with C++11 support! It's also smaller in size. And it has permissable license and as AV1, no patent license/royalties are necessary.

SDL3: it's not added yet, but there are talks of adding an API for SDL to access system libraries. Interesting to follow up.


All in all the general idea is to have some plugin interface for adding more codecs more easily, but I wanted to take a look on how is the current situation for video codecs regarding patents, licenses and royalties - h264 and similar are out of the picture as libraries because of those.