open-source cross-platform real-time video effects plugin system
open-source cross-platform real-time video effects plugin system
What are FFEffecTV and FrameLab?
EffecTV is a port of a Japanese special effects library from Linux to run on FreeFrame compatible host programs.
The driving force behind EffecTV is Kentaro Fukuchi, a visual artist and HCI researcher at the Japan Science and Technology Agency, with major contributions by Ed Tannenbaum, Sam Mertens, Pete Warden, Nicolas Argyrou, Peter van Paassen, Clifford Smith, Adrian Likin, Ryan Broomfield, Nobuyuki Matsushita, TimeCop, Erich SCHUBERT, Sam Lantinga Slouken, and Jamie Fenton.
The port to FreeFrame builds upon work by TimeCop, who did an earlier port of EffecTV to the Windows DirectShow architecture. His efforts were enormous, which is what made the long weekend port so easy.
I mention these credits at the beginning to emphasize the point that EffecTV is a collection of great visual effects written by a diverse group of exceptionally talented programmers. Kentaro and his co-contributors deserve all the credit for the visual magic - not I. Please tell them how much you like their stuff, and if you find bugs, tell me about them, as they are my responsibility.
Converting this code over was an illuminating excercise - each one of the effects is relatively simple, ranging from a page to maybe 10. All of them together take up 130K or so of memory, which is tiny. There is little in the way of API programming, most effects take a source and destination buffer and have-at-it using C pointers. Reading these is both a historical jaunt and an education in high-performance graphics computing.
During the porting process, which took most of the recent U.S. Independence Day weekend, I managed to convert everything from C to C++, eliminate all global variables, and refactor the parameterization ontology to match FreeFrame's. This brought EffecTV from being able to run one special effect per Linux box to being able to run 72 instances (or more) at one time. Most of the work should back-port to Linux without much fuss.
EffectTV needs a showcase, and I have wanted a better FreeFrame host for my DirectShow projects so I started building one. In contrast to the straightforward and elegant EffecTV, the going here involved learning all sorts of Windows idiosyncracies. It took 6 days to port EffecTV, and six weeks to write a demo shell around it.
At present, we have a FreeFrame port of 49 EffecTV effects into one FreeFrame plugin. This can run inside of most popular VJ programs as well as most professional multimedia authoring tools. To make sure that those evaluating this package can run a demo right away, I created a FreeFrame to DirectShow host plugin with a full-function control panel. I also adopted a Microsoft SDK example media player program to play from disk or capture device, through the DirectShow adaptor, which calls the EffecTV plugin, which can demo those 49 special effects.
I started with 48 special effects and wrote one of my own to add to the library. Its the last one, and it is useful for browsing the other 48.
Welcome to the EffecTV/FrameLab Demonstration Program.
This program is a simple media player that can work with webcams as well as media files from disk.It automatically inserts a FrameLab filter into the video flow. This filter hosts Freeframe plug-in modules, several of which are in this package. The media player does not handle everything you throw at - it is limited by the assortment of CODECs available on your computer and by the notorious incompatibilities of the DirectShow architecture, as well as those introduced by the ridiculous software patent system we are saddled with in the USA.
To see special effects applied to the media you are playing, you need to open the control panel. This is done inside the File menu by selecting FrameLab Panel
To load an effect, click on the "Browse" button, upper left on the property page, and navigate to the plugin folder and select FFEffecTV.dll (or the filename for some other Freeframe plugin.)
"FFEffecTV is a very complex Freeframe plugin. That is because it generates so many different special effects under one roof. To change the effect, adjust the topmost slider. Full left is bypass, full right is a matrix index into all the effects.
Depending on the effect, one or more of the other sliders can introduce more variations. The most elaborate effects use up to 4 parameter settings. The remaining parameters are available for use by other Freeframe plugins.
The "Sampler" effect is the only one of the 49 that was created by me. It is a "Sampler" in the sense of a candy assortment arranged in a grid, showing a sample of each effect applied to the input. Most of the remaining controls in this demonstration are used by the Sampler (when it is selected), to control the other 48 effects in a variety of ways.
To fend off confusion I labeled the controls A-H instead of 1-9. (Since other Freeframe plugins can be used with FrameLab, I didn't want to muddle their documentation. Here is the table of parameters
0: 1 of 49 selector of which EffecTV effect to apply.
A: first parameter to the current EffecTV effect, if applicable.
B: second parameter...
C: third parameter...
D: / aux: fourth parameter to EffecTV, aux selection control for SampleTV.
The remaining parameters only take effect when parameter 0 is set all the way right (to maximum) when FFEffecTV is the active plug-in:
E: /sel: primary selector for SampleTV (to pick one of the other effects to manipulate (since parameter 0 is tied up selecting us).
F / cmd: command mode selector for the Sampler. There are 9 commands available. Adjusting this parameter picks out one of these. Commands which are harder to undo won't take effect until the following event trigger parameter is initiated.
G / go: when this event triggers, it causes the change requested by parameter F to take place.
H has been set up to display help information for users working within a Freeframe interface that only supports the display of 16-character wide values. By adjusting the slider for parameter 9, you can view a very simple "Times Square" style rendition of a help/credits screen. This was done as an aid to live performance without altering the screen display
Command modes selectable via parameter F and what they do.
Adjust - applies the settings of parameters A-D to the effect selected by E. The effect is as if you were adjusting it directly on its home screen.
Jump - exits the Sampler and transfers to the effect selected by E.
Mashup(D) - This mode uses parameter D to select an effect that is run before the other members of the matrix are run. What you see is the result of chaining each effect to the output of D.
Mashup(D:E) - D selects the column range, E selects the row range, and the matrix shows the combinations. For example, D set to 0 generates one column each for the first 4 special effects. E works similarly.
MashLab(D:E) - works like Maskup(D:E) except that the row and column headers show the result of only doing one of the transforms, which gives a "column and row label effect".
Random Parameter Change. E and D select a range of effects which will be randomly varied when the G trigger is fired. This effect auto-repeats as long as the trigger is down. It has a less dramatic effect than one expects because many EffecTV plugins have zero parameters.
Scramble Order. E and D select a range, and the members of the range are shuffled at random.
Move. Moves the effect at D into E, shifting the effects in between up or down.
Resize the grid - cycles through grid matrix sizes from the initial value of 4x4, up to 5x5,6x6,2x2,3x3,4x4...
A fun trick is to set up a scramble by setting D to min, E to max, and F to scramble. Then set G to nonzero and leave it there. While it is scrambling, play with the mashup modes by adjusting F, which will be scrambling as well as mashing. This trick works with Random changes too.
FrameLab With Other Plugins
FrameLab is a general FreeFrame host adaptor for DirectShow. This allows many Windows video processing programs to run FreeFrame plugins. FreeFrame has several hundred plug-ins available, about half of which are Free Software. An excellent place to begin is with Pete Warden's collection of plugins. http://www.petewarden.com/.
The big list is at: http://www.petewarden.com/PluginHelp.htm.
FrameLab is designed to be automation-scriptable so these plugins can be programatically controlled. It will be extended to support more media formats, multiple plugins, hot-routing, network links, and beyond.
Chris Korda is another prolific FreeFrame plugin author. Chris led the FreeFrame revival by releasing FFRend which is a rendering host which can run in real time or not - but it won't pretend to by dropping frames. Plugins can be chained together, and if desired, modularized as meta-plugins. Parameters can be controlled by banks of oscillators as well as manually. I use FFRend extensively, for testing and producing high quality demonstration reels. Almost all of Chris's code is free and open - and a pleasure to work with.
FreeFrame standards group has recently released version 1.5, which adds support for OpenGL. While I expect to support this relatively soon, I decided to wait on adding that requirement to an already overdue project.
To learn more about this program, please visit our SourceForge page: http://www.sourceforge.org/MoneyShot.
About Open Source Availability
FFEffecTV is an open source FreeFrame plugin released under GPL. I have also released sources for several of my FreeFrame plugins, including the video scope, component mapper, chroma keyer, RAM dumper. All are available under GPL.
I have not released FrameLab as open-source yet. I likely will at some point, but present circumstances encourage me to hold-off until I make another round of improvements. In the meantime, feel free to use FrameLab in your own projects with the understanding that I will be changing the API sooner or later, and don't want to deal with backwards compatibility or developer support issues.
I will end by repeating the mantra once more: thank THEM for the great program, complain to ME about the bugs. This project has been fun for me, and if it has brought joy to you and your family and friends, then we have succeeded.