Best approach for external data stream integration

[Patryk Kizny]

Hey,

So the next idea I have in my mind is to use external data to dynamically modify a fractal.
The data is coming from external sensors over serial port. It may be arduino-based stuff, MIDI or OSC.
I wonder what approach would you recommend to input this data in real-time to Synthclipse rendering a fractal.

Let's assume that the data input frequency would be higher than typical frame rate. So let's say I'm targeting 25FPS realtime performance and I'd need either to sample inputs at the framerate frequency or more often and only feed it to Synthclipse renderer every 40ms.

Is it doable only with Javascript stuff? Or I'd need to code something in C++ and feed it into?
What would be the best approach?

Thanks!

[cKleinhuis]

i find similar to shadertoy the option cool to have the webcam input as texture channel

[_revers_]

@Patryk
You obviously need to pack data into texture. (Like in audio texture [music UC], where 512 samples are packed for one frame). Creating and filling a texture in JavaScript is doable but gathering an input from external devices is not. The other concern about JavaScript generated textures in every frame would be performance, which might be slow.
Please point me some C++ library (preferably cross platform) for reading input you want and I see how much effort would it take me to add this to Synthclipse.
My friend Han Maokun has done Kinect integration for Synthclipse which maybe I include in the next release.

@Chris
I will also consider capturing camera input.

[cKleinhuis]

That is col to have kinect data inside synthclipse out of the box, i hopeit supports the new kinectv2

[Patryk Kizny]

Please point me some C++ library (preferably cross platform) for reading input you want and I see how much effort would it take me to add this to Synthclipse.

Essentially what would be sufficient for me on the basic level is an access to a serial port communication.

What the library/module should do is:
- make it possible to configure serial port settings from Synthglipse GUI and set baud rate, polling rate, open and close port, listen to a stream
- parse the data
- allow to create simple bindings/mappings: table of incoming variables, dropdown or textfield to enter corresponding Uniform on the right
- listen to the communications on the oper port, parse the data and set the uniforms accordingly during the runtime

Let's say in simplest scenario the communication on serial port would have a form of simple ASCII variables sent in separate lines over the serial port.
Say:
"VarName1 VarValue1; VarName2 VarValue2; VarName3 VarValue3; etc;"
And the lib would assign these values to selected uniforms in the runtime.

I have no practical knowledge of C++, but I bet there's a dozens of libs for facilitating serial communication.
Therere's a bunch of tools that are available for PD/MAX MSP including simple communications interfaces and templates for Arduino - say Firmata (https://www.arduino.cc/en/Reference/Firmata) for example or Simple Message System: http://playground.arduino.cc/Interfacing/PD

Here's some (old lib) for Arduino and C++ with LibSerial:
http://sglez.org/2008/08/05/interfacing-arduino-with-c-and-libserial/

And for Windows:
http://playground.arduino.cc/Interfacing/CPPWindows

Here's more details on available libs for other software:
http://playground.arduino.cc/Main/InterfacingWithSoftware

My friend Han Maokun has done Kinect integration for Synthclipse which maybe I include in the next release.

Is there any way to get in touch with him and get access to it? Kinect is my 2nd most important area of interest after serial for arduino.

[Patryk Kizny]

You obviously need to pack data into texture. (Like in audio texture [music UC], where 512 samples are packed for one frame).

Not so sure. Why not read the data on C++ end, parse it and use to set uniforms at each frame?

[Han Maokun]

That is col to have kinect data inside synthclipse out of the box, i hopeit supports the new kinectv2

Hello everyone, I am Han Maokun:)
I've been on the work of Kinect sensor feature of Synthclipse these days, I have a Kinect module 1473 in my hand, the library used is libfreenect(https://github.com/OpenKinect/libfreenect), to support Kinect V2, libfreenect2(https://github.com/OpenKinect/libfreenect2) is needed, I think they can both exists in Synthclipse.

[_revers_]

@Patryk
Thanks for the links. I will take a look at those libs.
I've thought about packing data into texture since you said that the frequency of the data is higher than the framerate.

@Han
I will include your Kinect 'module' in the next Synthclipse release.

@Chris
I will consider supporting Kinect v2, but without a test device it might be hard. You would need to help me testing it before release.

[Patryk Kizny]

I got a bunch of kinects here so I'm also keen to help testing.

[_revers_]

Thanks, that would be very helpful. I let you know when I start implementation.

[cKleinhuis]

i have windows10 with kinect2 readdy for testing!

[cKleinhuis]

how is kinect implementation planned? as uniform input for variables? or using jsx scripting ?

[_revers_]

Cool, I give you both app for testing when it will be ready.

As of the implementation, depth map created by Kinect is provided as a texture (uniform sampler2D) with Uniform Control type "depthmap".

[cKleinhuis]

I see, how about the joint informations ? Perhaps encode joint positions i to a bitmap?

[_revers_]

I don't know what do you mean by "joint informations", because I don't have a Kinect device, but I'll see what I can do with Kinect libraries.