I've gone quite far, but I come to the point where I have no idea why something that should work in practice doesn't.
Goal: External handheld camera composited with Fragmentarium-rendered fractals.
Process:
- Shoot
- Track in Boujou
- Export camera as TXT (3x3 camera rotation matrix + vec3 position data)
- Put this as a const array in Frag (separate mat3 rotation array and vec3 pos array)
- Access the array rows based on frame numbers (in fact I used a uniform float slider animated linearly across time to work around imperfect frames addressing when based on time*framerate expression)
- Set fragmentarium camera accordingly
- Render out
- composite (in Premiere or any other app)
This is what I got and I am wrapping my head around for the last 3 days to learn why the camera from fragmentarium won't match the original.
https://vimeo.com/moogaloop.swf?clip_id=134293286&server=vimeo.com&fullscreen=1&show_title=1&show_byline=1&show_portrait=0&color=01AAEAMaths can be either good or bad. It can be halfway good, right?
You can't blame it also on insufficient data resolutions as all positions have 10 or 12 digits precision.
Here's what I know:
- Tracking data is 100% accurate as I verified track in boujou and it also holds while imported to AfterEffects
- I checked a few approaches: exporting matrices, exporting static euler angles, moving euler angles and going with a camera through AfterEffects. All result in same mismatches.
- I believe I have my maths right, but I need someone to verify this as I may not be 100% right. Im just stuck and don't know where to go from there. Maybe I just do a stupid mistake and Im blind to it.
My coordinate system is right-handed, Z pointing upwards and Y pointing forward, X pointing to the right.
Here's how camera data is structured.
Sample of data from Boujou with headers (just to give you understanding of how it's described):
# boujou export: text
# Copyright (c) 2009, Vicon Motion Systems
# boujou Version: 5.0.1 49900
# Creation date : Thu Jul 23 12:13:25 2015
# The image sequence file name was /Users/Paco/Desktop/TRacking/Footage/AAAA6057.MOV
# boujou frame 0 is image sequence file 0
# boujou frame 249 is image sequence file 249
# One boujou frame for every image sequence frame
# Exporting camera data for boujou frames 0 to 249
# First boujou frame indexed to animation frame 1
#The Camera (One line per time frame)
#Image Size 1920 1080
#Filmback Size 36 20.25
#Line Format: Camera Rotation Matrix (9 numbers - 1st row, 2nd row, 3rd row) Camera Translation (3 numbers) Focal Length (mm)
#rotation applied before translation
#R(0,0) R(0,1) R(0,2) R(1,0) R(1,1) R(1,2) R(2,0) R(2,1) R(2,2) Tx Ty Tz F(mm)
0.663026147349 -0.747853717219 -0.033333850271 -0.348748512798 -0.269176434522 -0.897729648569 0.662397767775 0.606843360959 -0.439283886009 -2.774962905268 -1.750638150532 1.534247204442 25.674514
0.660936880844 -0.749591107736 -0.035715693238 -0.351836510813 -0.267484052424 -0.897030295674 0.662852554645 0.605446490536 -0.440523595167 -2.772439108285 -1.749194060190 1.537645566714 25.674514
0.659828739397 -0.750512283653 -0.036842187127 -0.351650913278 -0.265088198736 -0.897813946250 0.664053966072 0.605359033118 -0.438831141974 -2.773846044461 -1.748700011027 1.537922082660 25.674514
0.658998995239 -0.751217972186 -0.037307942015 -0.352224544644 -0.264398401269 -0.897792490254 0.664573693749 0.604785121899 -0.438835688962 -2.772464665984 -1.749206942765 1.534491852473 25.674514
0.656638760299 -0.753211772356 -0.038698377977 -0.354782632433 -0.263201263587 -0.897136766926 0.665548512279 0.602824286863 -0.440054606806 -2.769924828617 -1.749933083429 1.535866913227 25.674514
0.654052549436 -0.755453530972 -0.038797230813 -0.355054518019 -0.261300668771 -0.897584675523 0.667945770112 0.600842677447 -0.439131557904 -2.766813396980 -1.748345004429 1.536441770413 25.674514
0.652479616903 -0.756858549580 -0.037887800847 -0.353853615148 -0.260079470259 -0.898413205711 0.670117876625 0.599603039583 -0.437513687044 -2.766718225801 -1.749131303252 1.535711352046 25.674514
0.650062470043 -0.758903613280 -0.038523898978 -0.355221351508 -0.258676966089 -0.898278363675 0.671741450615 0.597621563338 -0.437734954693 -2.765505040353 -1.749549835259 1.532746813238 25.674514
0.647093804549 -0.761332874859 -0.040519893549 -0.358198875094 -0.256674549898 -0.897670174015 0.673025388820 0.595390988422 -0.438801090371 -2.764800831837 -1.747220662126 1.533190992750 25.674514
0.645933673069 -0.762227937198 -0.042169440968 -0.359177593629 -0.254705294704 -0.897840001940 0.673617952723 0.595091408613 -0.438297922838 -2.762317965746 -1.746147663167 1.535634810674 25.674514
This is converted to fragestible data:
const mat3 iCamDataDir[250] = mat3[250](
mat3(0.6630261473,-0.7478537172,-0.03333385027,-0.3487485128,-0.2691764345,-0.8977296486,0.6623977678,0.606843361,-0.439283886),
mat3(0.6609368808,-0.7495911077,-0.03571569324,-0.3518365108,-0.2674840524,-0.8970302957,0.6628525546,0.6054464905,-0.4405235952),
mat3(0.6598287394,-0.7505122837,-0.03684218713,-0.3516509133,-0.2650881987,-0.8978139463,0.6640539661,0.6053590331,-0.438831142),
mat3(0.6589989952,-0.7512179722,-0.03730794202,-0.3522245446,-0.2643984013,-0.8977924903,0.6645736937,0.6047851219,-0.438835689),
mat3(0.6566387603,-0.7532117724,-0.03869837798,-0.3547826324,-0.2632012636,-0.8971367669,0.6655485123,0.6028242869,-0.4400546068),
…
const vec iCamDataPos[250] = vec3[250](
vec3(-2.774962905,-1.750638151,1.534247204),
vec3(-2.772439108,-1.74919406,1.537645567),
vec3(-2.773846044,-1.748700011,1.537922083),
vec3(-2.772464666,-1.749206943,1.534491852),
vec3(-2.769924829,-1.749933083,1.535866913),
Here's how I retrieve camera positions (this is coming from 3D.frag and sits within #vertex section)
Not sure why, but I needed to *(-1) the up vector to match (can this be wrong?)
// This is the file with coordinates and orientations
#include "CameraData_2.frag"
vec3 getCameraPos(){
// addressing array rows based on frame number. needed to add animated uniform float casted to int because tim*FPS was not reliable
vec3 cp = iCamDataPos[int(ceil(Frame))];
cp = (cp + iCamPosOffs) * iCamPosMod * iCamScale;
return cp;
}
vec3 getCameraRight(){
mat3 tm = iCamDataDir[int(ceil(Frame))];
return vec3(tm[0]) * iCamDirMod;
}
vec3 getCameraUp(){
mat3 tm = iCamDataDir[int(ceil(Frame))];
return -vec3(tm[1]) * iCamDirMod;
}
vec3 getCameraFw(){
mat3 tm = iCamDataDir[int(ceil(Frame))];
return vec3(tm[2]) * iCamDirMod;
}
I understand from that post That I can use matrix rows directly as normalized axis vectors.
http://www.gamedev.net/topic/666236-converting-axis-angles-forward-right-up-to-euler/#entry5213778Further within #vertex main() I have:
iCamWeight is just a float to enable and mixin my imported camera positions. It is set to 1.
Adding normalize() to Dir/Up/Forward won't change anything as I belive matrix data is already normalized.
vec3 Upn;
if (iCamWeight>0){
from = getCameraPos()* iCamWeight + Eye * (1-iCamWeight);
Dir = getCameraFw() * iCamWeight + normalize(Target - Eye) * (1-iCamWeight);
Upn = getCameraUp() * iCamWeight + normalize(Up) * (1-iCamWeight);
UpOrtho = normalize(Upn-dot(Dir,Upn)*Dir);
Right = getCameraRight();
} else {
from = Eye + Ey_;
Dir = normalize((Target+Targe_) - (Eye+Ey_));
Upn = normalize(Up);
UpOrtho = normalize( Upn-dot(Dir,Upn)*Dir );
Right = normalize(cross(Dir,UpOrtho));
}
That's pretty much it. What's wrong here?
I attach frags. To make get it up and running select Default, apply, then select range preset and apply.
Scrubbing in timeline animation mode should move the camera.
To test it against the source one needs to render and composite externally (I did manage to pull in single-frame background, but did not get Frag to update it per frame. I'll be happy to share backplates if anyone needs that. But hopefully you can help by looking at maths and the code solely with a fresh eye.
T H A N K Y O U