Also (fairly important) the initial direction vector of the viewing ray (dx,dy,dz) is normalised.
of course, but it seems you forgot about that:
; Now test if we're within the bailout distance
if (t9 = gdist - x1*x1 - y1*y1 - z1*z1 \
+ (x2=1.0/(dx*dx+dy*dy+dz*dz))*(t11=x1*dx+y1*dy+z1*dz)^2)>=0.0
No I didn't forget - the transforms scale the direction vectors.
For transforms that have uniform scaling (x,y,z) and no skewing/shearing etc. - such as those for the Menger Sponge - the calculations can be simplified to avoid the recalculation of inverse of the scale of the normal since the value will be the same as the overall contractive scale (that's why the routine still uses the contractive scale values).
x2=1.0/(dx*dx+dy*dy+dz*dz) = 1.0
No because just prior to that we transformed the direction vectors:
dxv[j] = dx = (x1=dx)*gt[k,0] + (y1=dy)*gt[k,3] + dz*gt[k,6]
dyv[j] = dy = x1*gt[k,1] + y1*gt[k,4] + dz*gt[k,7]
dzv[j] = dz = x1*gt[k,2] + y1*gt[k,5] + dz*gt[k,8]
Which means that they were at least scaled, possible skewed, sheared etc.