There are three types of formulas:
1) which uses analytic distance estimation: trig_DE, trig_optim (default)
2) IFS formulas and Mandelbox: menger_sponge, tglad, smoothMandelbox, kaleidoscopic, mandelboxVaryScale4D
3) rest of formulas, which uses "delta DE" algorithm for distance estimation
Third type of formula is the easiest for modification and doesn't need special knowledge about distance estimation, because it is done by external function.
So if you want to test your own formula, try to overwrite for instance hypercomplex
CVector3 newz(z.x * z.x - z.y * z.y - z.z * z.z - w * w, 2.0 * z.x * z.y - 2.0 * w * z.z, 2.0 * z.x * z.z - 2.0 * z.y * w);
double neww = 2.0 * z.x * w - 2.0 * z.y * z.z;
z = newz + constant;
w = neww;
r = sqrt(z.x * z.x + z.y * z.y + z.z * z.z + w * w);
There are two very important variables:
z - iteration vector (z.x,z.y,z.z) which you use for your formula ( z = formula(z) )
r - length of iteration vector - must be calculated after formula calculation - needed for bailout checking. If formula is 3-dimensional you can use r = z.Length();
You can't implement there formula like isInSet(x,y,z), because there has to be possibility to estimate distance to fractal surface. The output of formula can't be binary. It has to return actual length of iteration vector, which is used for distance estimation function.
If you put there your formula and re-compile the program, you can test it by selecting "Hypercomplex" formula.
If you use third type of formula, you don't have to transform formula in any special way.
Adding of completely new formulas (by not overwriting) is much more difficult, because you have to make some additional changes in other parts of program.
Edit: there is third also important variable: constant
which is c from e.g. z = z^2 + c formula.