roughness against aliasing

[Jesse]

A not to difficult way to reduce aliasing caused mainly by specular light on edges, is to calculate what i call roughness factor (bad english? maybe rough-factor) and tune the specular and diffuse light by it.

The r-factor can be calculated from the deviation of the normals in different distances to the center position of a pixel.
Therefore i calculated first the normals in a small distance and interpolated the x and y vector with the z vector (cam out), so that the new vectors lies on an isosurface and dont point towards the set.
Then i calculated standard deviation for the normals on some points from - n* vector to n * vector for the iso-vectors to give the
r-factor.

The specular light will then be linear interpolated by the r-factor from standard spec-light to a constant value, containing the average amount of specular light, dependend of its power function.
I am not sure how this can be adopted for the diffuse light, when calculating the dot product. I am using angles and a lookup-table for the diffuse light, so i made a second LUT that contains a smoothed curve version and interpolate then between these based on the r-factor.

That all is not perfect, but the improvement is obvious, left the previous version without r-factor but with 3x3 aliasing, right the additional function as described:

[KRAFTWERK]

Looks good! 

[hobold]

Well done! Highlight aliasing is one of the tougher nuts to crack. The "deviations of the normals" are in fact the local curvature of the surface, which is the exact thing you need to monitor. Highlights are actually reflections of light sources. And the direction of the reflected light ray varies all the more, the more curved the surface is. Thus the local curvature is an indication for how much a small bundle of nearby view rays (e.g. all those originating from one specific pixel) are fanned out / spread apart after the reflection.

This widening of the fan in turn causes the rays to subsample the reflected image, and eventually results in improper rendering of the reflected light source, i.e. the highlight.

[Jesse]

Well done! Highlight aliasing is one of the tougher nuts to crack. The "deviations of the normals" are in fact the local curvature of the surface, which is the exact thing you need to monitor. Highlights are actually reflections of light sources. And the direction of the reflected light ray varies all the more, the more curved the surface is. Thus the local curvature is an indication for how much a small bundle of nearby view rays (e.g. all those originating from one specific pixel) are fanned out / spread apart after the reflection.

Yes, we can say that the r-factor simulates somehow the spreading of the light by the specific surface reflections.
It is still a compromise, in general, you would sum up all the reflected light of all little fractional parts from the surface by calculating their normals and dot-products with the light source(s). But this wont work when you want to adjust the light sources in realtime afterwards 

That explains also why the result is not perfect, on an edge, the r-factor cant say if the surface bends towards an angle that reflects more light, it will normally decrease the specular light, what solves the biggest problems but leaves some light steppings.
Because this function doesnt need additional time compared with the default smoothing of the normals, i can live with it.