END OF AN ERA, FRACTALFORUMS.COM IS CONTINUED ON FRACTALFORUMS.ORG

Hi All,

First let me say that the NetRender feature is great, it cuts down the render time heaps, so please take the rest of this post as suggestion rather than complaint. I understand this is an opensource project with limited or no funding and resources, done in people spare time.

I could be completely wrong here, but my observation is that the NetRender server seems to farm out the work to the clients that are connected at the time the render button is pressed, once the client finishes it's work it doesn't get any more and just sits idle. Also if the client gets disconnected and reconnects it also does not pick up any more work.

The first part of my suggestion, and probably the easiest to implement is to break the rendering up into a group of small jobs, (this may already be the case, is this what the "lines" are?) and make the server the keeper of the state of the jobs in the queue, a client can then request some jobs from the queue to work on, maybe 1-10 lines/blocks per CPU on the client, when the client finished working on the jobs it has, it can submit them back as complete and request some more lines/blocks to work on. In this scenario the server would also be a client of itself, but it would only request based on #CPU-1 where there are more than 1 CPU or 1-5 lines/blocks where there is only 1 CPU. also the server component should remain as a higher priority than the client component on the server.

Structuring it like this will do a few things:
 - Clients that get disconnected can reconnect and resume working on the render
 - Allow you to add extra clients mid way through a render to speed things up
 - Faster clients can take a larger share of the work load

To make this work smoothly you will probably need some rules like:
 - If a client disconnects / reconnects then the server should mark all jobs not complete previously handed to that client as though they were never started
 - when all jobs have been handed out and a client requests more jobs then the jobs that have been in the processing state (handed out to a client but not yet completed) the longest get handed out again to the new requesting client as well to work on as well, if they were handed out longer than a time out period (n Seconds? or calculated as something like 2x average processing time for that client?). The server will also need to handle the scenario if both (all) clients return the results for that line/block

Now for the second part of my suggestion, instead of breaking up the work into lines, break it up into areas (blocks for want of a better name) based on which rendering features are required to render that block, this way when a client request jobs it can tell the server which rendering features  (and versions of the features) it can render.

How this could potentially help:
 - Initially this may help in providing compatibility for NetRender between multiple versions, older clients may not have all the latest greatest rendering features, but it can still do some, potentially a lot of  the work, the server can prioritise the blocks that require newer features to the newer clients and let the older clients take the other work.
 - some rendering features may be better suited to specific CPU architectures, eg some may perform better on X86/x64 cpu's and some might perform better on ARM CPU's, like wise some might perform better with intel / ATI / nvidia / other graphics acceleration. potentially future in versions clients could detect which types of blocks it will be faster at processing and tell the server which types to prioritise giving the client.

All this is just suggestions, take what you can out of this, I would be happy to hear what if any you end up using.   

Thanks for a great application,

Dave.

You are right, that NetRender doesn't reconnect if connection is lost or when client is started during already started rendering. This is actually under implementation. But rest of your statement is not true. Every client and server use the same scheduler for rendering. Every CPU core (separate thread) start from given line of image (starting line is distributed to all clients) and render line by line until finish own unrendered region. Then looks for the biggest unrendered region and start from middle of them. It's is done without any influence from server. Every thread independently decides what should be rendered. In this way clients doesn't need to wait for any orders from server (no idle state) - they just continue rendering until something is to do. Synchronization between server and clients is done by periodic sending to all clients list of already rendered lines. Based in this data rendering threads on clients side know what is still to do and what take to render if region is finished. When rendering of line of image is finished then image data is sent to server.
If any computer in network is faster than another,  then it will just render more lines. So utilization of all clients and server is 100% and there is no overlapping of jobs.
I think it doesn't matter if jobs are divided to lines or blocks. The most important is synchronization and optimized utilization of all resources.


I'm not going to provide compatibility between different versions of application (expect minor difference between version number - bugfixes). The program is still in beta phase, so rendering results are a little different between versions. If any client will give different results then you will see on image e.g. interlaced brighter or darker regions or some shifts of colors.
You wrote about optimizing between different architectures. It would be very complicated and actually it's not applicable in actual rendering engine. Every thread render each pixel of image from the beginning to the end. So the job for pixel cannot be divided between CPUs. By the way I don't know any program which work like you described.

If you use wi-fi, this could be the reason. Wi-fi is quite slow network (it's only about 3-5MB/s), so this could be bottleneck. If you render image which is fast to render, then there is transferred a lot of data. For 4k image it need to transfer about 150MB of raw data. NetRender will perform much better if you use it for rendering images which are slow to render (with many effects or complicated fractal formulas). Then data rate will be much lower (will be more computing than data exchanging)
It's preferred to use 1Gbit network for render farms (or at least 100Mbit cable connection).

Thanks, That is useful info, either way, NetRendering "as is" is still faster than without NetRendering.