Outerra forum

Outerra Engine => Technology => Topic started by: cameni on September 26, 2010, 01:44:26 pm

Title: Asynchronous Job Scheduler
Post by: cameni on September 26, 2010, 01:44:26 pm
A new technical (hi corona :)) article on the blog, talking about some of the engine components.
Read about the asynchronous job scheduler (http://outerra.blogspot.com/2010/09/asynchronous-job-scheduler.html) and logging/graphing components.

(http://www.outerra.com/shots/graphs.jpg) (http://outerra.blogspot.com/2010/09/asynchronous-job-scheduler.html)
Title: Asynchronous Job Scheduler
Post by: corona on September 27, 2010, 07:50:21 am
Hi cameni? (having an identity crisis there?)

Unfortunatly I don't have much to add to the post, other than saying I hope those timing/logging components will be available for content producers and end-users in the final engine to assist with tweaking/troubleshooting.

Are you able to log poly-counts (perhaps with a breakdown to the specific model), as well as texture usage, again with a breakdown to which model? I believe this would really help content-producers optimize their stuff to run well.
Title: Asynchronous Job Scheduler
Post by: cameni on September 27, 2010, 08:17:36 am
Yes those components are meant to help us and content producers to simplify identification of problems. Model importer provides the numbers per model during the import, along with other statistics and warnings.

Here are some screenshots with number of triangles displayed in the log window in the upper part of the window in various settings. Note this was highly unoptimized - no occlusion (amounts of polygons behind the hills), shadow polygons rendered even when not visible etc., but it can give an idea what is rendered.

complete (3,061,618 triangles)
(http://www.outerra.com/images/a-complete.jpg) (http://www.outerra.com/images/a-complete.jpg)

wireframe terrain
(http://www.outerra.com/images/a-wireframe.jpg) (http://www.outerra.com/images/a-wireframe.jpg)

no trees (2.127.728 triangles)
(http://www.outerra.com/images/a-notrees.jpg) (http://www.outerra.com/images/a-notrees.jpg)

no shadows (1.496.808 triangles)
(http://www.outerra.com/images/a-noshadows.jpg) (http://www.outerra.com/images/a-noshadows.jpg)

Lukla (4.007.780 triangles)
(http://www.outerra.com/images/a-lukla.jpg) (http://www.outerra.com/images/a-lukla.jpg)
Title: Asynchronous Job Scheduler
Post by: cameni on September 27, 2010, 08:20:06 am
Quote from: corona
Hi cameni? (having an identity crisis there?)
Yeah, decided to use a single identity everywhere :)
Title: Asynchronous Job Scheduler
Post by: corona on September 27, 2010, 09:56:58 am
Now I have an observation, looking at the fps stats, it appears neither trees nor shadows help at all? I realize its highly unscientific as we don't know the current fps counter, only avg/max/min, but its an observation never the less.

I also realize this is very much in development, but seeing low 30s on average on a pretty empty world with your, presumingly powerful system, is a bit worrying I must say. Of course I know its probably a debug build (which for those who dont know can significantly lower performance even on trivial code, let alone a graphics engine), and far from fully optimized at that. I guess my point is rather to stop people from jumping to conclusions from those numbers.
Title: Asynchronous Job Scheduler
Post by: cameni on September 27, 2010, 10:04:27 am
It's clamped to 30fps.
Otherwise removing some 400,000 trees would surely be noticeable :)

PS: And that fps doesn't in fact show fps but frame times in ms. Ask angrypig why. So please don't draw conclusions from it :D
Title: Asynchronous Job Scheduler
Post by: Blockaderunner on September 27, 2010, 10:10:52 am
wow. 30fps with 400k trees and 4kk triangles. not bad.

so is it a FPmS? :)
Title: Asynchronous Job Scheduler
Post by: corona on September 27, 2010, 10:14:14 am
Ah, that explains it. I also like that you support such a thing, as consistancy is more important that raw fps numbers as far as perceived performance goes. Especially in simulation style games, at least thats my observation.

Also, it just so happens that 33ms/frame ~ 30fps......hehe :-)
Title: Asynchronous Job Scheduler
Post by: corona on September 27, 2010, 10:34:43 am
I have another question. Back in college when I wrote little games (nothing at all like this obvsiously) what I did to track perf was basically log timings of various subcomponents, so far perfectly understandable.

Now how do you accuratly time highly threaded code inside a jobmanager, given that any thread can the paused/interrupted by a higher priority one in the middle of it, thus you can't just take endticks-startticks.
I guess you use a completely custom threadpool implementation, as I don't see how else you could get callbacks on job interrupts. I'm bad at explaining this.

job a is running at priority normal
half way through, you queue another job b, also at priority normal, and another job c, at priority high

do you have to wait for job a to finish, then process c, followed by b,
or (hopefully at least) job a will be halted, c processed immediatly, then a continued and then b.
or is a round-robin kinda thing, where each job gets x-amount of processor time relative to their priority, before being halted

I guess wether option 2 or 3 are preferable depends on what you define as a job, weather those are one off jobs, like load model x, download weather updates, or if jobs are continues loops, like processes phyiscs.

I'm rambling, sorry.
Title: Asynchronous Job Scheduler
Post by: cameni on September 27, 2010, 10:50:11 am
Well the jobs can be suspended only in the blocking code, when they'd have to wait for I/O anyway. Prioritization is decided there. Although we don't have explicit priorities on jobs, there can be implicit ones, respecting those will lead to a faster completion.

It's similar in behavior to the normal OS scheduler, except that we control how many jobs will effectively run in parallel so as to not overload the system, that would affect the overall smoothness in the end.

But to answer your question, the timing shown in the log windows isn't related to these jobs. The measurements are done for GPU jobs, either for the renderer thread or for GPU-side jobs outside of the main rendering loop (used to generate the geometry and textures etc).
Asynchronous CPU-side jobs are used to load textures, models and other data from disk or network.
Title: Asynchronous Job Scheduler
Post by: giallanon on May 27, 2011, 10:50:16 am
Are you using an external library to manage multithreading, or just plain old _beginthreadex?
Thank you
Title: Asynchronous Job Scheduler
Post by: cameni on May 27, 2011, 11:25:05 am
We have a wrapper library that hides some OS specifics, so that it can run on both Windows and Linux. Pthreads can be used for that purpose too.