Outerra forum

Please login or register.

Login with username, password and session length
Advanced search  

News:

Download Outerra Tech Demo. Unofficial Outerra Discord server, MicroProse Discord server for OWS.

Author Topic: Blue Marble  (Read 18479 times)

KelvinNZ

  • Hero Member
  • *****
  • Posts: 509
  • A clock that doesn't work still works twice a day!
Blue Marble
« on: January 15, 2013, 02:25:50 am »

Hi Brano,

Ok, i've been reading about the "Blue Marble" and I am wondering how you will be implementing it with possible biomes; water, vegetation, roads, railways etc...

Will Blue Marble just colour the earth regardless of where these land features are there or not? For example with Blue Marble in place and no vegetation in place will blue marble  just colour those areas anyway with the land features (perhaps vector data) just sitting ontop?

Will this also mean that the water will be coloured using Blue Marble or a separate process given the different water shades for different seasons, times of day i.e sun reflection and cloud cover impacting it.

I am interested to know how you will colour the trees and water and whether they will be placed just using vector data or some sort of overlay on top of Blue Marble. Also, will the engine, by default, have seasons and if so how will this work with Blue Marble or will that involve a colour shader/generator based on time of year?

I realise these are loaded questions but they will i'm sure help with mine and others' ideas about how these natural processes will translate into a virtual world and what challenges will arise. I guess all this requires some really clever coding methods!?

Cheers,

Kelvin
Logged
If a man who cannot count finds a four-leaf clover, is he lucky?

cameni

  • Brano Kemen
  • Outerra Administrator
  • Hero Member
  • *****
  • Posts: 6721
  • No sense of urgency.
    • outerra.com
Re: Blue Marble
« Reply #1 on: January 15, 2013, 12:02:48 pm »

This needs a longer explanation. The process is actually bit complicated and the algorithms are in development and so the exact form is still being refined.

Apart from the Blue Marble data there's also the MODIS dataset that contains land (and other) classification in a discrete form. As I understand it, it was created in the same process than Blue Marble, using several more infrared bands of collected data. The part that interests us most is the dataset that divides land type into 14-some classes. I think I have already shown it somewhere ... dunno where, reuploading:



This is available with 500m resolution just as the BMNG color data.
This representation more suitable for the engine, as it can serve data for the generators of vegetation and terrain textures. But if we used it directly, the world would consist of flat biomes and sharp transitions between them, which is not what we want.

What we want is this:
  • the colors should match the Blue Marble when viewed from above (+ atmospheric effects)
  • transitions between the biomes should be natural, fractal like
    you don't get a grass that's 30% color of desert and 70% color of savanna at some point in transition, you get 30% grass coverage there instead


Fractal mixer is already driven by the ratio parameter when generating textures and terrain data, as can be seen on this older screenshot of how the transitions are created:



The biome mixer works on the same principle but on a larger scale. It requires data prepared in a special way, basically land types with probability weights that are then fed into the fractal mixer that resolves the land type together with its colorization.
How to get these data by transforming the source land type and color input maps is the major problem. The algorithm basically tries to find an optimal mix of input land-type ingredients that can produce the observed color and maximizes the probability that given composition can occur at given place, constrained by other parameters such as min/avg temperature, precipitation etc.
It's a multidimensional problem of finding the best fit.

Seasons are an integral part of it, once you have the base types defined over the seasons (like, birch cover defined as a vector of colors over seasons), you are going to find whether the birch would explain the observed variation of colors at a particular place over the whole year, and what part it would cover.

Obviously, interpreting color data with the help of auxiliary datasets (major land type, temperature averages and extremes, precipitation, sunshine % etc) is quite demanding both algorithmically and on the processing power, and it also depends a lot on the quality of the input base types given to it to work with. It must be able to provide a quality feedback that tells how well it was able to match it, best suggesting the parameters of possibly missing land types if the deviation is too high. It has to avoid urban areas while analyzing but not when generating the output dataset.

There's also an alternative way that treats the data differently, in somewhat simpler manner. Unlike the first method it primarily considers the discrete land type dataset and computes the weighted land types needed by the biome mixer by analyzing the surrounding terrain types and computing the composition from that. Additionally it takes the color data and encodes it as a color difference against the computed composition color. This brings back the color variation that exists normally (for example, the changing color of the deserts) but would have to be achieved by defining multiple types of desert material using the first approach, increasing its complexity even more (though the result would probably be nicer).

I think ultimately it's going to be a combination of the two. While the first one allows to match real color data if these are available, the second one can be used also with hand-drawn climate maps or generated climate models, usable also by virtual world creators and scientifically. Combining them means to use the second approach to quickly restrict the size of potential land cover candidates, and then refine it by the first method locally.

At last (but not the least) it should also be effective at run time ;)

Not sure about the water coloring, but intuitively it could use the color part, decomposing it again into the light scattering parameters of water. We aren't coloring the water directly, it gets its color from light scattering in order to work correctly at angles and underwater. To match a given color you'd have to find the root elements again.

Also, while the processes above can match the seasonal data and produce our biome dataset, implementation of actual seasons is relatively independent thing, involving rendering of trees and vegetation states and a dynamic snow layer etc.
Logged

KelvinNZ

  • Hero Member
  • *****
  • Posts: 509
  • A clock that doesn't work still works twice a day!
Re: Blue Marble
« Reply #2 on: January 15, 2013, 03:41:43 pm »

Once again thanks for your explanation Brano.

Further questions arise for me and I have added an idea below if you wish to read it

1) How will multiple land classes that exist in real world be added to one Blue Marble area when in one area it has a certain shading to it?

2) How will you implement spacially placing objects/land elements globally and randomly to sit at certain ground levels and will this be for 3rd party developers to utilise?

3) Will you have every vegetation and land class feature available in the engine by default as per real world, if not, how will missed ones be added in future?

4) Will global vector data for shorelines, roads, and railways be implemented in the engine by default or will there be separate data that end users can "copy into a directory" but you will provide that are translated by the engine (this has never really been clear for me).

5) Will you add a feature into the engine to allow an end user to create, for example, 25,000 world airports by using co-ordinates for each runway and have the engine read this? is there a file type that can be read by the engine and manipulated by the end user to add this data into some sort of table then place the runways (in this case) over the globe? Or even more specific how will multiple buildings such as airports be added globally or would this be a job for 3rd party developers and would the engine just provide to tools?

6) do you ever get to relax after such mind intensive work?

My idea for Blue Marble and land classification:
Assuming you haven't thought of this would the Biome mixer allow for tagging the Blue Marble data so that each different shade of colour not only represents a certain vegetation type for example but also tells the mixer what to put in a specific place so that if there were two different types of vegetation or other elements in one place it could apply those two things based on the corresponding tag for the Blue Marble colour?

Blue Marble could have "tagged" areas. This way you could input more than just one corresponding vegetation type or even rocks, small grassy areas, foliage etc... into one area of the Blue Marble section that may not exist with the colour data alone.

If this basic idea worked wouldn't this allow some flexibility to add other elements in those areas in terms of spacially placing different vegetations and landclass types globally and randomly within a specific type of area I.E forests could have small bushes and broken branches in them as well to increase immersion but will be added to the Biome colour using the idea of tagging.

The tagging could allow for certain objects to be placed randomly in certain areas of the globe based on what an elements tag is. There could even be group tagging to say that this set of objects fall into x tag group so that when they are perhaps put into a mixer they would be placed globally and randomly. Obviously there will be certain heights at which these items would be placed so this would need consideration.

I suppose you could call "axillary datasets" by their tag so everything has a tag that can be called  in each algorithim that way tags could be increased and a standard is set on how to place multiple elements using Blue Marble so that the world won't be too linear in each area because in real life we all know there are random little bushes here and there, shelter belts and other random foliage all over the place when it comes to specific land classification areas.

If there are areas in the world that naturally have had fossils, ancient buildings, relics, etc... they could be placed there. Now this is starting to move outside of the engine itself but with this concept a reality in the engine it could allow for further 3rd party expansion.

Just an idea really and you may already have your way to do things I understand you have spent a good part of your time working out.

Water, yes, I see how this is coloured, obviously due to the absorption and scattering of light.

So I do understand the angle you are coming from which calls for a best fit approach constrained by hardware limitations and coding efficiency. I guess it is finding the best fit when it comes to land features as each aspect of the real world you want to simulate excluding one or two would entail a cpu thread to start working harder.

Thanks for your time to answer my questions and explain the ins and outs of the engine as we all enjoy absorbing your science inside the Outerra engine during its maturity. That's it for now so i'll leave you to get back to line 990,000,000,000,000 of your code :)

Regards,
Kelvin
« Last Edit: January 15, 2013, 03:55:22 pm by Kelvinr »
Logged
If a man who cannot count finds a four-leaf clover, is he lucky?

cameni

  • Brano Kemen
  • Outerra Administrator
  • Hero Member
  • *****
  • Posts: 6721
  • No sense of urgency.
    • outerra.com
Re: Blue Marble
« Reply #3 on: January 15, 2013, 05:27:56 pm »

Quote
1) How will multiple land classes that exist in real world be added to one Blue Marble area when in one area it has a certain shading to it?
That's why the compiled dataset will also carry the color difference for given place, applied over the created textures and vegetation.

Quote
2) How will you implement spacially placing objects/land elements globally and randomly to sit at certain ground levels and will this be for 3rd party developers to utilise?
I think you need to realize that what we are talking here about is the underlying natural world. It's not aiming to produce the exact look of our planet in its current state, but to create the world that can be taken for Earth at some time.

Atop of this world goes a modification layer that is vector based and can be created by other parties. This layer is for things that are not natural and are produced by the civilization, and as such have to be explicitly modeled - parks, fields, pastures ...
Thing is, you can usually recognize them well because they are artificial. If they were distributed and intermixed naturally, you would usually not be able to tell if it corresponds to reality or not, unless you knew the place very well. For natural appearance a random fractal distribution that statistically corresponds to some global data is a good fit.

Quote
3) Will you have every vegetation and land class feature available in the engine by default as per real world, if not, how will missed ones be added in future?
Obviously we can't have every one, so there must be a mechanism to add additional ones. There are two ways: adding new base types and compiling the biome dataset again, or using the custom vector overlay to add them where needed.
Tools for the vector overlay will be normally accessible, compiling new natural datasets is more tricky and produces large data, but may become available later with a separate planet creator package.

Quote
4) Will global vector data for shorelines, roads, and railways be implemented in the engine by default or will there be separate data that end users can "copy into a directory" but you will provide that are translated by the engine (this has never really been clear for me).
These are all vector data, when you create your roads now in Anteworld they are stored in cache/earth/ dir structure and you can distribute the files to someone who can copy them to their disk and will have them. Of course, ultimately it will require a better management of multiple such layers that can be switched on and off.

Quote
5) Will you add a feature into the engine to allow an end user to create, for example, 25,000 world airports by using co-ordinates for each runway and have the engine read this? is there a file type that can be read by the engine and manipulated by the end user to add this data into some sort of table then place the runways (in this case) over the globe? Or even more specific how will multiple buildings such as airports be added globally or would this be a job for 3rd party developers and would the engine just provide to tools?
Right now there's just the in-game editor for creation of the data, but there will be also an importer that can do it from some external formats. That means 3rd party developers will be able to import and edit their data (external datasets usually do not contain everything needed by OT).
Using them means installing the data for them and enabling the layer.

Quote
My idea for Blue Marble and land classification:
Assuming you haven't thought of this would the Biome mixer allow for tagging the Blue Marble data so that each different shade of colour not only represents a certain vegetation type for example but also tells the mixer what to put in a specific place so that if there were two different types of vegetation or other elements in one place it could apply those two things based on the corresponding tag for the Blue Marble colour?

Blue Marble could have "tagged" areas. This way you could input more than just one corresponding vegetation type or even rocks, small grassy areas, foliage etc... into one area of the Blue Marble section that may not exist with the colour data alone.

If this basic idea worked wouldn't this allow some flexibility to add other elements in those areas in terms of spacially placing different vegetations and landclass types globally and randomly within a specific type of area I.E forests could have small bushes and broken branches in them as well to increase immersion but will be added to the Biome colour using the idea of tagging.

The tagging could allow for certain objects to be placed randomly in certain areas of the globe based on what an elements tag is. There could even be group tagging to say that this set of objects fall into x tag group so that when they are perhaps put into a mixer they would be placed globally and randomly. Obviously there will be certain heights at which these items would be placed so this would need consideration.

I suppose you could call "axillary datasets" by their tag so everything has a tag that can be called  in each algorithim that way tags could be increased and a standard is set on how to place multiple elements using Blue Marble so that the world won't be too linear in each area because in real life we all know there are random little bushes here and there, shelter belts and other random foliage all over the place when it comes to specific land classification areas.
Since the resolution of the land type data is 500m, it would be quite monotonous if it was just flatly used for one type. I didn't go into detail here but the mixer has several stages in order to generate the detail for all levels. The land type itself is a composite of several materials. Just as now you have patches of dirt in grass and fractal color modulation on the vegetation (you would immediately see how flat it would be without it), there will be a variety of sub-materials from which the land type is composed. I suppose it will be possible to define and modify what comprises a land type and how the low-level material mixers are set up for it.

Quote
6) do you ever get to relax after such mind intensive work?
At least I can sleep well after tiring my brain :)
Logged

KelvinNZ

  • Hero Member
  • *****
  • Posts: 509
  • A clock that doesn't work still works twice a day!
Re: Blue Marble
« Reply #4 on: January 15, 2013, 08:25:20 pm »

Thank you for your time Brano, i'm running out of questions so I will rest on those answers that you have provided as they do help me to gain further knowledge of the underlying mechanisms that will drive Outerra engine.
Logged
If a man who cannot count finds a four-leaf clover, is he lucky?

krz9000

  • Member
  • **
  • Posts: 62
Re: Blue Marble
« Reply #5 on: January 16, 2013, 07:04:46 am »

I'm a big fan of openstreetmap. Could osm be an editor for outerra? In the end outerra would be a viewer for osm data...which I think is great since there are so many people feeding the dataset.
Logged

cameni

  • Brano Kemen
  • Outerra Administrator
  • Hero Member
  • *****
  • Posts: 6721
  • No sense of urgency.
    • outerra.com
Re: Blue Marble
« Reply #6 on: January 16, 2013, 08:33:42 am »

OSM could be a source for the vector overlay. An OSM parser is in the plan.
Logged

KelvinNZ

  • Hero Member
  • *****
  • Posts: 509
  • A clock that doesn't work still works twice a day!
Re: Blue Marble
« Reply #7 on: January 16, 2013, 03:52:06 pm »

I know Open Street Maps does have its inaccuracies. This information below is available in different map projections and data format also.

Not only are roads available so are shelter belts, airport polygons and fencelines, tree points, rivers, lakes, coastlines and more. I suppose it is the workload involved in getting this actually converted into useable Outerra data.

Vector data format
http://i.minus.com/iNFX0jO2wGoAd.jpg

Projections
http://i.minus.com/iSwvvJQPiOYYT.jpg

Main site
http://data.linz.govt.nz/search/category/roads-and-addresses/street-and-places-index/#/search/category/roads-and-addresses/?page=2&s=a

Cameni, could Outerra be overridden so developers can create a more accurate depiction of real world elements such as the ones above? How will you add this as a function for developers?
« Last Edit: January 16, 2013, 04:07:38 pm by Kelvinr »
Logged
If a man who cannot count finds a four-leaf clover, is he lucky?

Steve.Wilson

  • Full Member
  • ***
  • Posts: 176
  • Home Cockpit Builder
Re: Blue Marble
« Reply #8 on: January 16, 2013, 04:44:34 pm »

OSM data that can be corrected or modified at the user level would be most appreciated.  Laminar Research uses OSM for X-Plane 10, and there's a lot of missing and inaccurate data, especially for the US.  Corrections require a complete DSF tile recut, and that's just too far above the head of most users.  All for OSM, but if it can be kept as a separate database referred to at runtime, or at least easily rebuilt locally with a simple script, that would be a real plus for Outerra.
Logged

krz9000

  • Member
  • **
  • Posts: 62
Re: Blue Marble
« Reply #9 on: January 16, 2013, 04:58:53 pm »

that would be a dream come true...to be able to download OSM datasets and a outerra script converting the data to its internal formats.

i wouldnt worry about inaccuracies of OSM...these can be fixed "easily" (just takes the will to put in the correct data,..everyone can help). in the area i live the data is better then googles (down to vegetation, floor levels of buildings,...). the main OSM editor is very easy to use.
Logged

cameni

  • Brano Kemen
  • Outerra Administrator
  • Hero Member
  • *****
  • Posts: 6721
  • No sense of urgency.
    • outerra.com
Re: Blue Marble
« Reply #10 on: January 16, 2013, 05:02:37 pm »

OT uses its own vector data formats that are tailored for an effective use on the GPU. The engine already includes functionality to define these vector data using for example longitude and latitude coordinates, or ECEF coordinates. That means technically you are able to import road waypoints from other sources this way already, although right now it's only accessible inside the engine.

With a proper API you'd be able to write importer for various vector data sources. We would like to write one for OSM ourselves, since it's expected to be one of the main sources of vector data for OT world and should be optimized. And it will be probably processed regularly, so in case you find some errors, you could correct it in OSM and it would appear in next update, automatically replicating from the server.
Logged

KelvinNZ

  • Hero Member
  • *****
  • Posts: 509
  • A clock that doesn't work still works twice a day!
Re: Blue Marble
« Reply #11 on: January 16, 2013, 06:00:29 pm »

That sounds viable and if there was a vector data tool then this would make it so much easier to use an inbuilt tool to do this, all that the 3rd party would then need to do is put in the time to enter data.

Would this mean that a "vector data importer" will be available by default or will the engine just know how to understand it and 3rd party will need to create the tool?

I have noted the vector data from NZ so when OSM is used could it be overridden so that specific data can be used instead? I understand the OSM doesn't cater very well to some regions of the world but I understand that it is probably the most available data of the globe that can be used.
Logged
If a man who cannot count finds a four-leaf clover, is he lucky?

cameni

  • Brano Kemen
  • Outerra Administrator
  • Hero Member
  • *****
  • Posts: 6721
  • No sense of urgency.
    • outerra.com
Re: Blue Marble
« Reply #12 on: January 17, 2013, 01:45:28 am »

Would this mean that a "vector data importer" will be available by default or will the engine just know how to understand it and 3rd party will need to create the tool?
It's hard to write a universal importer that would be able to import any data out there, that's why I wrote we would create an API and let others write their tools using it. Actually, it may be a bit more complicated. For example the tool that imports elevation data is written in universal way in that it expects some layered sources of tiled data that can be registered into it with different priorities. The tool then handles the import process (remapping and compression) by itself, but it uses a standardized interface to talk to the source data providers during the run.
The reason for this is that it makes it very effective (it can process the whole earth in an hour) because it runs on GPU, uses asynchronous IO etc, and it allows the source data providers to remain simple.

I guess it will be similar with vector data processing.

Quote
I have noted the vector data from NZ so when OSM is used could it be overridden so that specific data can be used instead? I understand the OSM doesn't cater very well to some regions of the world but I understand that it is probably the most available data of the globe that can be used.
OSM license terms are infectious in the sense that if the OSM data were compiled into a single dataset together with other data, all of the data would have to have to be available under the OSM license. That practically means OSM has to be separate anyway, and you'd need the ability to turn it off in regions, possibly enabling other layers with other sources.
Logged

KelvinNZ

  • Hero Member
  • *****
  • Posts: 509
  • A clock that doesn't work still works twice a day!
Re: Blue Marble
« Reply #13 on: January 17, 2013, 02:09:02 am »

That all makes sense.
Logged
If a man who cannot count finds a four-leaf clover, is he lucky?

Midviki

  • Full Member
  • ***
  • Posts: 242
  • Nothing matters... but everything is important.
Re: Blue Marble
« Reply #14 on: January 26, 2013, 09:58:03 pm »

So I do understand the angle you are coming from which calls for a best fit approach constrained by hardware limitations and coding efficiency. I guess it is finding the best fit when it comes to land features as each aspect of the real world you want to simulate excluding one or two would entail a cpu thread to start working harder.

"hardware limitations" ...

That started an idea in me.

And I want to point out that time is flying and thing will get out of control.Lets assume that till 2015 you will get a beta version out and 2017 you will finish the product.In 3 more years it is 2020!!!

Do you know how the tech will look in 2020?... Well I can tell you how it looks now:

1) Now days a home PC can easily support like : 32GB DDR ram, a 4 core proc of 4GB ( that's 16gb ) , 256bit video.And this is all now.And it is not too expensive what I just said.If u'r like 18+ years old that computer is easy to get.But think about it if you will still use Outerra in 2020.The computers now will look ancient.And you will probably have 10 times better.

2) Fact! is that every year companies compete.And each year there is a new model... *New Ram*, *New CPU* , *New Video*.Well even if you will not afford the new hardware from 2020.. you will use the one from 2017.Cause it will be cheaper.But it is still better with 4 years from the best tech now!

3) Thing from how I see them are getting GREEN.And when I say green... I say nano tech.Fewer nano's equals lesser power consumption, lesser heat emission from the hardware.That means you can add more!No.. let me correct myself.They will want you to add more.Why?Because you can!So basically you will have a behemoth computer with more muscle than ever.


-----------------------------------

And now that I have made my point.I want to point out a thing or two about the biomes...

a) You will want to upgrade them!

Hmmm... because?..

b) At one point in time you will want them to be "Movie quality".Because it is not deniable!.. We will always want more!


c) Hmmm... the color "thingy" is a bit overrated.( Ex: "Yay!.. Lets go on Mars and paint it all RED and pretty!" )

People will get bored of Earth.Its like the casual hillbilly will say: ... "f*** that... we already got that s*!t, We want Pandora!"


d) There must be better and more powerful solutions.I'm not saying that the one you will want to do is bad.But think of the future.Think bigger.
Think of the user that will test / play the game and will want to have his own personal tree, with personal shapes, and even make his own string DNA for it and breed it for the world.And think about the hardware evolution that will be in the future.Think about the amazing jungles, filled with all kinds of stuff and life.Imagine the moving sand from the peaks of dunes in the desert, when you step on them... all the sand will fall gracefully!.. and all the sand storms you will see!!!I dare you to think bigger!


That being said... We are a bunch of minds around here.A brainstorm would be appropriate!Outerra must be unlimited!We need some solutions to make things to be unlimited and evolve once with time!I dare you to squeeze your brains in to something beautiful and flawless for an eternal Outerra experience.
Logged
ATI AMD RADEON HD 6670