Outerra forum

Outerra Engine => Off Topic => Topic started by: Redrobes on February 25, 2011, 03:31:08 pm

Title: Cartographers stuff
Post by: Redrobes on February 25, 2011, 03:31:08 pm
(note: split from here (http://www.outerra.com/forum/viewtopic.php?pid=3106#p3106))

No public demo is a shame but I understand. I also process images into ViewingDales own format for the same reason. It makes them easier
to render in OpenGL and it contains meta data thats static to the image and not worth recomputing on the fly.

We have a base DEM which is built from pure vector contours and is exported (via Global Mapper I believe) into a rastered pixel based height map. Its in float values or 16bit height integers. Again we use a format that we have had some input into generating. Its the HF2 format which was knocked out on another forum of terrain devs. There is an LGPL implementation of it from Aaron of L3DT if your interested in that. All the other DEM formats had one issue or another that made it unsuitable. This one is designed to be unlimited in scale so worth a look I think.

All the data is geo-referenced. Monks - who is going to sign up here too, will know more about that. Tolkien said in his letters that Middle Earth was based on Earth and that there are some matching places like Hobbiton being in Oxford where Tolkien studied etc. Anyway, we have mapped the data set to earth so I think that part is already covered.

I know we can export a physical resolution of 20kpix square which is about 200m. The intention is to take this into my GeoTerSys which is a terrain synthesis tool based on a simulation and run that. The main idea of GTS is so that it takes a rough terrain and makes it more realistic and up's the res of it by running a geological / climate simulation on it. See images on the medem site. Starting images are the deliberately rough ones. Anyway, the idea was that we would like 100 kpix per side which is about 40m or so. I am sure that is a lot of data tho - many Gigs at least. We like the sound of what you have where a) it zooms by managing LODs and b) fills in lacking res with sensible fractal stuff. We have about 100 layers of information we can use to guide shaders. We were going to use them to drive masks for Terragens procedural shaders.

As for the Tolkien Estate. Yeah maybe. Our project is all free - its fan based non commercial thing here. So the data is a give away thing and you can take it and try it and make some movies from it. Its built to match the shape of their map and the names have to match of course, but aside from that, it does not use anything from any commercial works. We would like one day tho to have a web site with a download of engine and data and allow people to fly through it. Maybe get a community thing going with the buildings from sketchup or something like that. We have found the generation of the actual DEM too painful and technical for general community spirit based work so it leaves a few poor souls like Monks and I to do most of it.

We can fly through it in top down Ortho only using ViewingDale as a map browser but the project is inherently 3D so that kinda misses the point.

We saw some vids of YouTube where someone was editing the height map in real time at all scales right up to continental. I don't know if that is part of this project or an off shoot of it. Certainly that would be very interesting for us. Guiding rivers to flow to the correct place dynamically would speed up our process immensely.

EDIT -- Link to HF2 specs
http://www.bundysoft.com/docs/doku.php?id=l3dt:formats:specs:hf2

Edit2 -- If your just into the maps from medem try this one:
http://me-dem.me.uk/galleries/vdale/MeDEM_Map1.jpg
As you might tell the full sized one is much bigger.

And thanks for the replies - ill say that here to keep the thread clean.
Title: Cartographers stuff
Post by: C. Shawn Smith on February 25, 2011, 04:03:00 pm
I've been a regular of the Cartographer's Guild for a little while.  There's some really cool stuff there.  I even wrote a PDF tutorial on the creation of realistically rendered maps, which is available on my website under the Photoshop tutorials link at the bottom.

I'm a writer, so the Outerra engine is interesting to me for the possibilities it may have in the future for rendering my fantasy world in realistic 3d.  I'd love to see a Middle Earth rendition within the engine as well :)

I'll have to check out your sites regarding Middle Earth.  I haven't seen a good rendering yet, and it's one of the worlds that is dear to me as a fantasy writer/reader :)

Welcome to the forums!
Title: Cartographers stuff
Post by: cameni on February 25, 2011, 04:21:52 pm
Thanks for the explanation. It seems you are all ready and knowing what it takes ;)
I'd be glad to see this project running in Outerra.
But ..

The Tolkien Estate - well the problem is that they are also after the non-commercial activities, after anything that refers to the Tolkien's name or work. They may see this project to be undermining their other commercial efforts, efforts that bring them money. They will surely try to stop it, from what I've read about their ways. That's why I'm reluctant to deal with any of Tolkien's work. A shame, but the monopolistic protection in the form of sw patents and IP laws seems to be a common problem nowadays in our part of the world, one which is almost impossible to solve without also letting the Rome to fall. But I digress :D

Realtime editing of the planet at all scales isn't a part of the project yet, but it's precisely the thing we want to do with it in this direction.
Title: Cartographers stuff
Post by: monks on February 25, 2011, 04:48:46 pm
Hi Cameni, you made me pretty excited when I saw your YouTube video, because this technology for modelling/ visualisation is what we've been looking for for quite some time. I've been with the ME-DEM project for about 6 or 7 years. Our web admin is Oshyan Greene who is with the Terragen team.

 Our terrain can basically be output at different resolutions. We're currently using a Global Mapper -> World Machine Pro -> GTS workflow (although WM can be skipped at a pinch for quicker demos).  World Machine will be used to process the terrain to it's full form.
 The shots in the gallery of the large scale terrain show what we term the "base dem"- the rough skeleton generated from contours- the base dem was a necessary step because using contours was the only way to guarantee correct flow across such a large model surface. Other modelling techniques don't really cut it.
 So, the base dem has no fractal detail or erosion on it.

 The Global mapper database is the persistent core if you like. We currently have around 150 layers of vector info. They get output as distrib maps with terrain. With projection info we can roundtrip between GM and other modelling software.

 The intention was to hook up GTS output directly to TG via XML. That way we get fluid sim generated texture maps direct to the renderer. GTS can also output "mega textures" to Viewing Dale too I beleive.

 The terrain was built from the outset to be georeffed and the river flow to be correct across the terrain in accoradance with the topo maps we are using as reference. The terrain is georeffed to Hobbiton = Sarehole in Birmingham and Minas Tirith = Florence, Italy. Surprisingly, many people are not aware that Middle Earth is Earth, but with an alternate history and Tolkien states unequivovcally in his letters that these two places are equivalent.
Lucky for us!  :D

 For large datasets we use the hf2 format which has compression and georeffed header options. It was created on the Terrain Summit forums which Oshayn set up (unfortunately gone now) by Aaron Torpy, RedRobes, Stephen Schimitt (World Machine), Mike (Global Mapper), Joe Slayton (Wilbur), Ray Gardner (Leveller), Johnanes Rosenberg (GeoControl).
 We've worked with these developers on and off because we came up against so many problems with trying to do what we wanted with this project. As a result I've tested for all of these softwares. The format was really the (unusual perhaps) meeting of GIS and game terrain devs/modellers, in part what the Terrain Summit was setup for.

 Hey cshawn..what a coincidence. I've got a thread over there on CartoGuild under ME-DEM.  :D
 No, not really anything in terms of renders as yet for ME-DEM.

The Tolkien Eastate, hmmm.. I remember the Minas Tirith Project which had a high profile on all of the CG boards, and I don't recall ever reading a single post about problems with the Estate, and it was right ontop of the films too but yes, I'm aware of the problems many people have had.
 Well, if it came to it, I'd be happy to bunker down and stay off the radar. With or without a web presence I'd still be doing this stuff.  :P I'd also be happy to get involved with other world building projects.

monks
Title: Cartographers stuff
Post by: cameni on February 25, 2011, 05:16:03 pm
Hi, monks. What video was that? Earth flyby?
Btw guys, thanks for the note about GTS - I wasn't aware of its existence, and it seems to be very interesting tool. I'll look at the hf2 format as well. It is a pleasure to talk with guys that already walked their path and know things :)
Title: Cartographers stuff
Post by: monks on February 25, 2011, 06:48:50 pm
No, well they're all amazing...but this more than anything:

Rendering V.
http://www.youtube.com/watch?v=BM7Pz7CgTnc&feature=related

 I remember talking with Joe Slayton about this kind of setup a few years ago now where you would paint directly onto a sphere, or a spherical projection. The inverse projection needed.  What you've done there is amazing, We told ourselves at ME-DEM that it was only a matter of time really. You look on the Ogre boards for example, there's a couple on there. And there's been a crop of procedural planetary renderers over the last 5 years or so. But purely proc renderers are not much help to us.
 Also, everyone (the software) was using bitmaps. It was obvious, looking at the GIS world, that everyone needed vectors. World Machine stole a march on that (as far as the indie devs went) with its splines. VERY cool way to work that.
 I modelled the Misty Mts on the European Alps by extracting the ridge lines using a World Machine plugin created by Howard Zhou. It was able to isolate ridges or valleys at user defined levels of complexity. I then used those to trace the splines when I was testing for WM Pro 2. It was a novel idea. I think the next step is to get full import of shapefile data into WM. It would make the whole process easier.
 I could have used the real dem data of course with blending but I wanted some control over the terrain as orders of complexity (ie ridges and valleys at various detail). Kinda a vector-like representation of terrain. I think you could probably see a world in terms of watersheds.

 ..and the one where you model those roads across that little hill with vectors... :cool:

 If you ever want anyone to test a terrain editor like that, let me know  :D ...I would be very happy to help.

monks
Title: Cartographers stuff
Post by: ZeosPantera on February 25, 2011, 11:11:00 pm
Quote from: cshawnsmith
I'd love to see a Middle Earth rendition within the engine as well :)


You know Cameni.. I'm sure nobody would mind if perhaps an island appeared somewhere near Australia that just happened to be shaped and built like middle earth.. I am not sure of the actual square mile footprint required but what an interesting adventure it would be crossing the sea of Krabgondia and landing on the shores of Gondor. I'm sure the google earth overview could even switch to their maps when you got close enough.

I could storm the black gates in a "Tatra-ish" vehicle!
Title: Cartographers stuff
Post by: cameni on February 26, 2011, 06:23:50 am
Quote from: monks
No, well they're all amazing...but this more than anything:

Rendering V.
http://www.youtube.com/watch?v=BM7Pz7CgTnc&feature=related

That's not our work but the work of Eric Bruneton and his students. We didn't get to this phase yet, although the vector edits used for roads and pads are one of the tools that count there. I need to extend the generator functionality a bit before I can move on to realtime world editing, but it will come - initially with support for realtime crater creation. The problem here is to make it happen across all levels of detail. That video is from before the time Eric added fractal refinement, but I think he still cannot go down to grass with it anyway.

Quote
..and the one where you model those roads across that little hill with vectors... :cool:
I think you refer to one of Eric's videos here too :)
Our road placement video went through a forest, I think.
Title: Cartographers stuff
Post by: cameni on February 26, 2011, 06:36:30 am
Quote from: ZeosPantera
You know Cameni.. I'm sure nobody would mind if perhaps an island appeared somewhere near Australia that just happened to be shaped and built like middle earth.. I am not sure of the actual square mile footprint required but what an interesting adventure it would be crossing the sea of Krabgondia and landing on the shores of Gondor. I'm sure the google earth overview could even switch to their maps when you got close enough.!

Sure, we even tried this back in early times, but using a very crude heightmap of ME. And I know it would find many fans, no doubt about it.
The problem is that you can't mention it is Tolkien's world, or ME, or anything from there. And even if you changed the names, I remember Tolkien Estate going after someone making just the map with the same shapes of the mountains. But that's what one would expect from a lawyers' guild living off the stuff.

One Estate to rule them all,
One Estate to find them,
One Estate to bring them all
and in the darkness bind them.

:D
Title: Cartographers stuff
Post by: Redrobes on February 26, 2011, 07:05:04 am
Is that movie done with Outerra ? Its the one that ebuneton mentioned in the text. Or does he have a similar one and has copied your "fractal noise details". The text is not very clear. In any case that video or your videos are both the kind of thing that we would like to see.

If you would like us to play with the engine that would be great or if someone here who has it would like to take the data and play with it that would also be great. If people are worried about the linking of this engine with the middle earth then we could show some vids without mentioning the name or do something similar.

GTS was never released, its a bit of a toy app that I worked on for some years. I use Perl to generate the scripts and its managed by GnuMake to build the tile set up. It takes about 8 hours to run the simulation over all the tiles for a duration that gives some basic results and spits out all the layers which are then used in another app which is my programmable texture shader (on the CPU not GPU). The images in the galleries are 2D bitmaps from that. Those and the height map are put into the Dragon Flight 3D renderer which is real time but very small. Its a debugging tool so has no LODs or any kind of fractal noise or stuff like your wavelet compression etc. Thats free to download and try from here (http://www.viewing.ltd.uk/cgi-bin/viewingdale.pl?category=dragons_flight). Just run it in a directory with a height.bmp and an optional color.bmp file. It will do 16bit height as well but you need to create a modulo 256 bitmap and call it height_minor.bmp like the ones in the gallery. Heres cshawnsmiths island from the intro thread chucked into it.

(http://viewing.ltd.uk/Temp/CG/General/cshawnsmith/render1.png)
Title: Cartographers stuff
Post by: cameni on February 26, 2011, 08:10:03 am
Quote from: Redrobes
Is that movie done with Outerra ? Its the one that ebuneton mentioned in the text. Or does he have a similar one and has copied your "fractal noise details". The text is not very clear.
Nope, Eric has been only inspired by Outerra and added some fractal detail in his last video, but not to the detail level of OT. Eric is more interested in replicating various phenomena mathematically, as he said his primary goal is to publish research papers. His approaches are more "correct" in terms of being capable to match the actual behavior (as in atmospheric scattering or ocean rendering), but also more demanding.

Quote
If you would like us to play with the engine that would be great or if someone here who has it would like to take the data and play with it that would also be great. If people are worried about the linking of this engine with the middle earth then we could show some vids without mentioning the name or do something similar.

You got me wrong - I don't care what people would do with the app, I was just saying that I myself would be reluctant to take on ME because of TE. And that's from someone who's genuinely interested in making the worlds known from literary works. With TE it's simply almost inevitable that you will receive a cease&desist letter and the beautiful project would end up in hiding.

As for the testing - I hope well be able to test it with you once we get it past the release of game alpha. For the alpha we need to update the mapping tool to run on GPUs, accelerating it quite a bit. It currently takes somewhere around a week of hard 4-core work to process data for Earth.
I planned to support the creation of other worlds as "mods" for it anyway, to allow for Shawn to test out the fantasy world of his novels.

I guess GTS algorithms could be reworked to run on GPU, that would make it a very cool component of the future world creator based on OT engine.
Title: Cartographers stuff
Post by: Redrobes on February 26, 2011, 08:36:52 am
Ok - that sounds very promising then. We will definitely both be keeping our eye on the progress of the engine. I have always wanted a game where you can play in space and then drop down into worlds and explore. Its never happened yet. The closest I know is Mission 6 of Halo Reach - "Long Night of Solace". Exactly what I wanted but for just one planet and it had a fixed transition instead of the transition being interactive. Still, Bungies engine must be able to cope with planetary zooms. Worth looking at if you dont know what I mean.

GTS is written in C++ and the main data structures are already (planned from the start actually) formatted up in CUDA sized chunks. The algorithm is embarrassingly parallel so in theory it should be a short hop to OpenCL. Do you program in GLSL, CUDA or OpenCL ? I would consider putting it into the building of the terrain. Whether it could be done in real time I don't know so much. My issue for not going full CUDA already is that I think I would need to have huge IO instead of compute so I figured id get about a 5x speed improvement instead of the 100x that people show some things going at. If you have precomputed all the wavelets and cached them on the GPU then I guess your IO is much less than doing it all at full res. So I think my problem is closer to your 5 day job than the real time frame rate job. But yes, there are not too many people with code that computes the fluid flow over terrain. Its not a miraculous bit of code but its tricky enough. That Eric chap has a student who has got a video (http://www-ljk.imag.fr/Publications/Basilic/com.lmc.publi.PUBLI_Article@11e7a157f59_4402c7/river.avi) of river flow calculation in real time there. That is quite impressive too. But they are calculating the flow from preset  river banks. We want to know where to put the banks in the first place based upon the terrain shape. I still think that is harder.

Since I come across few people who program the GPU directly, id be interested in your thoughts about the compute vs IO of calculating fluid flow. I have not done much in the way of GPU shading and CUDA yet. Most of my apps use multiple cores of CPU. I was hoping that intel were going to do the Larabee with lots and lots of x86/x64 cores on one chip but they abandoned it. At the time of writing GTS which was a couple of years ago now, nVidia had the OpenCL drivers on the registered dev only so I could not get to use them and I was reluctant to use CUDA if OpenCL was going to become the standard for GPU coding. Once they opened up the drivers fully I was onto other things and it kinda got left out.
Title: Cartographers stuff
Post by: cameni on February 26, 2011, 09:20:49 am
You are right, if the computation is too light it wont bring up that much in comparison to the IO. But a 5x speedup is still desirable when the whole thing goes for a week. Additionally, there will be more computation when it will be merged with the wavelet decomposition and compression. Also, as you said, if the algo operated on pre-compressed data, recompressing on the fly, the IO would be smaller at the price of heavier computation reqs (but when the CPU/GPU would be idle waiting for disk IO anyway). Also, I'm much used to GPU programming now, on CPUs I usually can't resist the itching to optimize stuff somewhat prematurely :)

We are using solely GLSL inside the engine (or rather a custom compiler into GLSL that gets us rid of some of the aspects of its poor design). We have experience with CUDA from previous work, but for the tools we want to use OpenCL as they may end up in a world builder product one day and we don't want to limit the target platforms.

Rivers .. my idea is to let the designer to mark points on continent border where there will be river estuaries, and have the algorithm trace the path up the gradient, generating tributaries and ending up in mountain springs. Note this would run on a non-eroded terrain, and the generation of river basins would effectively create the erosion patterns. I have to dig up my old notes somewhere :)

I think this approach is more in line with the way I see the creation of an artificial world. I want to sketch the continents, draw mountain ranges with a simple tool there, place the volcanoes and lone mountains here and there, place the estuaries or the main paths of large river where I know I want them to be, and then let a generator chew on it, creating a believable world. Then a touch here and there, some similar edits on more detailed levels .. things like that.
Title: Cartographers stuff
Post by: C. Shawn Smith on February 26, 2011, 12:16:26 pm
Quote
Heres cshawnsmiths island from the intro thread chucked into it.

:D

My 3d applications basically showed the same thing ... and you can see very obvious flaws with the current painting of the height data (most especially around the coastal areas).  When the world tools are introduced, a lot of these problems will disappear if I'm still using this height map as a base.  However, Cameni's explanation of the tools would negate even needing to paint a height map .. which will be MUCH easier to deal with :).
Title: Cartographers stuff
Post by: Redrobes on February 26, 2011, 02:39:17 pm
I think that by using a real time terrain engine with some real time fluid flow you would be able to raise and lower the terrain height and steer rivers to the desired place. That would be a huge improvement on the way were doing it.

The problem with designating the outflow estuary and asking the computer to plot it backwards is that the function uphill is a one to many. Think of it downhill. All the rain in a certain catchment area all flows into the sea at the one estuary. Therefore plotting the river route uphill would be a very difficult problem. You cant take the path of highest ascent otherwise streams running down valleys would divert off to the side and straight up the walls of the valley. Also, rivers split going uphill so you would have to know the terrain ahead of the path in front of you. I think its easier to plot it going down hill where for every point you can say that water takes the line of steepest decent and joins to form larger rivers.

The complications arise when you make the terrain dynamic. If it erodes or the amount of water falling is not steady state then small changes in the erosion of the terrain can drastically change the river direction. Once a basin has filled and overflows, a small erosion on the exit can catastrophically make the basin empty and if a single flow stream has too much water in it then it can fork and flood causing new tributaries which may erode and dominate the flow. Its not at all clear at the outset where a river will be without some fluid flow down it to check. Its a more difficult problem than first appears.

If you could let rain fall evenly over the terrain or, like GTS, use a rain mask or a climate model then all those drops can flow until it hits the sea. If you could in real time monitor and adjust the terrain then that is what Monks calls the modelling in the rain approach which we think is the ultimate way to make the terrain. You need to be able to change the climate dynamically, push up mountains or flatten areas and let the water, or even snow too, do its thing.

GTS does this but not in real time. Not in a time frame that is even close to real. Its a slow process which you can get a snap shot now and again to see how your doing. You might do 500 iterations of flow and then get another frame spat out. On larger terrains like 2000 elements sqr its about an iteration per second. So a few mins between updates. We tend to run it on a small area quite quick and then let it run the same script again on the whole tile. If that looks ok then we let it do all the tiles like that which is hugely slow and would benefit from some serious acceleration or a PC cluster.

If I have a little time ill run GTS over your island and show some demo pics just for kicks. But I would not expect it to look great. It does some stuff great but not others. In this case I don't think it will look the way you would want it to.
Title: Cartographers stuff
Post by: monks on February 26, 2011, 03:04:41 pm
Apart from the modelling in the rain idea- where you have realtime feeback of fluid flow, there are other workarounds. We see realtime fluid is the ultimate approach because we're kind of terrain purists (at ME-DEM)- that is, you haven't really got a map (of any depth) without terrain. And that makes rivers for us important (well really fluid interaction the most important thing. Obviously there are others in the real world, but the two main ones are glacial action and water erosion). You could simplify that even more and say it's all about temperature gradients...which is really what drives GTS.

I like your idea regards rivers. Come the timne, I'd be interested in running that on our maps.

One workaround, is you could try run the erosion (a chanelling/incising erosion would be best
rather than a high sediment one) on the uneroded terrain to see where erosion was naturally
headed. Then use that as a guide to mark your river estuaries and your river paths on your
design map. Then when you run the erosion again, everything would connect up. I do this to
establish the main rivers, and then I add noise around the rivers and get the procedural
tributaries from the erosion as extras- the stuff I don't have time to do.

 Other than contours- which require a particular kind of madness  :lol: ..- I use a smudge tool
which I paint from the estuary along the river path into higher ground. You could automate that
I imagine. If you visualise the land as contours, the smudge tool creates ridges and valleys
depending on the land. You could automate that using input from a desigm map. Stepping through
pixels.

http://www.skindustry.net/medem/files/Temp/ContoursIdea3.png

 I also use a gradeint tool across a selection, but the smuge tool is the most reliable (but
it's the slowest!).

 Robes also wrote a nifty river flow tool for GTS (which he's not mentioned) which takes a user
input vector map of where rivers are, and bascially shunts water flow along them- it was
created to solve the problem of flat areas (by artificially multiplying the flow magnitude)-
they're the killers, it's difficult to avoid them with large "realistic" models. But with
interpolation across contours you can minimise them.

 Given a design map and some programming, rivers to order should not be *that* difficult if you
hook up the two in the programming. Get the main rivers with the design map, and then the
tributaries will follow procedurally.
 Ideally, for these large worlds, you want tools which are vector based- rivers included.
It strikes me Games have just not given any priority to rivers before; understandably. But really rivers are
where it's at if you talking about pure, realistic terrain. They define connectedness and
geography. It's what makes a world.

 
 ...Tolkien wanted other people to collaborate on his world. His famous statement about "Other
hands". The internet is the perfect platform for that vision. He did become "crest fallen"
about that in later years though, as nobody did add to his creation.
 There was a moment in Tolkien's life where he to make the decision between "cash or kudos" as
he put it. I think he chose cash and as a father of four with a sick wife who was put upon for
many years lecturing, I don't blame him  :P
 Tolkien would have tolerated the non-commercial fan work- I don't doubt that. The problem was,
not that he was avaricious, but that he had very high standards. I think Tolkien would have
hated what the lawyers do- what they do amounts to political control, and Tolkien loathed
politicians- it's well documented.

 I agree if you're making a game (I wasn't sure if you was), then it'd be risky to use anything
M-E shaped in it. But if you need terrain, you could use it with some modifications.

 It'd be great (awesome) to make movies in the engine. And as RedRobes said, it really wouldn't
rebound badly to you guys. And, the tools testing, yeh :)

monks
Title: Cartographers stuff
Post by: cameni on February 26, 2011, 03:12:55 pm
I know how the water flow and erosion is simulated normally, but that was not the point. The point was to allow placing the river artistically. For example, in ME you are already given the path of Anduin. You don't know all its tributaries, or streams in mountains. I wanted to take on the problem from the opposite end - we know how erosion looks like after some thousands of years, so we could use a fractal algorithm that simulates the resulting effect.

That would work like this. On the Anduin or on its know tributaries I'd generate branch points, spawning smaller rivers. These rivers would be generally attracted to the nearest mountain range, but the direction will be wobbling in a fractal manner. The strength of attraction will be higher with higher branching level of the river and also with the slope. This would allow for meanders in lowlands and normal fractal paths elsewhere.

Mountain ranges would be shaped by this process too. Each generated river would claim it's drainage basin, that serves two purposes - the branches won't cross each other, and the areas will be covered by branches when the distance from another basin is becoming large. River branches will be V-carving the hills with varying angle of V and strength of it. I can't probably explain it all quickly here.

Of course, these are just ideas so far, but in my mind they kind of work :D
This approach is less parallelizable, but would be also much faster. But I would not mind if it wasn't - what I value more is the ability to control and predefine parts of the world artistically, not by selecting from random creations, but by determining details at various levels while letting the remainder be handled by the algo. Essentially the same way we are handling fractal terrain now.
Title: Cartographers stuff
Post by: cameni on February 26, 2011, 03:31:47 pm
Quote from: monks
I agree if you're making a game (I wasn't sure if you was), then it'd be risky to use anything
M-E shaped in it. But if you need terrain, you could use it with some modifications.

For the game we are going to use earth data, we are not considering anything else now. My concern was for any project that could use an OT based world creator to replicate M-E, but as I'm generally not interested in creating a non-original work myself, it doesn't directly affect me .. just saying, basically :)

But I am interested in creating an OT based world creator one day, with tools that will allow creating precisely these kind of worlds (and more), so I'm always interested to hear ideas around that. Even when I know such a world creator can be made only after we manage to secure ourselves with something for a wider audience, propagating the engine. I hoping some steps towards that can be made during development of the game.
Title: Cartographers stuff
Post by: monks on February 26, 2011, 04:04:23 pm
Yes, I think maybe a balance between the two methods: create the main designated rivers from the design map with your idea Cameni, and then revert to the downhill method for the tributaries. If you can do it all with the single algo, then all the better of course. If we used a combination, then we could use GTS for the second part.


 Exactly- the idea I mentioned about viewing the world abstracted into L-system like branches is springing to mind again. Mountains and rivers are in fact interdigitating two sides of the same coin. There's a perhaps useful system of classification for river networks (Strahler Stream order). Even with subtle terrain, less visually obvious ridge forms, they are still obvious to water, there are still watersheds.

 My idea was to design software that saw the whole world abstracted (well each continent separately) as vectors (L system if you like, but that basic tree-like structure). The tree-like systems would be the rivers, and the ridges (drainage divides) interdigitiating. It's possible to extract ridge and basin info, so any new terrain created by the user (painted with the more  traditional tools) would be seen like this by the software.
 You do get basins which don't fit into the scheme- where water hits a sump but these are rare. The Sea of Rhun might be one of these. The system would need to handle those two as maybe isolated patches.

 Here you can see world seen on this scheme:
http://upload.wikimedia.org/wikipedia/commons/thumb/b/b1/Ocean_drainage.png/800px-Ocean_drainage.png

 There is research on the analysis of the fractals of the terrain at different altitudes too, but that's really more relevant to generating infinite scales of procedural terrain rather than rivers, but if the two are two sides of the same coin, it is relevant. Well, that's getting into more researchy stuff I guess.

monks
Title: Cartographers stuff
Post by: C. Shawn Smith on February 26, 2011, 07:18:09 pm
Quote
If I have a little time ill run GTS over your island and show some demo pics just for kicks. But I would not expect it to look great. It does some stuff great but not others. In this case I don't think it will look the way you would want it to.

Cool.  The map itself needs a lot more refinement, but it works as a "base" line for now.  Once I get Photoshop reinstalled, I'm going to dedicate more work on it to get the height dynamics a bit more geologically accurate.

Here's the actual color map I made prior to the height map: http://www.cshawnsmith.com/novels/high-resolution_thebadlands.jpg
Title: Cartographers stuff
Post by: Redrobes on February 27, 2011, 10:43:57 am
The artistic approach is fine for many maps. Just so long as you don't have rivers flowing down an uphill bit of terrain it should always look fine. You have some videos showing paths and roads and the terrain seems to fall real nice down to the road such that the road is flat. Not sure how you did that but it worked real well. If you can do something similar along the length of a bit of river then you should be able to control the terrain surface so that the river is always flowing downhill.

To be honest, if you could somehow alter the terrain so that it always made arbitrary artistic rivers flow correctly then lets face it, thats a much better way of doing rivers than were doing them. We're looking at static terrain and calculating the river and hoping for it to flow in the right direction then manually changing the terrain - rinse and repeat. If you could set the river and automatically adjust the terrain then I think people would rather set the rivers down and get whatever mountains they are given than set the mountains and get whatever rivers they are given. So if you can do it then kudos to you.

A balance between the two would be ideal tho and I agree with cameni in that for most games designers, getting control of what they want and being able to deliver that terrain is more important than strict geographical accuracy.

@cshawnsmith, I might make a little series of images to show GTS first then run it on your island cos I think its just going to flood in the middle. Since your island climate is so dry then you might not have very much water, not many rivers, and not much water erosion to look at. The scale of your map would probably mean that any rivers would be very thin anyway.

I'll set up a ramp with bumps all over it and then get it to calculate the water flow on top and make a render from that. It would show it off a little clearer I think.
Title: Cartographers stuff
Post by: cameni on February 27, 2011, 12:17:04 pm
Exactly, the algorithm I have in mind will have to modify the terrain, not only to make the river itself flow downhill but also to shape its drainage basin and the mountains. As monks said, these are two sides of the same coin. Without water the land would look absolutely different, so I presume we could generate the land with simple mounds for mountain ranges, and let the rivers backtrack and claim their basins, carving the land in the process.

The road system will be used also for the rendering of the rivers, but this only covers the making of river beds and banks so that it all fits into terrain perfectly. Large scale terrain modifications would not be effective to do in real time, and it would not be reasonable for other reasons as well, but in fact almost the same approach can be used in an offline terrain generating tool.
Title: Cartographers stuff
Post by: monks on February 28, 2011, 11:14:07 pm
I don't know why someone hasn't tackled this problem head on before. Maybe they have, I don't know. Back 4 years ago when we were discussing it at ME-DEM and on the Terrain Summit it was barely a problem because no one was using worlds large enough to see large sections of a river, let alone both ends at the same time as in these planetary renderers.
 So, no-one cared whether they flowed downhill or not.

 One approach is to build the terrain around the rivers, which is what you've suggested Cameni- kind of. One idea I had was to procedurally build the terrain immediately under river beds. That was the first priority. Impose gradients on them (between two or more control points) such that they flowed properly.Then procedurally fill in the terrain around those. Strange idea but I guess these are the kinds of things you have to consider when you want correct flow according to design maps. It's conventional terrain modelling in reverse basically.

A skeleton of river heights as a terrain could then by combined with the original terrain perhaps.

 Probably best off start off with terrain. User draw rivers. Software then reconfigures the terrain so that the rivers work.
 
 If I had programming ability I honestly reckon I could come up with something that would work- it might run like a piece of crap  :D ...but it would basically solve the problem.
 
  monks
Title: Cartographers stuff
Post by: Edding3000 on March 01, 2011, 11:07:04 am
Quote from: monks
If I had programming ability I honestly reckon I could come up with something that would work- it might run like a piece of crap  :D ...but it would basically solve the problem.
 
  monks
The problem is, that everybody thinks they can do it. Also on the infinity forum.
But them are always people without programming capabilities. I know exactly why this is.
There is not a simple solution to this problem though. Although pregenerated terrain (earth) makes it easier.
But you still do not KNOW of the fractals that get generated and so you cannot know if a flow will be uphill or downhill.
Also with deeper LOD levels rivers will jump all over the screen because they will need to be rerouted.
The coarse lod will have huge rivers because of the limited mesh set. So you must generate/load deeper LOD's to investigate, and that takes a LOT of processor time.
In my opinion the only way to come up with a river system is the following:
pre-generate it.

And this is just not do-able.
Title: Cartographers stuff
Post by: cameni on March 01, 2011, 12:15:41 pm
Quote from: Edding3000
In my opinion the only way to come up with a river system is the following:
pre-generate it.

And this is just not do-able.
Note, we are talking here the whole time about generating the rivers in advance. I've read somewhere that rivers are the holy grail of fractal worlds. But it will stay so unless people stop doing stupid things like trying to generate the whole world using a single fractal algorithm. The ridged multi-perlin or whatever, expecting that he or somebody someday discovers a magical single-function fractal that generates real worlds.

So, we are here talking solely about worlds that are pregenerated, prepared in advance, and even designed by humans. I for one am totally uninterested in single-seed single-function fractal worlds - that's absolute boredom, and no infinite numbers of them could save it.

We are talking about generating world by dealing primarily with the rivers and mountains, these two being really two sides of the same coin. From this realization and from this discussion I guess that monks and Redrobes are qualified enough, I wouldn't compare them to the I-can-do-better junkies on some forums :D

Though I was surprised to read monks can't program too :)
Title: Cartographers stuff
Post by: monks on March 01, 2011, 12:41:59 pm
Yes, I wasn't saying (and I was going to qualify what I said before someone jumped on it), that I could solve the problem in the domain specific to MY/OUR problem: design map/ rivers- not ANY randomly multi-fractal landscape. My solution would give a terrain which was a skeleton of river bed heights which would still require some effort to fill in with other details around that skeleton. It would still be a two step solution if you like- not the ultimate answer to everything- which is 42 I believe anyway :)

 I was going to mention the work done by on MojoWorld in reference to the search for the "Holy Grail"...I can't recall his name, but he was a physicist, and wrote the rivers plugin. Is that operating on a planet? I'm not sure if Mojo is a full planet renderer.

 I think there are some pretty good solutions already for creating riverine systems for random procedural worlds. If you look at Wilbur and GeoControl for instance. Wilbur/Fractal Terrains Pro uses basin filling and channel incision.

 Yes, totaly agree about the boredom with those worlds. I was only going through some YouTube videos for planetary renderers last night, and I saw some of those.

 I did do some programming as a hobby for a few years (procedural and C, C++)- did a couple of programming modules too with uni...so I have a reasonable idea of "how things work".  ;)

monks
Title: Cartographers stuff
Post by: Redrobes on March 01, 2011, 01:06:58 pm
I think maybe one day... one day at the moment well into the future... we might make the terrain at the same time as view it. As far as I can tell from what people have been saying, there is considerable effort in making the LODs for the 3D. For my ViewingDale in 2D I have a custom format  for the images which are 2^n resampled for one thing and I do creates the LODs on demand which go into temporary mip files. But that does incur a stutter so panning around for a while is jittery the first time before it becomes smooth. It sounds like you guys are doing some fourier transforms to get your LODs and thats not a fast process so for now at least that is done offline and prepared earlier. If that is the case then I would guess there is no option but to bake the heights into the LODs at that time which for the way we have spoken about altering the terrain to fit an artistic placed river is something that is not so easy. I had not considered the fractal effects which may impact those too.

The video Monks linked to from ProTerra shows him editing the land in real time so this is something that has been achieved without rivers at least. Now my river tool creates the rivers on top of any terrain, fractal or not. In theory, with enough compute (and I don't think we have enough on one PC no matter what GPU you have under the hood) we could do that process.

Ahh new post... was that Mojo guy called Hugh Malan ? I believe he wrote a real time world creator in shader language and was a Pandromeda at the time - Realworldz. If its not him then I don't know.

I came on here to post some new links to some GTS vids I did a day or two ago. This is a 3D ramp I made, sub divided the surface and then added some bumps and pits to it then threw GTS at it to see what it made of it.

http://me-dem.me.uk/videos/gts/Trough4_A.avi
http://me-dem.me.uk/videos/gts/Trough4_B.avi
http://me-dem.me.uk/videos/gts/Trough4_C.avi

GTS is just a numeric iteration so as long as you can get a function to return the height at X,Y then it can run on a terrain whether fractal or baked or whatever. I can see some good points that Edding is bring up tho with the deeper LODs if you want detailed river accuracy. It may be true that as you travel toward the river and change LOD level then your river switches course... yeah maybe, not sure.
Title: Cartographers stuff
Post by: cameni on March 01, 2011, 01:41:48 pm
Well, in Outerra we'll use vector data for rivers, and that's another reason why I prefer the fractal river model over the simulation, although you can get vector data from that too with some additional effort.

With vector data and the way they are applied there's no problem with LOD. We will not simulate the water flow in coarse LODs and then wonder what to do with rivers changing width and courses.
As for how real-time the generation of world can be, I'll reserve my judgment for a later time. With the fractal river backtracking it could be pretty fast, but then again - we are talking about whole planets and the limiting factor is amount of IO. The problem is that the simulation should run globally. Though if the world was successively divided into separate river basin and sub-basin areas as the rivers are being generated from estuaries, one could achieve much better locality and optimized transfers there. We need some experimenting here to see if it's any good.

For terrain without rivers the editing would not be problematic - LODs are obtained through wavelet decomposition and the way it's implemented is really simple and hw friendly.

Thanks for the posted videos, really interesting. For a while I was wondering if you did make it in a sand box in your backyard :D
Title: Cartographers stuff
Post by: Redrobes on March 01, 2011, 02:12:24 pm
Heh - nah, its not really that accurate. It tries to model sediment pick up and deposition and calculates a little water momentum but its still lacking stuff to make it accurate. I didnt calculate pressure for one thing. I assumed the water was incompressible. Maybe thats a reasonable assumption I don't know enough. At the time we had very little reference to match it against. Ill post this link of some guys videos who did make a good sand box. I think I could have done mine better had I seen these at the time.

http://serc.carleton.edu/NAGTWorkshops/geomoph/emriver/index.html

So my sim is a fixed texel sized array of parameters which is iterated so that it transfers water across from one to another and tries to keep track of momentum, temperature, and any induced state changes etc. Because there is a lot of parameter info then it might not run well without IO. However as cards get more GPU memory then maybe the array size could go up so that its not doing any transfer over PCI, just between video mem and PE/cell/ALU local cache which is fast and therefore I would see the big speed increase.
Title: Cartographers stuff
Post by: monks on March 13, 2011, 03:42:38 pm
I actually though that was rendered at first  :D  Then I got images of blokes in tank tops and wellies...haha. Very interesting the way the number of curves affect the gradient, and velocity- not something you would immediately consider.

 Yes, the subdivision along river basins sounds like a usefiul abstraction.

 I've been doing some tests regarding procedural river generation so I thought I'd share the results with you.
 I ran the terrain through "Wilbur" using the precipiton and incise flows algorithms. I'm not using basin fill here.
 The pics show the terrain in World Machine with some erosion added in there. Apart from some of the results we've had with GTS, these are the best rivers we've had to date.

Note, the terrain here was processed without any fractal detail added.This is as about as basic in appearance as the terrain gets. I added 2 or 3% noise in Wilbur. The terrain was a 4096. I've yet to run tests on an 8192, memory permitting (Wilbur is not designed to handle large terrains). The terrain is flat in many areas, with minimal noise, but I've manged to get rivers flowing the entire length of the dem with this process.


(http://www.skindustry.net/medem/files/Phase2/Terrain_1.0/NASARun/2011/Screens/RiverTests_800.png)



http://www.skindustry.net/medem/files/Phase2/Terrain_1.0/NASARun/2011/Screens/RiverTests5.png

with coastline
http://www.skindustry.net/medem/files/Phase2/Terrain_1.0/NASARun/2011/Screens/RiverTests5a.png


http://www.skindustry.net/medem/files/Phase2/Terrain_1.0/NASARun/2011/Screens/RiverTests6.png

with coastline
http://www.skindustry.net/medem/files/Phase2/Terrain_1.0/NASARun/2011/Screens/RiverTests6a.png

http://www.skindustry.net/medem/files/Phase2/Terrain_1.0/NASARun/2011/Screens/RiverTests7.png

with coastline
http://www.skindustry.net/medem/files/Phase2/Terrain_1.0/NASARun/2011/Screens/RiverTests7a.png


This process creates very long rivers and it preserves the coastline very well.
 You can find info on the precipiton algo online of course- if you're interested. The incise flows algo is also really useful. You would probably find that applying a basin fill after noise, would help too.
 First I run the precipiton a good few times to start cutting into the terrain, then I run incise flows- that picks up on the established flow. I repeat those two steps a few times.
 My terrain has the advantage that I've created a pretty predictable surface via contours in Global Mapper before running precipiton. You  could create that using your algorithm that you intend to use to get the basic design map in, then run your equivalent of the precipiton and inicise flows to fill in all of the blanks

 I would absolutely love to get this into the Outerra engine!
 
monks
Title: Cartographers stuff
Post by: cameni on March 14, 2011, 05:11:21 am
Those rivers look good indeed.
In preparation of a new, fixed Earth dataset, I'm currently reworking the tool that maps and compresses external raster data into our internal format suitable for streaming.
Also in the light of this thread I decided to integrate the functionality directly into the engine code, as it will be also necessary for real-time terrain editing and computation.

I'm not sure though how to run the terrain computation algorithms on that thing. You guys talk about arrays of 4096² or 8192² size, but the planet here has got 6 faces of 524288² data points, that's ≈3.3TB of raw data (with 2 bytes per sample, which may not be enough). Even though the continents will be smaller, it's quite unimaginable to run multiple passes of incremental algorithms here and expect it to terminate in reasonable time. I guess an algorithm that first partitions the world into separate drainage regions - much like what Monks posted (http://upload.wikimedia.org/wikipedia/commons/thumb/b/b1/Ocean_drainage.png/800px-Ocean_drainage.png) but not per ocean but per rivers - will be necessary. And that's why I am also more attracted to algorithms that try to generate the resulting appearance, not to emulate the process that led there. Though in the end it will probably be a combination, like you both said.
Title: Cartographers stuff
Post by: monks on March 14, 2011, 11:28:46 am
I think I'm understanding what your intentions are more are now. Emulation rather than simulation. I was watching some of your videos and I can see what is obviously Earth data but integrated into different shaped continents- they seemed to be anyway.
 You are going to use Earth data and superimpose your design map on that and change the data where necessary. Then you'll blend in the area using fractal code that will emulate the appearance of the Earth data. Is that what you're going to do Cameni?

 I was considering using Earth data myself. The most difficult feature to emulate is the structure- especially with the present tools. You really have to manually do it. Even the most obvious structures like mountain ranges are very time consuming to create. Most artists either use real world data and integrate that or more or less place fractal masses and erode it. The less
obvious structures are easier to get away with. But in a planetary renderer you can just see the difference between a procedural engine mush, and the real world.
 In terms of laying down the design map stage, I think the contour tool that I suggested where the tool operates on vectors would save oodles of time AND it would guarantee predictable flow, even over the large surfaces (with the help of interpolation algorithms which you would only have to run at bake time). I'm sure that if someone coded it on a gpu, it would run very well in rt, even with larger point sets. Wilbur has a contour shader as does Leveller, that updates with the modelling, but it's the underlying mechanics of painting with pixels and the lighting in 3D, that slows it down. You wouldn't even need a 3D interface, I think 2D (with the occasional bake to 3D) would be easier to work with.
 I've asked Mike at Global Mapper if he'll implement it. He's put it on his to do list but he reckons it won't be that easy to do. I asked for brush pressure sensitivity- and perhaps that's out of his area of expertise.

 World Machine's spline tools are very useful as well but when you're considering modelling a real looking mountain range you need all the help you can get. That's where the watersheds representation could be applied. If you could procedurally generate an L-system (for want of a better description) of ridges/basins, it could potentially make the creation of very large mountain ranges with correct drainage, a possibility. A future version of World Machine may include this tool. You would define your primary backbone rigde using a lofting curve for heights, (possibly define the boundary or footprint of the range- much like a reverse watershed) and then hit generate and grow the network of smaller, lower ridges off that.
 This could be applied to all of the terrain data, not just mountain ranges, Define the watershed area, define the principle river (which would serve as the backbone ridge), hit generate, and you grow from that a network of basins. The program would always have to fill in the other part though to save time- ie the basin networks for the mountains, the ridge networks for the watersheds.  Maybe there are some subdivision routines out there that would fly on the gpu.
 Different terrain morphologies could be created much like different trees are created with L-Systems. Then you get into the idea that you could sample r-w terrain and if you could repesent it internally on this scheme, develop libraries of morphologies. If the sampling was good You wouldn't even need erosion! GeoControl has this feature that lets you generate infinite levels of detail, by zooming in and re-generating the terrain. It breaks down a little bit a after a while (you lose the initial looks of the terrain), but it is really excellent. The program represents terrain in a heirarchical form- well it uses the noise octaves, and stores terrain morphologies as library presets under this scheme. It doesn;t use the watershed
ordering though, but I reckon  he's got the right setup to take it in that direction.


  I think if we're going to have very large terrain creation, usefully predictable hydrology, good intuitive artist tools, we're going to need both contours and the networks idea. To do that we'll probably need the software to be able to sample input terrain and internally represent it as vector networks within an overarching framwework of watershed subdivision.

Apologies for banging on, but this stuff really interests me   :D

monks
Title: Cartographers stuff
Post by: cameni on March 14, 2011, 03:47:07 pm
Quote from: monks
I think I'm understanding what your intentions are more are now. Emulation rather than simulation. I was watching some of your videos and I can see what is obviously Earth data but integrated into different shaped continents- they seemed to be anyway.
 You are going to use Earth data and superimpose your design map on that and change the data where necessary. Then you'll blend in the area using fractal code that will emulate the appearance of the Earth data. Is that what you're going to do Cameni?

No, the continents are shaped alright, those are real Earth data.

And no, I didn't mean to reshape Earth - but it might not be such a bad idea :)
You are right about creation of all the structures being time consuming, so some reusing and mixing of existing "templates" might be something that can be used in manual mode.

But I meant something different - emulating the resulting appearance of erosion on some artistically sketched terrain by means of fractal generation. Much like how you describe the reverse watershed area generation and refinement. Watersheds for deeper levels (of river branching) will be also creating additional mountain ridges. Original idea was to sketch the contours of continents, place the estuaries for major rivers and have the generator generate the rivers upstream while also modifying terrain elevation along the way. With many more possible additions, like river branching depending on land type it's passing through, ability to place mountain range attractors so that the rivers are drawn towards it, carving the mountain valleys and peaks with the last river levels and so on.
Title: Cartographers stuff
Post by: monks on March 18, 2011, 06:10:47 pm
Sounds like a good plan. Contours are so useful for obvious reasons. You are only defining a tiny subset of the surface. Interpolation and gridding them is the big time saver. You are guaranteed gradients that you couldn't paint in greyscale or indexed colour very easily if at all- certainly not in 16 bit greys. It's much better than the paint and blur approach. You always get stepping with painting.
 You'll be able to set up good conditions for your main flows as well.

 If you're coding an interface for the contour drawing stage You might want to code automatic z tagging of drawn contours. Tagging vectors can be very time consuming given enough of them. You could set up a dialog that automatically tagged all contours with the same height until told otherwise or tagged them incrememntally via a hot key. Tagging mutiple contours of the same height with selction and ctrl + mouse but also speed things up.

 The traversal upstream sounds pretty cool if you can pull it off. You might want to program in more meandering in the floodplains. You aso might want to observe exponential fall offs as well with curves wthin the terrain- rather than a linear fall-off. Like you say, there are a number of parameters you might want to include.

 Apprarently there was a SIGGRAPH paper on creating terrain by the calcualting rivers first. It focussed on a single watershed, which is at least half the problem solved if you divide things up on the watershed scheme. I've never read it though and unfortunately I don't know who the author author was.
 After a quick look I found this:
 http://www.springerlink.com/content/kxw0uwt010p80331/

 http://www.slidefinder.net/R/RiverLand_Efficient_Procedural_Modeling_System/29957735

 ppt available here with some references:
 http://www.cs.sjsu.edu/~teoh/research/presentations/

 I see Howard Zhou gets a citation. I tested his Terrain Synthesis plugin for World Machine.  :)

 One idea is to allow the user to input morphology maps- atlas- that could apply to both simulated erosion algorithms and also your own approach. Any synthesis done within the map areas would influence the paramaters and therefore the terrain appearance. L3DT uses a Design Map which incorporates this idea for texture creation. As far as I know it doesn't use it for the actual terrain creation.
 Concentrating on the rivers will help to provide the large scale structures. This approach might help with the smaller scale variations too.

 Btw, Brano's news was fantastic!  :cool:  Cheers guys.

monks
Title: Cartographers stuff
Post by: cameni on March 18, 2011, 06:24:12 pm
Cool, thanks for the links.
Here's also the pdf: http://www.cs.sjsu.edu/~teoh/research/papers/isvc09.pdf (http://www.cs.sjsu.edu/~teoh/research/papers/isvc09.pdf)

And hi, I am Brano :D
Title: Cartographers stuff
Post by: monks on March 18, 2011, 06:33:29 pm
haha  ...ah you've spoilt my illusion of this huge Dev Team with its different departments.   :D

monks
Title: Cartographers stuff
Post by: cameni on March 25, 2011, 06:21:38 am
Btw today I updated our plugin for XnView (http://outerra.com/xnview/) image viewer allowing to view raw *.hgt or *.srtm terrain files, with autodetection of image dimensions using neighboring row correlation computation, and more.

Might be handy for some following this thread :)
Title: Cartographers stuff
Post by: Redrobes on March 25, 2011, 07:57:26 am
Thats cool. So if we supply some raw height tiles that render correctly using this plugin then your good to go. Im a bit tied up for a short while but just starting to ease off other projects so I will look at a medem height tile conversion pretty shortly. I have written a few little apps which convert our HF2 or RAW data into PNG and saves them out so its kinda similar.

This is the usual link I get my SRTM data from: http://dds.cr.usgs.gov/srtm/version2_1/  Do you know of a link that has it at any better res than these downloads ?
Title: Cartographers stuff
Post by: cameni on March 25, 2011, 08:07:34 am
There are also Viewfinder Panorama DEMs (http://www.viewfinderpanoramas.org/dem3.html), with links to other sources. A global coverage 3" compilation at rmw.recordist.com (http://rmw.recordist.com/).
Title: Cartographers stuff
Post by: SpaceFlight on April 08, 2011, 05:46:09 am
There is a satellite mission (TerraSAR-X/TanDEM-X) running right now, to make a very precise 3D map of earth´s surface:

http://www.bbc.co.uk/news/10422511

Quote
The seamless DEM of the Earth's surface will be built up over three years of joint operations. Ultimately, it should have a vertical resolution of 1-2m and a spatial resolution of 12m - far superior to any previous global data set.

I suppose the complete DEM will be available in about 3-4 years, but one can check out samples here (I dont know if the format they use is compatible with OT though):

http://www.infoterra.de/free-sample-data

Also, http://www.infoterra.de/ sells the data, so right now, except for the samples, it is not for free.

More info on the mission here:

http://en.wikipedia.org/wiki/TerraSAR-X

http://en.wikipedia.org/wiki/TanDEM-X

edit:
Hmm, 17€ per square kilometer for a basic DSM, I guess that is not an option then.  :rolleyes:
Title: Cartographers stuff
Post by: monks on April 08, 2011, 09:18:30 am
That is going to be some DEM, but they're never going to give that away. Prices for other dems should drop though.  :) I thought there was already 10m dems available? Or is that just US coverage?

monks
Title: Cartographers stuff
Post by: Redrobes on May 30, 2011, 07:27:35 pm
Cameni, just an update and a question. Update first.

We made the 40K map and textured it out but found in the result the light maps were bad so were redoing them. Its off processing again.

The question is this, since you don't take in any textures and generate all the lighting then you are only interested in the height maps. On middle earth tho, we absolutely must have Lorien, Fangorn, and Mirkwood as a thick wooded area in a specific delineated region and places like Mordor as volcanic hot, inhospitable land.

Are you going to have any kind of input masks or control over the climate, temperature, vegetation and so on. We are generating the masks from GeoTerSys but we also have input bias maps for vegetation which force or prevent veg growth. We have a base temperature map we put in and let GTS generate the final adjusted one. But that allows us to dictate where there will be snow or where its artificially hot.

I was looking at your pics of Sahara and its textured with some trees on it. You have great auto generation of trees but are you expecting to input some bias into that system at some point ? A programmable climate model or masked set of climate properties ?
Title: Cartographers stuff
Post by: cameni on May 31, 2011, 01:28:19 am
It will use climate/land class input together with the heights. The exact format is not yet determined though.
For Earth there exists a 500m dataset with some 15 discrete values determining land type, like the desert, savanna, woods of different types, up to polar regions.

Tree cover will use the input from land class data to bias the probability up or down for given type, so forests will be forests but the probability computation and fractal behavior will still apply, creating occasional clearings and not placing trees on steep rock walls, and also creating natural transitions to the surrounding land types.

The problem is that those values just tell about the dominant land type. A transition from desert to savanna takes place on some tens of kilometers, with changing ratio between sand/grass surface, but the dataset doesn't tell any of it - it's like if there's more sand than grass, it's a desert.
One possibility is to postprocess these discrete datasets into ones that encode probabilities, but looking at the Earth's dataset shows it won't be that easy - for the transitions between large continuous areas it may work, but at the places where land type changes quickly between grass/woods/swamps it would inappropriately smooth out the details.

Although, with artificial maps this problem usually doesn't occur - they don't have the detail in them in the first place :)

In addition to land class maps we'll be also processing color maps that bring additional colorization for the same land class, like different shades of sand color.
Title: Cartographers stuff
Post by: Redrobes on May 31, 2011, 06:22:26 am
OK thats cool. We can generate those maps I think. We have masks for all sorts of different terrain types and we use those to generate the veg bias map but we could generate something else just as easily.

Have you thought about snow as well. Originally we used to map snow to any height > X but found that it does not look like snow at all. Its especially noticeable when your at altitude X looking across at the mountains and there's a ruler straight snow line. GTS now models temperature flow like water so snow can pour down a mountain side and go below the Height > X line and maintain it being snow since it carries coldness with it. In this way we get glaciers and a wobbly snow line. We extended this idea so that the direction of the sun melts snow on southerly faces too which looks quite nice as well.

We can provide a snow mask or the post processed temperature map showing where snow would be or bake it in as part of the climate / terrain type map. In your engine, calculating that snow fall line is probably going to be hard as is much like a fluid flow problem that needs to be precomputed.

Since we have the colour map we can do a fade transition from deserts to vegetated areas no problem. What we cant do, which your engine does, it to put in 3D trees so your engine produces such good results looking across terrain. Our terrain only looks good in our 3D tools when viewed from a distance where trees and micro detail is lost to the overall terrain shape. Why we like the look of Outerra so much is that you can get close to the ground and let the fractal engine take over.

Since your engine knows about the probability of putting in trees, why not adjust the tree height (and tree style when you get that bit done) as well as the spacing for them. We have seen pics with these lone trees in deserts because presumably there is a non zero chance of a tree. If that tree height were similarly scaled by the probability then you might get the odd stumpy tree instead of one full sized one all on its own. In this way as the probability of going from one terrain type into another changes the trees would turn to scrub and it would appear more natural.
Title: Cartographers stuff
Post by: cameni on May 31, 2011, 07:11:41 am
Snow is subjected to a heuristics that basically says this: snow can hold on places that are cold enough and flat enough. So it computes the temperature for a given place, and biases it with the degree of steepness of terrain (the steeper terrain, the higher the computed virtual temperature is). In effect this allows the snow to hold on flatter places in lower elevations, and on gradually steeper parts as the elevation rises, and it almost completely breaks the line. It's also kind of logical, when you think about it.
It looks like this (more at http://outerra.blogspot.com/2010/07/collection-of-screenshots-from.html):

(http://www.outerra.com/shots/k300.jpg) (http://www.outerra.com/shots/k300.jpg)

Additionally there's a fractal bias added to break it even more.
Snow this way naturally holds onto the valleys, but we want to help it even more by accounting for terrain curvature as well - "V" shapes will lower the temperature so that the snow can pour down.

Technically, the equation should additionally include precipitation and temperature should be computed also from latitude and be locally adjustable, to account for other effects like warming/cooling from ocean currents and shielding by mountain ranges and such.
But just using the altitude and latitude together with the equation should be good enough for many cases.

As for the fading between land types - what we will ultimately need is a format that carries probabilities of occurrence of land types. A format that says: this is a mixed type, with 70% of dry ground and 30% grass (eventually in a particular period of year). We need this to feed the fractal generator that would produce a fractal mix of land that obeys the probabilities, and looks natural - clumps of grass in a sandy land, in this case.

Quote
Since your engine knows about the probability of putting in trees, why not adjust the tree height (and tree style when you get that bit done) as well as the spacing for them. We have seen pics with these lone trees in deserts because presumably there is a non zero chance of a tree. If that tree height were similarly scaled by the probability then you might get the odd stumpy tree instead of one full sized one all on its own. In this way as the probability of going from one terrain type into another changes the trees would turn to scrub and it would appear more natural.
The height of trees is already determined by the probability, there are lower trees at forest borders, for example. You'd immediately notice it if it wasn't the case. Of course, in different biomes the trees will be generated appropriately, so there will be palms in oases and such. Or the stumps here and there, but that will be probably done differently - a low probability generator placing these, I think.
Title: Cartographers stuff
Post by: Redrobes on May 31, 2011, 07:35:41 am
I think the situation is a little more complex than that and cant be predicted accurately with a spot check bit of math probability. The feed of the snow and collection of it is important. See this pic for a good example.

http://i21.servimg.com/u/f21/14/11/62/01/01_ste10.jpg

We try to make the same calculations for it like this:

http://me-dem.me.uk/galleries/gts/3DTexDemo045.png

and we could give you these masks to help guide your texture engine. Well, were all a long way off of being ready so there is plenty of time to get all this sort of stuff in order.
Title: Cartographers stuff
Post by: cameni on May 31, 2011, 07:55:14 am
Well yes, the snow tongues aren't so prominent, and it's true it can't be simulated accurately using just a locally operating math. External bias map would help covering these cases, while still keeping the look consistent with the underlying high frequency terrain details.

The thing is, we cannot just apply a mask that says snow, it needs to consider the generated detail, that wasn't present when simulating. So what we are trying to do is to get a reasonably well looking and believable terrain first, without using external simulated sources. Then we would use the output from a simulation to generate bias maps, enhancing the generated output and making it more physically correct.
Title: Cartographers stuff
Post by: Redrobes on June 16, 2011, 01:09:46 pm
FYI: AMD and OpenCL tools.

http://ir.amd.com/phoenix.zhtml?c=74093&p=irol-newsArticle_pf&id=1572937
Title: Re: Cartographers stuff
Post by: Redrobes on September 21, 2011, 09:49:22 am
Hi cameni, Can I ask you about this bit:
... a custom compiler into GLSL that gets us rid of some of the aspects of its poor design.
I was messing about with some GLSL and trying out some lite stuff and I remembered that you used a custom compiler for it. So what bits about the GLSL source language did you think were a problem that you wanted to fix with a custom generator ? I was toying with GLSL just to get my feet wet with it with a longer term view to implementing the main processing loop of my GTS in it. I believe all I need to do is lots of float texture manipulation but knowing why you found some issues might keep me out of a swamp somewhere down the line. Its very hard to find any float texture math examples in GLSL on the web. I don't think I need double float for it so in theory I could do it on my current card.
Title: Re: Cartographers stuff
Post by: cameni on September 21, 2011, 10:50:10 am
Well, there's a lot of tiny bits that make it an ugly language for me. For example, here's how we write shaders:
Code: [Select]
VERTEX_SHADER
void v_unit(
    in int vertex_id    : VERTEXID,
    out float4 opos     : POSITION,
)
{ ... }

FRAGMENT_SHADER
uint4 f_compress(
    float4 fragcoord : FRAGCOORD,
    uniform sampler2D image,
    uniform int mip = 0
    )
{ ... }

The same code in GLSL has to be split into two files, where each function is 'main', with global variables instead of the parameters, making the code more messy and less readable. Original GLSL didn't support #include (added now). In GLSL you have half-ambiguously blocks and structures, but struct cannot be used here and there. Block scope is weird. Const has an additional constraint that it can be constructed only from constant expressions, but it was so awkward that now the board allows to use it in functions in its C-like meaning. Boolean vector operators are not supported, you must use lessThan etc, but they forgot vector and/or/xor. And so on.
Then there's a lot of minor cosmetic stuff that still bothers me - vector types named as float2, int3, uint4, bool3 in DirectX or Nvidia's cg are here vec2, ivec3, uvec4, bvec3. And lots of other stuff ...

However, we wouldn't probably make the compiler just because of this. Initially we were using Nvidia's cg that is capable of compiling into various profiles, one of them glsl. Nvidia of course didn't make the profiles for ATI, and the glsl output was limited and buggy and the support miserable. So we were forced to either cross over to the glsl or to make a preprocessor. We already had some apparatus that took cg generated code and provided wrappers so that the shaders could be integrated effortlessly with the engine. Given that I have written several parsers and code generators in my life, I decided to make a cg-like compiler to glsl, that would also generate all the gluing code.

Nowadays we can just write a new shader, and immediately start using it with a single-line directive in the engine. The generator takes care of everything in between. We also enhanced the language constructs from the original cg syntax.