Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Redrobes]

(note: split from here)

No public demo is a shame but I understand. I also process images into ViewingDales own format for the same reason. It makes them easier
to render in OpenGL and it contains meta data thats static to the image and not worth recomputing on the fly.

We have a base DEM which is built from pure vector contours and is exported (via Global Mapper I believe) into a rastered pixel based height map. Its in float values or 16bit height integers. Again we use a format that we have had some input into generating. Its the HF2 format which was knocked out on another forum of terrain devs. There is an LGPL implementation of it from Aaron of L3DT if your interested in that. All the other DEM formats had one issue or another that made it unsuitable. This one is designed to be unlimited in scale so worth a look I think.

All the data is geo-referenced. Monks - who is going to sign up here too, will know more about that. Tolkien said in his letters that Middle Earth was based on Earth and that there are some matching places like Hobbiton being in Oxford where Tolkien studied etc. Anyway, we have mapped the data set to earth so I think that part is already covered.

I know we can export a physical resolution of 20kpix square which is about 200m. The intention is to take this into my GeoTerSys which is a terrain synthesis tool based on a simulation and run that. The main idea of GTS is so that it takes a rough terrain and makes it more realistic and up's the res of it by running a geological / climate simulation on it. See images on the medem site. Starting images are the deliberately rough ones. Anyway, the idea was that we would like 100 kpix per side which is about 40m or so. I am sure that is a lot of data tho - many Gigs at least. We like the sound of what you have where a) it zooms by managing LODs and b) fills in lacking res with sensible fractal stuff. We have about 100 layers of information we can use to guide shaders. We were going to use them to drive masks for Terragens procedural shaders.

As for the Tolkien Estate. Yeah maybe. Our project is all free - its fan based non commercial thing here. So the data is a give away thing and you can take it and try it and make some movies from it. Its built to match the shape of their map and the names have to match of course, but aside from that, it does not use anything from any commercial works. We would like one day tho to have a web site with a download of engine and data and allow people to fly through it. Maybe get a community thing going with the buildings from sketchup or something like that. We have found the generation of the actual DEM too painful and technical for general community spirit based work so it leaves a few poor souls like Monks and I to do most of it.

We can fly through it in top down Ortho only using ViewingDale as a map browser but the project is inherently 3D so that kinda misses the point.

We saw some vids of YouTube where someone was editing the height map in real time at all scales right up to continental. I don't know if that is part of this project or an off shoot of it. Certainly that would be very interesting for us. Guiding rivers to flow to the correct place dynamically would speed up our process immensely.

EDIT -- Link to HF2 specs
http://www.bundysoft.com/docs/doku.php?id=l3dt:formats:specs:hf2

Edit2 -- If your just into the maps from medem try this one:
http://me-dem.me.uk/galleries/vdale/MeDEM_Map1.jpg
As you might tell the full sized one is much bigger.

And thanks for the replies - ill say that here to keep the thread clean.

[C. Shawn Smith]

I've been a regular of the Cartographer's Guild for a little while.  There's some really cool stuff there.  I even wrote a PDF tutorial on the creation of realistically rendered maps, which is available on my website under the Photoshop tutorials link at the bottom.

I'm a writer, so the Outerra engine is interesting to me for the possibilities it may have in the future for rendering my fantasy world in realistic 3d.  I'd love to see a Middle Earth rendition within the engine as well

I'll have to check out your sites regarding Middle Earth.  I haven't seen a good rendering yet, and it's one of the worlds that is dear to me as a fantasy writer/reader

Welcome to the forums!

[cameni]

Thanks for the explanation. It seems you are all ready and knowing what it takes
I'd be glad to see this project running in Outerra.
But ..

The Tolkien Estate - well the problem is that they are also after the non-commercial activities, after anything that refers to the Tolkien's name or work. They may see this project to be undermining their other commercial efforts, efforts that bring them money. They will surely try to stop it, from what I've read about their ways. That's why I'm reluctant to deal with any of Tolkien's work. A shame, but the monopolistic protection in the form of sw patents and IP laws seems to be a common problem nowadays in our part of the world, one which is almost impossible to solve without also letting the Rome to fall. But I digress

Realtime editing of the planet at all scales isn't a part of the project yet, but it's precisely the thing we want to do with it in this direction.

[monks]

Hi Cameni, you made me pretty excited when I saw your YouTube video, because this technology for modelling/ visualisation is what we've been looking for for quite some time. I've been with the ME-DEM project for about 6 or 7 years. Our web admin is Oshyan Greene who is with the Terragen team.

 Our terrain can basically be output at different resolutions. We're currently using a Global Mapper -> World Machine Pro -> GTS workflow (although WM can be skipped at a pinch for quicker demos).  World Machine will be used to process the terrain to it's full form.
 The shots in the gallery of the large scale terrain show what we term the "base dem"- the rough skeleton generated from contours- the base dem was a necessary step because using contours was the only way to guarantee correct flow across such a large model surface. Other modelling techniques don't really cut it.
 So, the base dem has no fractal detail or erosion on it.

 The Global mapper database is the persistent core if you like. We currently have around 150 layers of vector info. They get output as distrib maps with terrain. With projection info we can roundtrip between GM and other modelling software.

 The intention was to hook up GTS output directly to TG via XML. That way we get fluid sim generated texture maps direct to the renderer. GTS can also output "mega textures" to Viewing Dale too I beleive.

 The terrain was built from the outset to be georeffed and the river flow to be correct across the terrain in accoradance with the topo maps we are using as reference. The terrain is georeffed to Hobbiton = Sarehole in Birmingham and Minas Tirith = Florence, Italy. Surprisingly, many people are not aware that Middle Earth is Earth, but with an alternate history and Tolkien states unequivovcally in his letters that these two places are equivalent.
Lucky for us!  

 For large datasets we use the hf2 format which has compression and georeffed header options. It was created on the Terrain Summit forums which Oshayn set up (unfortunately gone now) by Aaron Torpy, RedRobes, Stephen Schimitt (World Machine), Mike (Global Mapper), Joe Slayton (Wilbur), Ray Gardner (Leveller), Johnanes Rosenberg (GeoControl).
 We've worked with these developers on and off because we came up against so many problems with trying to do what we wanted with this project. As a result I've tested for all of these softwares. The format was really the (unusual perhaps) meeting of GIS and game terrain devs/modellers, in part what the Terrain Summit was setup for.

 Hey cshawn..what a coincidence. I've got a thread over there on CartoGuild under ME-DEM.  
 No, not really anything in terms of renders as yet for ME-DEM.

The Tolkien Eastate, hmmm.. I remember the Minas Tirith Project which had a high profile on all of the CG boards, and I don't recall ever reading a single post about problems with the Estate, and it was right ontop of the films too but yes, I'm aware of the problems many people have had.
 Well, if it came to it, I'd be happy to bunker down and stay off the radar. With or without a web presence I'd still be doing this stuff.   I'd also be happy to get involved with other world building projects.

monks

[cameni]

Hi, monks. What video was that? Earth flyby?
Btw guys, thanks for the note about GTS - I wasn't aware of its existence, and it seems to be very interesting tool. I'll look at the hf2 format as well. It is a pleasure to talk with guys that already walked their path and know things

[monks]

No, well they're all amazing...but this more than anything:

Rendering V.


 I remember talking with Joe Slayton about this kind of setup a few years ago now where you would paint directly onto a sphere, or a spherical projection. The inverse projection needed.  What you've done there is amazing, We told ourselves at ME-DEM that it was only a matter of time really. You look on the Ogre boards for example, there's a couple on there. And there's been a crop of procedural planetary renderers over the last 5 years or so. But purely proc renderers are not much help to us.
 Also, everyone (the software) was using bitmaps. It was obvious, looking at the GIS world, that everyone needed vectors. World Machine stole a march on that (as far as the indie devs went) with its splines. VERY cool way to work that.
 I modelled the Misty Mts on the European Alps by extracting the ridge lines using a World Machine plugin created by Howard Zhou. It was able to isolate ridges or valleys at user defined levels of complexity. I then used those to trace the splines when I was testing for WM Pro 2. It was a novel idea. I think the next step is to get full import of shapefile data into WM. It would make the whole process easier.
 I could have used the real dem data of course with blending but I wanted some control over the terrain as orders of complexity (ie ridges and valleys at various detail). Kinda a vector-like representation of terrain. I think you could probably see a world in terms of watersheds.

 ..and the one where you model those roads across that little hill with vectors...

 If you ever want anyone to test a terrain editor like that, let me know   ...I would be very happy to help.

monks

[ZeosPantera]

I'd love to see a Middle Earth rendition within the engine as well

You know Cameni.. I'm sure nobody would mind if perhaps an island appeared somewhere near Australia that just happened to be shaped and built like middle earth.. I am not sure of the actual square mile footprint required but what an interesting adventure it would be crossing the sea of Krabgondia and landing on the shores of Gondor. I'm sure the google earth overview could even switch to their maps when you got close enough.

I could storm the black gates in a "Tatra-ish" vehicle!

[cameni]

No, well they're all amazing...but this more than anything:

Rendering V.

That's not our work but the work of Eric Bruneton and his students. We didn't get to this phase yet, although the vector edits used for roads and pads are one of the tools that count there. I need to extend the generator functionality a bit before I can move on to realtime world editing, but it will come - initially with support for realtime crater creation. The problem here is to make it happen across all levels of detail. That video is from before the time Eric added fractal refinement, but I think he still cannot go down to grass with it anyway.

..and the one where you model those roads across that little hill with vectors...

I think you refer to one of Eric's videos here too
Our road placement video went through a forest, I think.

[cameni]

You know Cameni.. I'm sure nobody would mind if perhaps an island appeared somewhere near Australia that just happened to be shaped and built like middle earth.. I am not sure of the actual square mile footprint required but what an interesting adventure it would be crossing the sea of Krabgondia and landing on the shores of Gondor. I'm sure the google earth overview could even switch to their maps when you got close enough.!

Sure, we even tried this back in early times, but using a very crude heightmap of ME. And I know it would find many fans, no doubt about it.
The problem is that you can't mention it is Tolkien's world, or ME, or anything from there. And even if you changed the names, I remember Tolkien Estate going after someone making just the map with the same shapes of the mountains. But that's what one would expect from a lawyers' guild living off the stuff.

One Estate to rule them all,
One Estate to find them,
One Estate to bring them all
and in the darkness bind them.

[Redrobes]

Is that movie done with Outerra ? Its the one that ebuneton mentioned in the text. Or does he have a similar one and has copied your "fractal noise details". The text is not very clear. In any case that video or your videos are both the kind of thing that we would like to see.

If you would like us to play with the engine that would be great or if someone here who has it would like to take the data and play with it that would also be great. If people are worried about the linking of this engine with the middle earth then we could show some vids without mentioning the name or do something similar.

GTS was never released, its a bit of a toy app that I worked on for some years. I use Perl to generate the scripts and its managed by GnuMake to build the tile set up. It takes about 8 hours to run the simulation over all the tiles for a duration that gives some basic results and spits out all the layers which are then used in another app which is my programmable texture shader (on the CPU not GPU). The images in the galleries are 2D bitmaps from that. Those and the height map are put into the Dragon Flight 3D renderer which is real time but very small. Its a debugging tool so has no LODs or any kind of fractal noise or stuff like your wavelet compression etc. Thats free to download and try from here. Just run it in a directory with a height.bmp and an optional color.bmp file. It will do 16bit height as well but you need to create a modulo 256 bitmap and call it height_minor.bmp like the ones in the gallery. Heres cshawnsmiths island from the intro thread chucked into it.

[cameni]

Is that movie done with Outerra ? Its the one that ebuneton mentioned in the text. Or does he have a similar one and has copied your "fractal noise details". The text is not very clear.

Nope, Eric has been only inspired by Outerra and added some fractal detail in his last video, but not to the detail level of OT. Eric is more interested in replicating various phenomena mathematically, as he said his primary goal is to publish research papers. His approaches are more "correct" in terms of being capable to match the actual behavior (as in atmospheric scattering or ocean rendering), but also more demanding.

If you would like us to play with the engine that would be great or if someone here who has it would like to take the data and play with it that would also be great. If people are worried about the linking of this engine with the middle earth then we could show some vids without mentioning the name or do something similar.

You got me wrong - I don't care what people would do with the app, I was just saying that I myself would be reluctant to take on ME because of TE. And that's from someone who's genuinely interested in making the worlds known from literary works. With TE it's simply almost inevitable that you will receive a cease&desist letter and the beautiful project would end up in hiding.

As for the testing - I hope well be able to test it with you once we get it past the release of game alpha. For the alpha we need to update the mapping tool to run on GPUs, accelerating it quite a bit. It currently takes somewhere around a week of hard 4-core work to process data for Earth.
I planned to support the creation of other worlds as "mods" for it anyway, to allow for Shawn to test out the fantasy world of his novels.

I guess GTS algorithms could be reworked to run on GPU, that would make it a very cool component of the future world creator based on OT engine.

[Redrobes]

Ok - that sounds very promising then. We will definitely both be keeping our eye on the progress of the engine. I have always wanted a game where you can play in space and then drop down into worlds and explore. Its never happened yet. The closest I know is Mission 6 of Halo Reach - "Long Night of Solace". Exactly what I wanted but for just one planet and it had a fixed transition instead of the transition being interactive. Still, Bungies engine must be able to cope with planetary zooms. Worth looking at if you dont know what I mean.

GTS is written in C++ and the main data structures are already (planned from the start actually) formatted up in CUDA sized chunks. The algorithm is embarrassingly parallel so in theory it should be a short hop to OpenCL. Do you program in GLSL, CUDA or OpenCL ? I would consider putting it into the building of the terrain. Whether it could be done in real time I don't know so much. My issue for not going full CUDA already is that I think I would need to have huge IO instead of compute so I figured id get about a 5x speed improvement instead of the 100x that people show some things going at. If you have precomputed all the wavelets and cached them on the GPU then I guess your IO is much less than doing it all at full res. So I think my problem is closer to your 5 day job than the real time frame rate job. But yes, there are not too many people with code that computes the fluid flow over terrain. Its not a miraculous bit of code but its tricky enough. That Eric chap has a student who has got a video of river flow calculation in real time there. That is quite impressive too. But they are calculating the flow from preset  river banks. We want to know where to put the banks in the first place based upon the terrain shape. I still think that is harder.

Since I come across few people who program the GPU directly, id be interested in your thoughts about the compute vs IO of calculating fluid flow. I have not done much in the way of GPU shading and CUDA yet. Most of my apps use multiple cores of CPU. I was hoping that intel were going to do the Larabee with lots and lots of x86/x64 cores on one chip but they abandoned it. At the time of writing GTS which was a couple of years ago now, nVidia had the OpenCL drivers on the registered dev only so I could not get to use them and I was reluctant to use CUDA if OpenCL was going to become the standard for GPU coding. Once they opened up the drivers fully I was onto other things and it kinda got left out.

[cameni]

You are right, if the computation is too light it wont bring up that much in comparison to the IO. But a 5x speedup is still desirable when the whole thing goes for a week. Additionally, there will be more computation when it will be merged with the wavelet decomposition and compression. Also, as you said, if the algo operated on pre-compressed data, recompressing on the fly, the IO would be smaller at the price of heavier computation reqs (but when the CPU/GPU would be idle waiting for disk IO anyway). Also, I'm much used to GPU programming now, on CPUs I usually can't resist the itching to optimize stuff somewhat prematurely

We are using solely GLSL inside the engine (or rather a custom compiler into GLSL that gets us rid of some of the aspects of its poor design). We have experience with CUDA from previous work, but for the tools we want to use OpenCL as they may end up in a world builder product one day and we don't want to limit the target platforms.

Rivers .. my idea is to let the designer to mark points on continent border where there will be river estuaries, and have the algorithm trace the path up the gradient, generating tributaries and ending up in mountain springs. Note this would run on a non-eroded terrain, and the generation of river basins would effectively create the erosion patterns. I have to dig up my old notes somewhere

I think this approach is more in line with the way I see the creation of an artificial world. I want to sketch the continents, draw mountain ranges with a simple tool there, place the volcanoes and lone mountains here and there, place the estuaries or the main paths of large river where I know I want them to be, and then let a generator chew on it, creating a believable world. Then a touch here and there, some similar edits on more detailed levels .. things like that.

[C. Shawn Smith]

Heres cshawnsmiths island from the intro thread chucked into it.

My 3d applications basically showed the same thing ... and you can see very obvious flaws with the current painting of the height data (most especially around the coastal areas).  When the world tools are introduced, a lot of these problems will disappear if I'm still using this height map as a base.  However, Cameni's explanation of the tools would negate even needing to paint a height map .. which will be MUCH easier to deal with .

[Redrobes]

I think that by using a real time terrain engine with some real time fluid flow you would be able to raise and lower the terrain height and steer rivers to the desired place. That would be a huge improvement on the way were doing it.

The problem with designating the outflow estuary and asking the computer to plot it backwards is that the function uphill is a one to many. Think of it downhill. All the rain in a certain catchment area all flows into the sea at the one estuary. Therefore plotting the river route uphill would be a very difficult problem. You cant take the path of highest ascent otherwise streams running down valleys would divert off to the side and straight up the walls of the valley. Also, rivers split going uphill so you would have to know the terrain ahead of the path in front of you. I think its easier to plot it going down hill where for every point you can say that water takes the line of steepest decent and joins to form larger rivers.

The complications arise when you make the terrain dynamic. If it erodes or the amount of water falling is not steady state then small changes in the erosion of the terrain can drastically change the river direction. Once a basin has filled and overflows, a small erosion on the exit can catastrophically make the basin empty and if a single flow stream has too much water in it then it can fork and flood causing new tributaries which may erode and dominate the flow. Its not at all clear at the outset where a river will be without some fluid flow down it to check. Its a more difficult problem than first appears.

If you could let rain fall evenly over the terrain or, like GTS, use a rain mask or a climate model then all those drops can flow until it hits the sea. If you could in real time monitor and adjust the terrain then that is what Monks calls the modelling in the rain approach which we think is the ultimate way to make the terrain. You need to be able to change the climate dynamically, push up mountains or flatten areas and let the water, or even snow too, do its thing.

GTS does this but not in real time. Not in a time frame that is even close to real. Its a slow process which you can get a snap shot now and again to see how your doing. You might do 500 iterations of flow and then get another frame spat out. On larger terrains like 2000 elements sqr its about an iteration per second. So a few mins between updates. We tend to run it on a small area quite quick and then let it run the same script again on the whole tile. If that looks ok then we let it do all the tiles like that which is hugely slow and would benefit from some serious acceleration or a PC cluster.

If I have a little time ill run GTS over your island and show some demo pics just for kicks. But I would not expect it to look great. It does some stuff great but not others. In this case I don't think it will look the way you would want it to.