I would like to design and texture a landscape in blender, and then render it in a custom openGL engine. designing the landscape is pretty self explanatory, however i don't get how to "brush" on materials onto the landscape. The engine should know what texture to use for each face (and thus have vt texture coordinates), but also know what material the object is made for in order for walk sound and stuff like that.
This is what i mean by "landscape", excluding houses, trees, grass, and possibly boulders:
You need to create mutliple material in blender, which have their own texture, specular, etc... and then when you design your landscape, asign proprely each triangle to the correct material, and then do your uv.
When you render the landscape, you usually group rendering per group of triangle sharing the same material (usually linked with the same shader too), and you can keep the material created in blender.
Well not really by hand since I think you can do your geometry and then select all triangle you want to asign the material too, and set it directly. You can start asign at the beging and all triangle created after that will use the same material than their parent too.
Finally, here comes the wall Trump will be jalous of :p
I'll describe the creation of terrain step-by-step:Modelling
Yesterday on IRC you mentioned that you wanted a system that can handle overhangs and caves, so I've made this terrain to test things on:
I'm going to assume you already know the basics, so I'm going to describe this part rather quickly.
I started off with a grid mesh of 100 vertices long and wide (shift-A -> grid).
You can then start creating hills and mountains with the sculpt tool, which acts very similar to common terrain-editing software. First go to symmetry and turn off symmetry on the X-axis, unless you want symmetric terrain. Then you just use the brush to raise and lower the terrain with the add/subtract buttons in the brush settings.
To edit the terrain in a more detailed way, switch back to edit mode. You can then move vertices around normally again. On terrains you'll regularly want to have smooth shapes, so it can be useful to press O before moving vertices so that it will interpolate the vertices close to the selection. Use the scroll wheel to increase or decrease the area of influence. You can also use this method to create the hills/mountains if you don't like the brush tool.
Here's how moving vertices normally (left) compares to moving them after pressing O:
If you want to work with large areas at once, it can be useful to select them in a method similar to painting. Press C, then you can add vertices to the selection by dragging over them with left click and remove them from the selection by dragging over them with middle click. Right-click anywhere to stop this "paint selection" mode. This can work in combination with the O thing described above, which is useful if you want to raise or lower a large area (this is how I made the canyon in the example terrain).
Now we can start on the important thing: rendering the textures. Method 1: Multiple materials
The method you were planning yesterday involved several materials and setting them per polygon. While this would be a more efficient method speed-wise, in most cases it isn't really recommended. I'm still going to do it in this example though, so you can easily see the advantages and disadvantages of both methods.
To do this, we first need a few materials to decorate our terrain. The way the materials look in Blender have no effect on the end result, just make sure that you have the same amount here as you have in your engine. I created those 5 materials, which are all solid colours:
I plan to use the grass material most, so I selected the grass material and all polygons of the models and clicked "assign" below the materials. To make it easier to assign the materials, I put Blender on "face select" mode (the button is below the 3D view).
The 2nd material is the stone material, so I started selecting the faces I want to use for stone. The mountain is a large part that's mostly stone, so I set Blender to also select hidden faces (the "limit selection to visible" button next to "face select"), pressed B and drew a box around the mountain:
Then, you can refine your selection by pressing C and "painting". By left clicking and dragging, you add to the selection. By dragging with middle click, you deselect the faces you "paint over". To end this "paint mode", right click anywhere on the 3D view. Eventually the selection looked pretty much like this:
There are more areas that would use the stone material on the map, but it's easier to work in steps than to select absolutely everything that uses the same material at once. So select the stone material and click "assign".
If we now look at the terrain you'll already see one of the downsides of this method:
Each polygon can have only one material, so there's a hard transition between multiple materials. The transition is also a very jaggy line. You can somewhat avoid the jaggy line by moving the vertices a bit but there's no easy way to make the transition smooth.
The other materials were then applied in a similar manner, which gave me this result:
The blue plane is just meant to illustrate water, but it is completely separate from the terrain mesh (which is also usually the case in game terrains with water).
I also wanted to be able to take comparing screenshots of terrains being used in an engine, so I imported the terrain mesh into Unreal Engine. The texture mapping here was done with triplanar mapping, I'll come back to that when I explain how the materials work. You could still achieve a similar result with standard UV mapping if you unwrap the mesh properly first.
(https://s21.postimg.org/69ripry5j/temp.png)Method 2: Alpha maps
To be able to use an alpha map, you're going to need to set the UV coordinates. For stuff like character models you'll want to do that manually to make sure you'll end up with a texture that's easy to work with in an editor such as Gimp or photoshop. However, for a terrain this is different. You'll mainly want to use the alpha map as efficiently as possible, and its structure doesn't really matter since you usually don't use an external editor for the alpha map.
The way I prefer to do this is by letting Blender generate them automatically. Press U -> Smart UV Project to start the script. For me, the settings below seemed to work pretty well, but always check the generated coordinates to make sure that there are no overlaps (for me, the default settings had trouble with the overhangs). I'd advise you to set the island margin to at least 1% since otherwise some texels may be shared by multiple regions, which is not what we want.
This mapped the UV coordinates like this:
There's still quite a lot of empty space in it so it's not ideal, but you can edit the UV layout in the same way as you edit a 3D model so you can manually alter it a bit. For this example I'm going to keep it like this. Partially because I don't have a lot of time, partially out of laziness ;)
In the previous example, we used 5 different materials. I'm going to create a simple material based on a 3-channel alpha map (RGB), so I'm limited to 3 materials. You can also use a 4-channel alpha map (RGBA), but I can show RGB images more clearly on screenshots. You can also use 2 alpha maps to have up to 8 textures per region, but I didn't do this since I also wanted to show what to do when not all your materials can be used at once.
To get around the 3 texture limitation, I'm first going to split the map into 3 'biomes'. Then I created a material for each biome:
Then I set each 'biome' to its material, which gives a result that somewhat looks like our previous attempt: (I moved the water down a bit so that it doesn't cover the model)
Note however that these zones will not be directly visible in the final game. Because of this they don't have to be that accurate, just keep in mind that you're limited to 3 textures in each one.
Try to plan these regions carefully, because you'll want to be able to hide the borders. In the beach region I'll use the sand, dirt and grass textures, in the default region I'll use the grass, dirt and stone textures, and in the mountains, I'll use the grass, stone and snow textures. Grass and dirt are used in both the beach and default regions, so as long as I'll use only those texture at the border between them then the seam will be completely hidden. On the border between mountains and the other regions there's only grass, so that seam can be hidden as well.
Now we can finally start to work on the alpha map. In the UV screen we create the texture image with all texels set to one of the primary colours. I chose green because I'm going to use the green channel for grass, which is the most common texture on my terrain. Make sure to use the RGB sliders to make sure you have one primary colour set to 1 and the others to 0. I let the dimensions at the default 1024x1024, but you may want to increase it for large terrains. Downscaling is easy with any image editor, upscaling is harder. Besides, alpha maps often contain large areas of the same colour so they should be compressed pretty well.
Here's a screenshot of the settings I used:
To be able to properly paint the alpha map, you'll have to set the alpha map as texture for each of the materials.
Now we can finally start painting the alpha map. Set Blender to "texture paint" mode. This will show the alpha map (which is currently completely green). To look back at the regions, you can press the tab key to switch back to edit mode, which display material colours rather than textures.
Before you make the first stroke, decide which channel you'll use for which texture in which region. I'm going to do it like this:
|Default region||Beach region||Mountains region|
I'm first going to paint everything that should be red, so I set up the brush to paint fully red. You can click those coloured rectangles under that colourful wheel to get a menu where you can set that colour with the sliders. Set red to 1 and the others to 0. Also make sure to set the strength of the brush to 1, otherwise you'll get overlayed textures on your final terrain.
After painting all areas in the correct colours we can once again see one of the downsides of this method:
The previous method looked simpler in Blender than in the engine, but it still looked natural and gave a good impression of what the final result will look like. Unfortunately we now get this weird, alien look that looks absolutely nothing like what we hope to achieve. Anyway, save both the model and the texture (they are saved separately, click image -> save image in the UV view to save the alpha map texture) and import them into the engine. Depending on your engine there may be built-in tools to render something like this properly, but I'm going to assume no system like that exist yet in your engine and I'll first explain how the materials work.The materials
I'm going to explain how these materials work, and give examples in Unreal Engine. The Unreal Engine shaders read almost like flowcharts, so it should be relatively easy to read (also, the last time I did GLSL was several years ago).
The texture mapping will be done with triplanar mapping. This means that the textures will be mapped based on the world coordinates. This is done based on 3 planes: XY, YZ and XZ.
To get the sample for a plane, scale the world coordinates by a factor (so we can resize textures) and then just use 2 of those coordinates as UV input for a texture sample. If we use the X and Y coordinates, then we get the sample for the XY-plane:
If we use that as a material on a sphere, it will look something like this:
For the other planes, it will look almost the same, just rotated 90°.
By copy-pasting twice we can easily get the samples for all 3 planes, but now we still need a way to combine them. We'll do this based on the normal, so that the final colour will be most similar to the sample of the plane the polygon is most parallel to. We could simply use the absolute of the normal for that, however this will blend the samples a lot and would cause blurry textures (red, green and blue in this image correspond to the YZ, XZ and XY plane samples, respectively).
We get a much better result if we raise the normal to the 4th power (or, multiply it with itself twice) and renormalize:
If we multiply all samples with their respective weights calculated from the normals, then we get the full code for triplanar mapping:
The "pebbles" texture I used here has very distinct areas so the result doesn't look that great, but it does a good job of showing how the samples are chosen and smoothly blended together at the seams. On more uniform textures such as the ones I used in the terrain the result looks a lot better. If you want to have a road on your terrain with a texture similar to this, then it would still look ok as long as that road is rather flat. The seams mainly become an issue with polygons at angles close to 45°.
This is an interesting page if you'd like to learn more about triplanar mapping. (https://gamedevelopment.tutsplus.com/articles/use-tri-planar-texture-mapping-for-better-terrain--gamedev-13821)
Now to use the alpha map we basically have to copy what we currently have 3 times (once for each texture), and multiply each value from the weight that we read from the alpha map, then add those colours together. Reading from the alpha map is really simple, you just take a sample from it at the UV coordinates of the current fragment:
Our material suddenly got quite big, but it's finally complete now:
That's one region done. For the other 2 regions, we just create a new instance of the same material, and swap some of the textures.
When we then apply the materials to the terrain everything looks a lot smoother than in our previous attempt, and the seams between regions are completely invisible.
Visually, the technique with alpha maps gives the best result, especially on low-poly terrain. It's a bit heavier on the graphics card but modern GPUs should have absolutely no problem rendering it. It's also quite easy to circumvent the 3-4 textures limit if you split your map in several regions.
I know I kinda suck at explaining, but I hope you still enjoyed reading this wall ;) . If anything is still not clear, just ask and I'll try to clarify it a bit more.EDIT:
I forgot to include the download link to the example models and alpha map. You can download them here. (https://drive.google.com/file/d/0B_kOEGPSxGbXSjZuQWhUaHNzT2s/view?usp=sharing) I unfortunately can't include the textures since they are included in Unreal and thus copyrighted by Epic Games, but you should easily be able to find similar textures on a site like opengameart.org .
It will work, it will just have visible seams.
I just implemented the UV coordinate technique you described as a "low quality" mode of my material (which actually is a realistic application of this since triplanar mapping requires more texture lookups, but I mainly did that to easily be able to toggle between both modes). The grass and dirt textures I used (well, grass and rust, since there was no dirt example texture, but that worked too :p ) are very uniform so it doesn't stand out that much, but if you look at the middle of the top screenshot you see a seam in the textures.
If your texture contains more contrast then the seam will be far more visible when you use UV coordinates, while triplanar mapping will still neatly tile the textures.
Triplanar mapping usually has a higher visual quality and modern computers should have no problems at all rendering it. This is why I advised it.
Your terrain looks quite low-poly, so unless you have a huge render distance, it may still have a good performance even without LOD. On modern graphics cards, performance depends much more on the amount of pixels than on the amount of triangles (especially if you calculate lighting and such per-pixel). You could still give it a try if you're doing it mainly for the programming experience though.