Since Terresculptor HMES is essentially a tool for use with game engine editors or other 3D applications like Blender or Max, it has to play nicely with system resource use. TerreSculptor will typically be running next to other applications, so it has to maintain a small GPU memory footprint, leaving the majority of GPU memory available for whatever main application is running alongside it. So what is the best way to provide multiple levels of terrain detail with a small GPU footprint and fast application response. This has been an area of design that I have thought about for a considerable time.
Why the GPU memory issue?
Let's do some loose calculations for the required GPU memory for various terrain mesh sizes.
Assuming a non-LOD simple triangle plane mesh with vertex, normal, color and index arrays, we require:
- 3 floats per vertex (12 bytes for vx, vy, vz)
- 3 floats for normal per vertex (12 bytes for nx, ny, nz)
- 1 uint for color per vertex (4 bytes for argb)
- 3 index uints per triangle (12 bytes per triangle), this could of course be optimized for ushorts if small terrain sections are used.
A 1024x1024 terrain mesh has 1,048,576 vertices and 2,093,058 triangles. So multiplying the vertex, normal and color (12+12+4) by 1,048,576 and the index (12) by 2,093,058 we get approximately 52MB (12MB vertex, 12MB normal, 4MB color, 24MB index).
If we multiply this up for larger terrain meshes we get approximately 208MB for a 2048x2048 terrain (52MB*4), 468MB for a 3072x3072 terrain (52MB*9), and 832MB for a 4096x4096 terrain (52MB*16). Now it becomes clear why we need to do some work with reducing our GPU memory footprint, a 4096x4096 terrain mesh won't even fit into a video card with 512MB, and requires more than 80% of the memory on a video card with 1GB. This is where LOD or Level of Detail algorithms come into play.
One method we can implement to save some memory is to split the terrain mesh into smaller sections such as 256x256, and only send those sections that are currently in the camera frustum to the video card. This works great so long as the camera is not panned far back and viewing the entire terrain.
Even if we implement a terrain mesh algorithm that uses lower resolution for sections that are further from the camera frustum, we still can use up a lot of GPU memory. For example a 2-stage 2x reduction LOD may render the near 10% of mesh triangles at full resolution and the far 90% of mesh triangles at 1/2 resolution, with a 4096x4096 terrain this is still 270MB (208MB*0.9 + 832MB*0.1). This is still more than 50% of a 512MB video card and almost 30% of a 1GB video card.
With Terresculptor we also require an LOD system with full resolution at the near camera area in order to support future mesh vertex painting, so this excludes other specific terran mesh LOD algorithms that for example reduce all triangulation based on adjacent terrain altitude difference.
The current LOD system in Terresculptor provides for three user-selectable LOD modes: Aggressive, Progressive, and None. This may be enhanced or changed in future releases of the software.
Aggressive LOD renders a low resolution proxy mesh, providing a very low GPU memory footprint as small as 13MB. This is perfect for shared use when another large application is running alongside Terresculptor, such as Unreal Engine 3, and when all you are doing is simple terrain creation or modification.
Progressive LOD uses a 2-stage reduction LOD as mentioned above. The 100% resolution near sections are required for future application features.
No LOD simply renders the full mesh, which on most systems is perfectly fine with 1024x1024 and smaller terrains.
This is a screenshot of the Aggressive LOD on a 3601x3601 SRTM DEM file, using a maximum of 52MB of GPU memory.
The system is an i5-750 with ATI 4870 1GB and Windows 7 x64.

This is a screenshot showing the terrain sections' bounding cubes, the area which determines section frustum culling.

-