My View David's view on anything and everything…

27Nov/13Off

Multi-Threaded

The next TerreSculptor build release will be available soon.
I had estimated for the end of November, but it looks like it will be sometime around the first or second week in December.  As usual, I have coded significantly more than what was originally planned for the build release.

This next build release will include the following features and more:

1.  The ability to open and save a TerreSculptor project file.
Please note that the initial file version will be limited in what application areas are saved to the file, such as the current Grid and Viewport settings, etc.
Additional application areas will be constantly added to the project file format over the next months of development, including the upcoming Terrain Stack which is where a lot of the application's future capabilities will be implemented.

2.  The ability to open and save Noisemap Generator settings to file.
Have you just created an awesome Noisemap?  Now you can save the settings to a file and open it again whenever you need.
The ability to add to and manage the dropdown list of Noisemap Presets is coming soon.

3.  A new Hold-and-Swap buffer for all Noisemap Generators.
A click on the Hold toolbar button will grab a copy of the current Noisemap parameter settings into the Hold Buffer.
You can then perform a few tweaks and adjustments to the Noisemap without losing the original settings that are in the Hold Buffer.
Then clicking on the Swap toolbar button will allow you to toggle between the current settings and the settings in the Hold Buffer for visual comparison.

4.  All Noisemap Generators are now fully multi-threaded.
The threading code design is not limited to a specific thread count and fully supports unlimited threads.

The imposed thread-count limitations are:
- The number of supported processors in the installed version of Windows.
- The number of threads supported by the installed physical processor(s).
- The number of lines in the Noisemap resolution (the length dimension).

A computer with a quad core processor and a noisemap that is equal or larger than X×4 resolution can use all four threads.
A computer with a quad core hyper-thread processor and a noisemap that is equal or larger than X×8 resolution can use all eight threads.
A computer with two xeon octal core hyper-thread processors and a noisemap that is equal or larger than X×32 resolution can use all thirty-two threads.

The use of more threads when executing algorithms is typically not a linear gain.  In other words, having twice as many threads does not always result in an execution time that is one-half.  The running threads have to share certain system resources such as access to memory.  So gains are typically slightly less than the total number of threads.

Intel processors with Hyper-threading do not supply the same throughput on a hyper-thread as they do on a core.  Typical performance benefits for hyper-threading are gains of 10% to 20%.  So a quad core hyper-threading processor will typically only see between 4.5× to 5× the performance increase with all 8 threads.

There is no benefit to specifying more threads than there are physical cores plus hyper-threads available.  Typically doing this results in negative performance improvements due to thread context and switching overhead.

I have an AMD Opteron 16-core system on the way and will post threading statistics on it in the very near future.  An Intel Xeon dual 8-core Hyper-thread system will be purchased in the near future for additional threading tests up to 32 threads.  These systems have to be workstations since many servers do not support the installation of a gaming-level PCIe x16 video adapter required to run TerreSculptor.

Multi-threading capabilities are also planned for many of the TerreSculptor Modifiers, Extractors and other Devices.

Here are some Noisemap Generator results using one of my systems:

Intel Core i7-2600K @ 3.4GHz 4-Core Hyper-thread (8 thread) Processor
16GB Mushkin Redline DDR3-1600 CL7 RAM
ASUS Sabertooth P67 Motherboard
Windows 7 Professional 64-bit
Heightmap dimensions of 4096×4096
Gradient Noise Preset: Eroded Rivers

1 thread = 26330 ms (26.33 seconds)

2 threads = 13190 ms (13.19 seconds) (Task Manager 25%)

1.996 times faster (199.62%)
Only 50.09% of the time that was required for a single thread execution.

4 threads = 7780 ms (7.78 seconds) (Task Manager 50%)

3.384 times faster (338.43%)
Only 29.54% of the time that was required for a single thread execution.

8 threads = 4890 ms (4.89 seconds) (Task Manager 100%)

Note that this will be 4 cores plus 4 hyper-threads.
5.384 times faster (538.45%)
Only 18.57% of the time that was required for a single thread execution.

-eof-

Filed under: TerreSculptor No Comments
24Oct/13Off

Under Development 2013-10

I am currently finalizing the development of the TerreSculptor proprietary file format, which is used for loading and saving the main application "World" files, the Noisemap Generator preset files, and additional current and future application-specific files.

The initial application file open and save, and Noisemap preset open and save, will be in the next build update that will be available within the next few weeks.  During the Alpha and Beta phases of development, the file formats will be backwards compatible for any bug changes or errors, but may require re-saving of data files at the Release Candidate in order to be fully compatible with the final release version.

A lot of time and attention has been devoted to the design and layout of the proprietary file format, which unfortunately is rarely done by many other software companies.  One of the main goals for the proprietary file format is full lifetime backwards compatibility through all software versions.  Additional features include: 64-bit file length support; block-based design; per-block features including compression, blobs, embedded files; CRC integration; file repair and recovery capable format.

-eof-

Filed under: TerreSculptor No Comments
18Oct/13Off

Build 290

Build 290 for October 17 2013 is available for download on the lilchips.com website.

See the Build.txt file included in the download or the build information on the Wiki for the complete list of changes and updates.

- The 3D Editor's maximum supported heightmap resolution has been increased from 4096x4096 to 65536x65536.

To put this into perspective, a terrain of 65536x65536 with 256 spacing and 1:2cm scale would result in a size of 335.5km x 335.5km = 112590 square km.

Results will vary on the actual maximum size that can be created or loaded based on current system memory fragmentation.
The current limit of the application runtime is a single contiguous memory block of up to 2GB which is the array maximum size in the Microsoft CLR.
In the future I will be investigating large array management code that will allow for heightmaps that are significantly larger and can be created even with minor memory fragmentation.

Since the 3D Editor now manages large heightmap sizes, the Heightmap Converter tool will be changed to a file format converter that supports unlimited heightmap size and most of the Modifier tools will be removed from it.

- The file import and export code has been dramatically improved. Loading and saving files will be faster and with better error management and event logging.

- The file import and export data type conversion of Real, Scaled, or Auto is now fully functional. I will document this setting on the wiki for each file type.

- There are a number of new file formats supported.
PNM (.pnm) numerous improvements and additional formats
SB (.sb) raw signed byte format
R64 (.r64) raw 64-bit format RAW (.raw) raw signed byte format
TGA (.tga) Version 2 format Extension Area block management

-eof-

Filed under: TerreSculptor No Comments
16Oct/13Off

Autumn

Where has the summer gone?

It is difficult to believe that it is October already.

A large amount of work has gone into TerreSculptor over the past two months. Most of this work has been to the file format and file read/write classes, including the refactoring of file classes and methods for better reuse, the addition of significant file data management functionality, and improving file performance and error logging.

During the next few months there will be additional improvements to some of the file formats along with the completion of other features including the Terrain Spacing, loading/saving the application TSW main file format, user-defined Colorsets, Noise Generator and Modifier Preset load/save, and the application color theme.

The next build update will be available for download later this week.  It is currently undergoing testing of the change list.

Filed under: TerreSculptor No Comments
18Aug/13Off

August Already?

You might have noticed that my last post mentioned that the next TerreSculptor build should be available in July.

Well, July has gone by and we are well into August.

It would seem that it has taken considerably longer than what I had estimated to get the latest set of fixes and features completed.

I typically get between 60 and 70 hours each week into software development, however, at the current half-a-million lines of code I have been spending a considerable amount of time in reorganizing and refactoring the code base to allow for easier maintenance and future features addition. This project really requires a team of two to four software programmers.

The next build release for TerreSculptor will be available in a few weeks time, hopefully by the end of this month.

 

A few of the changes and updates in this next build include:

  • Editor 65536x65536 Terrain Resolution. The editor maximum heightmap dimensions has been increased from 4096x4096 to 65536x65536. Resolutions this large will require 64-bit Windows and 8GB+ of memory.
  • Numerous additional import/export file formats.
  • Significant improvements to the file read/write classes. For better performance, better error handling, and a base class setup that will allow additional file formats to be added more easily.
  • Floatmap support. A lot of the code has been completed for the future support of floatmaps (floating point heightmaps). TerreSculptor will allow working in either standard 16-bit integer heightmap mode or 32-bit floating-point floatmap mode (requiring twice the memory), with simple changing between modes at any time if desired.

I have also been reworking the user interface in preparation for the change to a dark color theme similar to most other 3D applications.

 

I have decided to push the status of TerreSculptor back to Alpha.

Since the software is still free-for-use including commercial use, this delay is good for everyone out there who is using the software.

It was at pre-Beta since I was hoping to have it in full Beta by the end of this year. However, there are still so many features that I want completed for the first retail release that it will be sometime next year, with some of the following main features planned:

  • CryEngine metrics support.
  • Colorset Editor.
  • Layer Creator (currently the Extractor dropdown).
  • Mask Editor and Compositor.
  • Additional terrain generators.
  • Multi-Noisemap Generator.
  • Additional modifiers.
  • Additional Erosion modifiers.
  • Project file open/save.
  • Terrain LOD system.
  • Terrain Stack.
  • and much more...

 

I have also started drafting some How To tutorials for proper methods of obtaining and converting Digital Elevation Model files that will make it onto the wiki soon.

-eof-

Filed under: TerreSculptor No Comments

Recent Posts

Categories

Links