Author: dgreen

New Workstation

All of the components are finally here for my new workstation.
I decided to go with the 24-Core Xeon W7-2495X processor, instead of the 16-Core or 20-Core model, as I may as well go all-in and get the top W2400 processor.
The W7-2495X has 24 P-Cores and it readily beats the i9-13900K with its 8 P-Cores and 16 E-Cores in multi-threaded workloads.
I had a difficult time getting the full amount of memory that I wanted, which was 1TB or more. I had to settle on 512GB since 8x 64GB RDIMMs was all that is readily available at this time.
I am hoping that the 128GB and 256GB DDR5 ECC RDIMMs will be hitting market over the next couple of years so that I can upgrade the memory to the full 2TB.
I am planning on creating three videos for my YouTube channel Tech Show section, showing the components and build of the workstation.

New Services

As soon as I have the new workstation build completed, I will be adding new services to the Services page on the web site.
The new workstation with 512GB of memory will allow me to work on large point-clouds and heightmaps up to 370,000 x 370,000 pixels.
When I can upgrade to 2TB then I can manage heightmaps up to 730,000 x 730,000 pixels.
The new services will include massive terrain heightmap conversion, creation, editing, stitching, and tiling, all for very inexpensive pricing so that it is affordable to everyone.

TerreSculptor Updates

The total number of supported file formats has now reached more than 160, with additional 64-bit TIF and GeoTIF format support.
The next build release of TerreSculptor will include an upgrade to Microsoft dotNET 4.7 from 4.6.
Microsoft dotNET 4.7 is included with Windows 10 as of the Creators Update in April 2017, and is of course also in Windows 11.
dotNET 4.7 is supposed to include new features for improved Windows 10/11 High DPI Screen Scaling support for WinForms applications such as TerreSculptor.
I will be implementing any changes for 4.7’s better support over the next few builds.

TerreSculptor Scaling

Out of the 150,000+ total combined installations of the free TerreSculptor 2.0 and retail TerreSculptor 3.0, I have had two people with really strange 4K monitor Screen Scaling issues.
On their displays, even when they switch to 100% Scale on the Display Settings in Windows 10/11, the TerreSculptor dialogs are still scaled and zoomed and stretched to 140%+, and usually distorted in shape.
It is as if their system has some really odd Font scaling setting.
So this causes issues on some of the dialogs with the custom controls and viewports.
Unfortunately, I have not been able to fix this issue for these two people, so I recommend that anyone interested in TerreSculptor be sure to try out the free 2.0, or 3.0 in demo mode, to make sure that the software renders correctly on your display.
Microsoft dotNET 4.7 is supposed to have additional features to support custom controls with better High DPI Screen Scaling, so I am researching this now to see what I can do to support these outliers with their display issues.

-eof-

The following are the software development tasks that are scheduled for possible release this year.
It is doubtful that every one of them will be added, but as many of these as possible will be completed.

Biome Update

The completion of initially three different Biome Devices.
These Devices will make creating Weightmap and Splatmap sets significantly easier.
The Weightmaps and Splatmap will also fully support the maximum heightmap size of 1 million pixels.
The goal is to provide a high resolution interface display of the splatmap up to 16384 pixels.
And to be able to automatically display the splatmap in the editor Colormap material, and allow for Color ID export (see the File Format Update).
This requires the ImageArray Update.

Camera Update

The Camera Update includes adding an orthographic view camera, and providing numeric edit controls for entering in the camera position for all view modes.
Bookmarks are also on the list of camera features.

Command-Line Update

The Command-Line system will allow for batch file scripting of a subset of features in the software.
This includes the conversion of file formats, and a limited set of modifiers during the conversion such as normalize and resample.

DEM CRS Support

The ability to extract the Coordinate Reference System from the header or meta tags of Digital Elevation Model files for the purpose of auto-scaling the terrain to its real world elevation range.

File Format Update

This update will add a number of new file formats to the software.
It is proposed to add EXR, FBX, IMG, PCX, and USD, although it is possible that not all of these formats will make it into the first update release.
The FBX file format will include exporting the heightmap as a mesh including splatmap Color IDs for use with Blender etc.

Generator Update

Additional Generator Devices are planned including Gradient Pyramid, Gradient Triangle, Gradient Slope, Texture Granite, Texture Wood, and others.

Geology Update

Additional Geology Devices are planned including Chasm, Crater, Dozer, Faultline, Fold, Mountain, Shelf, and Volcano.

ImageArray Update

This system is to provide a memory RGBA Image array that supports up to the maximum heightmap resolution of 1 million pixels.
The Windows APIs for images only supports bitmap arrays up to 2GB which is a 23170 x 23170 resolution RGBA image.

Project Open Update

This task is to set the editor properties for all scene objects to the values stored in the Project file.
This will set the main editor scene back to the settings that were saved, such as Camera position, Background properties, Fog properties, Material setup, etc.
This will allow for the editor to resume exactly where the project was last saved.
This update requires that the Camera Update be completed as well.

Quick Mask Update

A quick mask editor for the Terrain Stack is planned, that will allow for drawing basic mask shapes.

Splatmap Device

The Splatmap Device is a new Terrain Stack Device that will allow for selecting four masks or weightmaps on the Terrain Stack and saving them as a Splatmap file.
This requires the ImageArray Update.

V-BASIC Update

The V-BASIC Interpreter is a traditional BASIC style scripting tool built right into the editor.
It will allow for running scripts to perform most all of the functions available in the software.

Additional Updates

Additional partially completed updates will be completed as time allows, which includes the Multi-Thread Update, SIMD Update, and others.

-eof-

The new Intel HEDT High-End Desktop W2400 and W3400 processors are finally making it to Canada retailers.
I will be building a workstation using either a 16-Core W5-2465X or 20-Core W7-2475X processor.
These W2400 processors are in the latest Intel Xeon series and support up to 2TB of memory.
I would like to go with a W3400 series processor such as the 16-Core w5-3435X, which supports up to 4TB of memory, but the extra processor cost and motherboard cost (for 8-Channel memory) will probably push it out of my budget.

My plans are to use this new workstation with TerreSculptor and Unreal Engine 4 and 5.

TerreSculptor supports heightmap terrains up to 18 Exabytes, which is 2 Billion x 2 Billion pixels.
With 512GB of memory I will be able to test the software up to 370727 x 370727 maximum terrain size (370km x 370km at meter spacing).

With Unreal Engine 4 and 5, my current top-end computer with 128GB of memory can import heightmaps into World Partition up to 32km x 32km. This is 1024 sq km.
With 512GB of memory that will quadruple the area that I can import, meaning that my importing service will be able to handle heightmaps up to 64km x 64km or 4096 sq km.
Unreal Engine is poorly optimized for Landscape heightmap importing, so even with the heightmap data being half the resolution quality (UE 16-bit vs TS 32-bit), Unreal Engine can only import a heightmap that is a fraction the size that TerreSculptor can work with.
Epic needs to work on improving this.

The 64km x 64km 4096 sq km import would only be possible with Unreal Engine 5.0.3, as 5.1+ limit the import size to 16384 x 16384 (16km x 16km).
With World Composition Tiling I should be able to do right up to the full 370km x 370km that TerreSculptor supports, with 512GB of memory.
The actual World Composition size would be a tile set of 45 x 45 tiles of 8129 x 8129 which is a total world size of 365805 x 365805 or 365km x 365km.

I will have to wait until I get the workstation built, and perform a number of Unreal Engine tests, and then update my Services page for the additional new Landscape sizes that I will be able to support.
My current services are limited by my current top-end computer which is an AMD R9-5950X 16-Core, ASUS ROG X570 Crosshair Hero, 128GB DDR4, ASUS ROG RTX-3090 24GB.
Bumping up the maximum memory that I will have available is more important than cores, so starting off with 512GB upgradable to 2TB will be a great start.

With 2TB of memory (the workstation future maximum) TerreSculptor will be able to work with heightmaps up to 741454 x 741454 (740km x 740km).

-eof-

Over the next few months the following features are under active development.

Splatmap Device.

A splatmap device for the Terrain Stack is under development.
This will allow for choosing any three or four masks from the Terrain Stack items and packing them into the ARGB channels of the Splatmap texture.
This feature will be using the new ARGB Image class that has been under development that supports up to 1 million x 1 million image size.
If the splatmap device dialog has sufficient space on it, I am considering adding the ability to also save tiled output.

The Biome menu items.

I currently have three biome devices under development: Beach/Island, Beach/Mountain, and Meadow/Mountain.
These will be composite weightmap extractors that create a splatmap output.
These biome devices will be substantially easier and faster for creating splatmaps out of terrains.
If the biome device dialogs have sufficient space, I am considering adding the ability to also save tiled output.

3D Preview Texture View.

In order to facilitate the biome devices and additional future features, the 3D Preview Control will be getting a texture view mode.
For the biome devices, this will allow for viewing the extracted weightmap texture on the 3D terrain in the preview window.
The design goal is to display a grayscale texture for a single weightmap view, and red, green, blue, and yellow textures for multiple weightmap views.

More Geology menu items.

The initial design document is drafted for the upcoming Mountain Geology Device.
This will support multiple mountain types, and will allow blending a mountain into any region of the heightmap.
When placed onto the Terrain Stack, hundreds of mountains can be created on the terrain.
Initial design documents and some output tests are also drafted for the Chasm, Crater, Faultline, Fold, and Shelf geology devices.

More File Formats.

EXR and FBX and PCX file formats are under development.
I am also researching the file format information for IMG DEM format.

DEM Header Support.

Additional support for Digital Elevation Model header values is under development.

DEM Elevation Auto Spacing.

I am currently looking into being able to automatically set the Terrain Properties Spacing Y value to the imported DEM data elevation range.
This would benefit those who are using TerreSculptor in their pipeline for converting DEM files into use with Video Game Engines.

YouTube Videos.

For the months of April, May, and June, I have a lot of videos under development for the YouTube channel.  This of course takes away from my time to develop the software.  So the timelines for the above items a bit fluid.

-eof-

A lot of work was completed on TerreSculptor in the past two months.

In addition to all of the work mentioned in the last blog post, the new BigPNG is also now complete.
This allows PNG file formats to support files up to the full 1 million x 1 million datamap size.
The TileEx Creator is also completed, which allows for tiling the full 1 million x 1 million datamap size.
Plus more Device multi-threading and a lot of other smaller updates.

I have been spending a lot of time in the past two weeks updating the original TerreSculptor Reference Manual for Version 3.0 of the software.
This PDF format manual is a 400 page book on everything about TerreSculptor.
This manual will become part of the regular install distribution files.
As new features are added to TerreSculptor, the manual will also be updated.
The latest version of the manual will also always be available on the Google Free Assets Drive (see your software Help menu for the link).

Starting in a few days will be the work on two new major features, the new Biome Menu item for Automated Splatmap creation, and the main viewport Terrain Auto-Material Material.

The new Biome Menu will be used for biome and splatmap extraction features.
I have been thinking about how I will develop the initial Automated Splatmap creator, and it will initially have at least six to eight RGB and RGBA/RGBAK Splatmap auto-creation tools.
The design will be a single dialog with multiple sliders that allow for specifying the settings for most of the more common Splatmap configurations.
Such as Beach Flat/Steep, Grass Flat/Steep, and Mountain Flat/Steep.
And adjusting the sliders will automatically create a full Splatmap based on the settings, that can be saved to disk.
This will automate the task that currently requires using multiple Weightmap Extractors saved to disk, and then assembling it all manually in the Splatmap Creator.
I am still considering how to perform such things as an automated transfer and viewing of the resulting Biome Splatmaps in the current Terrain Colormap Material.
This will also require creating a set of common coloring setups for the current Terrain Colormap Material system.

The new main viewport Terrain Auto-Material Material will allow for a higher quality rendered view of the terrain.
This will improve visualization of the terrain system.

Some of you may have also noticed the Mesh List Function Panel button and panel contents.
This is a future feature that will allow for placing a variety of primitives into the viewport scene, such as rough buildings and location placeholders.
This will also include such things as scattered trees for terrain size perspective.
The initial Mesh List functionality should be available in 2023.

I also still have some new Filter and Geology Devices in the works, which will be showing up randomly over the next few months.

-eof-

Development has been going quick on TerreSculptor 3.0, with a large number of new features in the past couple of months, and new build releases typically every 5 to 8 days.

As of January 9th the following new features have already been completed:
– The integration of LibTIF.Net and support for Large TIF and GeoTIF files up to 1 million x 1 million resolution.
– The new 2D Texture Plane view in the viewport for viewing masks and weightmaps more easily.
– All of the latest 11+ Devices have been added to the Terrain Stack, bringing the total number of Stack Devices up to 111 (still to be updated is the Composite Weightmap Extractor for 112 Devices).
– RAW Interleaved file format has been added, with support for up to 256 channels, and loading and saving into any channel number.
– All of the Gradient and Shape Generators received Previews for live editing of their parameters.
– Plus dozens of minor updates.

What will be happening during Q1 2023?
– More Devices will get multi-threading.
– More file formats: I am currently working on Large PNG support, Open EXR, FBX, PCX, and others.
– The new TileEx Creator. I decided to rename the Split Device to TileEx Creator. This version of TileEx will support tiling the full 1 million x 1 million heightmap size, with only support for grayscale formats.

What about Q2 through Q4 of 2023?
– The new Auto-Material Material which will color the viewport terrain with various gradients based on elevation and slope.
– The Biome menu will be available with the ability to create complex Splatmaps directly in one dialog.
– With the Biome features I would also like to implement automatic biome extraction.
– More Devices, including a bunch of new Geology Devices and more Filter Devices.
– Updates to the Convolution Filter Device with support for up to 9×9 size filters, providing new filters such as motion blur and others.
– The Quick Mask Creator.
– Additional work on the Mask Editor. This tool will eventually have complete painting and brushes and mask creation.
– I am considering adding another toolbar to the Terrain Stack so that viewing of the Datamap, Mask, ResultMaskA and ResultMaskB can be done in the viewport. Plus possible Mask Overlaid viewing.
– Plus whatever else I can fit in!

As a solo software developer, I typically work 12 to 16 hours a day, 7 days a week, on this project.  In order to bring you the new and updated features as rapidly as possible.

-eof-

The last month of 2022 saw a lot of updates to the TerreSculptor software.
The maximum heightmap size was increased to 1 Million x 1 Million, this will, however, require a computer with 4000GB of memory.
A good number of the Devices were Multi-threaded, with more still to complete.
There were seven new Devices released in December: Border, Dilate, Erode, Sinewave, Smear, Spin, and Stretch.
That brings the recent Q4 total up to ten new Devices including: Feedback, MinMax, and Translate, and the software total up to 112 Devices.
Note that Devices only get added to the main TSworld file format and the Terrain Stack after a few weeks of exhaustive testing, so the recent set of new Devices will be available on the Terrain Stack in Q1 2023.
Plus features added to the Terrain Stack, and interface updates such as being able to pan the 1:1 3D Preview mesh.
If you check out the Release Notes text file distributed with each new build, there is typically 50 or more new items and changes with every release, and build releases occur around once every week.

Starting in January of Q1 2023, there will be more Device Multi-threading.
Another six more Devices are planned for Q1 of 2023, but this time for the Geology area of Devices.  In addition to the new Geology Devices, there will be a few more Filter Devices.
Plus libtif file format integration to increase the maximum possible TIF file size to hopefully match the heightmap maximum of 1 Million.
Plus more GeoTIF file format compatibility with the libtif library.

High on the development list for completion is also the 2D Plane Texture View in the main viewport for viewing Masks and Heightmaps as a flat grayscale texture.
An Orthographic Camera view, plus updates to the existing Cameras.
Plus the Auto-Material Material system that allows for easy designing of complex terrain coloring by slope and elevation, which will also introduce some new color gradients specifically for this material.

Additional high priority features that should get some development time in 2023 includes: libpng to increase the PNG file format support and maximum size; EXR file format; FBX File Format; the Split Creator for exporting tiles up to the maximum heightmap size; DEM Scaling in the import/export pipeline; possibly integrating MapBox online DEM sources; adding a more complete Command Line Parser, adding Third-Party Plugin capabilities; the new Quick Mask Device, the Continent Creator, and more work on the Mask Editor; multi-lingual support for a few major languages; and much more.

Further future features include: additional Noisemap Generators; additional Extractors; additional Erosion Devices; more File Formats, Texture Generators; and much more.

-eof-

The BigArray Update was completed in about one quarter of the time scheduled for its development, with the first beta release within about four weeks after starting on it.
TerreSculptor’s code base for heightmaps now supports resolutions up to 2 Billion x 2 Billion which is 18 Exabytes.
The build release for Version 3.0 is using a maximum resolution value of 262,144 x 262,144 which is a 256GB heightmap.

The next major update under development is the Multi-Thread Update.
This will update as many modifiers as possible to support multiple processor threads for faster execution times.
A four-core processor will typically get a boost of around 350% faster code execution.

Multi-threading as many of the modifiers as possible will help with faster execution times for larger heightmaps.
This will also be noticeable on the Terrain Stack where multiple modifiers can be executed sequentially.

Some of the modifiers use algorithms that cannot be easily multi-threaded, and those will remain as single-threaded methods.

The Hydraulic Erosion modifier should be getting a performance boost, as should the Rain Erosion modifier.

In addition to the Multi-Thread Update, the next few months will also include the following major Updates:
– File Format Update 1: TSmap file format support for BigArray sizes including GZip – completed as of 332 Beta.
– OpenGL Update 1: Changes to some of the OpenGL functions for the viewport world – completed as of 332 Beta.
– Filter Update 1: Add Filter menu and move filter type modifiers to menu – completed as of 332 Beta.

Once the final build is released for the completed Multi-Thread Update, then the following months into Q1 and Q2 of 2023 will be used for the following major updates:
– File Format Update 2 which includes support for LibTif.net including better TIF image and GeoTIFF support.
– Filter Update 2 with new Filters.
– Modifier Update 1 with multiple new Modifiers.
– OpenGL Update 2 with larger Viewport World setup.
– System Update 1 with additional system information for the motherboard.
– and hopefully Material Update 1 with the new Auto-Material.

My big list of future features is slowly getting completed, but still has more than four dozen major updates and thousands of minor updates.

-eof-

As of next week I will be starting on the BigArray Update for TerreSculptor 3.0.
The BigArray Update is expected to take at least two to four months to complete, so no updates will be released during this time as this is a change that requires a complete code update.

The BigArray Update changes how the memory array system in TerreSculptor manages heightmap files.
The current array system is limited by the C# API to an 8GiB Floating-Point array.
This means that the current maximum heightmap resolution is 46329 × 46329 using this array type (46329 is the the square root of 8GiB).
For the BigArray Update I am changing the entire array system so that it allows for a maximum resolution of 2 Billion × 2 Billion values.
Initially the software will use constants and tests for limiting the maximum heightmap size to 1 Million by 1 Million pixels.
A 1 Million by 1 Million pixel heightmap would require a computer with more than 4TiB of memory.

The BigArray system will be completely memory based, so the maximum heightmap resolution will be limited to the installed computer system memory.
I am staying with memory based arrays rather than file based mainly for performance reasons.
This is because the current performance of DDR4-2133 is 40 times faster than an SSD at SATA 500MB/s speed.
File based arrays would mean that something that requires only 1 minute to execute in memory could require as much as 40 minutes or more to execute disk based with page swapping on a fast SSD.
I am also not simply changing to a tiled array system using the limited C# arrays, like most other heightmap software that is currently out there (you know who you are), because that results in edge issues between tiles for many of the modifiers and filters.
So in order to work on larger and larger heightmaps, more computer system memory will be required.
Note that as you go larger in heightmap size, the time to operate on that array data increases, so faster computers with more cores will also be necessary.

Many of the modifiers in TerreSculptor require multiple temporary arrays to operate on the heightmap.
I will be updating more dialogs to show how many arrays they require for operation, so that better estimating of total memory footprint can be accomplished.
There will also be updates to the supported file formats to allow supporting this change to larger heightmaps.

I am also working on multi-threading as many of the modifiers and functions as possible, so that larger heightmap sizes operate at the highest performance possible.

The maximum heightmap resolution for common memory setups is as follows, note that these numbers do not include the operating system footprint, so the actual available maximum size will be a bit less.

16GiB = 17179869184 bytes
= 4294967296 floats
= 65536 x 65536 maximum resolution

32GiB = 34359738368 bytes
= 8589934592 floats
= 92681 x 92681 maximum resolution

64GiB = 68719476736 bytes
= 17179869184 floats
= 131072 x 131072 maximum resolution

128GiB = 137438953472 bytes
= 34359738368 floats
= 185363 x 185363 maximum resolution

256GiB = 274877906944 bytes
= 68719476736 floats
= 262144 x 262144 maximum resolution

512GiB = 549755813888 bytes
= 137438953472 floats
= 370727 x 370727 maximum resolution

1024GiB = 1099511627776 bytes
= 274877906944 floats
= 524288 x 524288 maximum resolution

-eof-

TerreSculptor 3.0 2022.233 for August 21st is now available.

The main features of this release are:
– The PNG Image importer supports UE5 Landscape Tile Export.
– The Image and ImagePlus folders are persisted between sessions.
– The Normalmap Creator supports saving to TGA file format.

For the future build releases over the next year, the main features under development will be adding multi-threading to the single-threaded Modifiers, and adding multi-lingual support to the dialogs.
Both of these features are estimated to take up to 2000 hours of work.
The multi-lingual support will be added to the dialogs for French, German, and Spanish, which might require some dialog control resizing.  This language support will be slowly added to each dialog, so for the next year or so the software will be in flux for which dialogs have been converted and which are still pending, so the software will be a mix of the current culture and the default English.

TGA file format is also being added to the Bitplane Creator and Splatmap Creator file save functions.

-eof-