Loading...
  OR  Zero-K Name:    Password:   

Calling all Mappers

6 posts, 364 views
Post comment
Filter:    Player:  
sort


13 months ago
Dear mappers of ZK/BAR,

I would like to train a stable diffusion model on heightmaps with the goal of being able to generate heightmaps that are either playable, or useable as a starting point on a mapping project.

In the spirit of open source, and as a courtesy to those who would like to participate, I would like to ask each and every mapper if they would like to consent for their heightmaps to be used for this project. The model will also be shared freely for all of us to benefit from, and all contributing mappers will be credited alongside it. It is my hope that we can form a repository of eligible heightmaps over time, so that later models can be trained which will produce even more refined results.

Currently, the project is just a fledgling idea without too much ambition beyond having some fun and seeing what comes out of it. Keep in mind that it will not produce 1:1 replicas of any one mapper's work and it's likely that the results will be indistinguishable from any said mapper. It is also worth mentioning that this isn't intended to texturize, normal map or apply DNTS in the process either - as of yet.

If you would like to contribute, please provide your consent here and feel free to reach out to me on discord Dregs#0945 to contribute your heightmaps.

I am currently only looking for 1:1 aspect ratio maps, as this will be the easiest way to begin training the first experimental model. I would ask that you please send your highest quality bmp/png files for this.

All questions are welcome.
+8 / -0
13 months ago
Will you use some pretrained model or make a new one from a scratch?
The SD model might have troubles keeping the planar symmetry. What I would recommend would be to train it on 2:1 rectangles. You can preprocess the dataset by finding the line that splits the map (should be easy to do, just find the lowest MSE between pixel values of those two rectangles/quadrialterals for each line going through the middle every 3 or so degrees). You will then have a mix of rectangles and quadrilaterals - you can use even those quadrilaterals if you crop them.
If you plan to implement it in Tensorflow, RandomZoom and RandomRotation layers will enhance the model performance (it is like another preprocessing step, but it a part of the model pipeline, not active during inference - an alternative would be to use tf.dataset and map some function on it, but this is easier), because it makes the dataset larger. When using the 2:1 rectangle variant with RandomRotation, the splitting line would shift, but it would still help imho.
+0 / -0
My understanding is that the intention is probably to finetune a preexisting SD checkpoint, so fairly small amount of data is needed?
+1 / -0


13 months ago
CZrankSilent_AI With these ideas you can take a crack at it too - I've clearly not been down the same rabbit holes as you. As a first, low effort step I was going to use SD's baked in training tools to make a simple checkpoint (and anything else necessary) based on 50 or so heightmaps. If the maps don't come out with symmetry, I would experiment with the Latent Mirroring extension and see what comes out with that applied. I'm not incredibly technical.
+0 / -0
13 months ago
Didn't know SD had some baked in training tools, that's nice! That might work. Can you, though, force it to output only in one black-white channel instead of RGB? But I guess, you can always transform it to black-and-white after the model outputs something.

However, the point with making the dataset larger still stands, you need less data for pretraining, but 50 heightmaps is not much. If you send me the dataset of those heightmaps I can make it artificially larger by applying those rotations and zooms, it is quite simple to do it in scipy.
I am not familiar with SD models, but I will have to do some VAE for a thesis very soon, so I may try to train it to generate heightmaps if I find time for it.
+0 / -0


13 months ago
quote:
Didn't know SD had some baked in training tools, that's nice! That might work. Can you, though, force it to output only in one black-white channel instead of RGB?


Yep, very easy to make a model now. Black and white should just be implicit as a result of the dataset, but if it isn't for any reason it's also an easy fix. You'd be surprised how much you can do with how little data atm.

Will also definitely call you up if the need arises for that task, thanks for offering.
+1 / -0