I will look into it. The Inpainted 3D mesh could be a nice upgrade for viewing in 3D from the current technique of rendering on a plane with a displacement modifier.

The Inpainted 3d Mesh it produces is still quite similar to a plane with a displacement. They use a few good tricks to make it better than a simple displacement, but the PLY it produces remains a single polygonal surface. The 4 videos it produces at the end of the process really show the limits of that technique: you can't really move your camera any further than that without breaking the illusion.
But it's still amazing that this is all done from a single picture, and in the right hands it's possible to make something useful out of that PLY model with colors stored at vertex maps.

I get this error:
WARNING: Could not find the Qt platform plugin "windows" in ""
WARNING: This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

I get this error: WARNING: Could not find the Qt platform plugin "windows" in "" WARNING: This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Please check the 'Issues' on the project repository: https://github.com/thygate/stable-diffusion-webui-depthmap-script/issues

Updates

• v0.2.5 bugfix
  • error checking on model download (now with progressbar)

The newest version implementing multi-resolution merging outputs high resolution depth maps now, and thus has sharper edges. This should already improve your results ..

Cutting the depthmap at some depth level, clipping white and making the rest transparent should not be too hard ..
Is that what you want ?

Yes, a simple cutout mask. :) However, maybe @Extraltodeus would be a better target in this case, he made a similar script, I saw him doing absolute black/white masking. I use both of your scripts in parallel for post-processing.

Yes, a simple cutout mask. :) However, maybe @Extraltodeus would be a better target in this case, he made a similar script, I saw him doing absolute black/white masking. I use both of your scripts in parallel for post-processing.

Same thing over here.
But more often than not I bring the depthmap in photoshop and I adjust it there. That's the step I'd like to be able to do directly in the WebUI if possible.

The newest version implementing multi-resolution merging outputs high resolution depth maps now, and thus has sharper edges. This should already improve your results ..

Cutting the depthmap at some depth level, clipping white and making the rest transparent should not be too hard .. Is that what you want ?

What is the use of the transparent areas in a mask like this exactly? In my early tests a full white image with a transparency gradient gave me the same results as a second image with the same gradient, but in RGB colors, without any transparency channel. I'd have to test again, but if you have a good usage case, I will test with that instead and learn from your process along the way !

As for myself, what I'd like is a 16 bit color curves control with a GUI and a live histogram, just like in photoshop. Since I suppose this is quite complex, a good first implementation would be a Color Levels effect where you can select the darkest and brightest depth values delimiting the object you want to mask. Then the middle values in between those two parameters would be made white, and both the closest and furthest "borders" of our new mask would be feathered by a user selected value, and that feathering between full-white and full-black would be interpolated in 16 bit using the original depthmap data - this makes a feathering effect that is also 3d in itself, and it can allow for much better compositing in some circumstances.

If you haven't already, go try the 2.0 Depth model - there is a brand new PR for it over here:
#5542

I can't stop playing with it, it's so much fun.

@AugmentedRealityCat That's actually a nice idea. I have to look into normal alignment, which shouldn't be a big problem.

Edit: here is the updated verion: https://www.youtube.com/watch?v=AeDngG9kQNI importdepthmap_1.0.3.zip

Thank you so much for this extremely quick response to my feature request ! It's just unbelievable.
That's exactly what I meant, and then some, there is dynamic control of depth and everything - this is better than what I do in C4d, and I thought I was clever !
I hope you'll get more users, but if not know that you have at least one who is very satisfied.

I have to jump in and say how happy I am that I've come across this thread. The depthmap extension is of course amazing, but I also love the blender addon so so much. and the fact that it was written by chatGPT - amazing. Thanks everyone!!

A tiny automation Addon for Blender. Most of the script was written by chatGPT. I only made the polish and the inputs. https://youtu.be/vfu5yzs_2EU importdepthmap_1.0.2.zip

I get a Error Message using the addon
Report: Error
Python: Traceback (most recent call last): File "CAUserslMetulski AppData1Roaming181ender Foundationeender13.01scripts\addonsNimportdepthmapLinit_.py", line 98, in execute sna_buildsetup_63DA10 File "CAUsergMetulski AppDataNRoamingeender Foundationeender13.0 \scripts1addonsNimportdepthmap1_init_.py", line 126, in sna_buildsetup_63DA1 bpy.ops.object.shade_smooth('INVOKE_DEFAULT, use_auto_smooth=True, auto_smooth_angle=math.radians(60.0)) File "E:1131ende63.01scriptslmoduleslbpylops.py", line 130, in call ret = _op_call(seltidname_pya C_dict, kw, C_exec, C_undo) TypeError: Converting py args to operator properties: : keyword "use_auto_smooth" unrecognized
location: :-1

Can you share your settings like Blender Version; Python Version and Image Size. Thanks.

I will also ask ChatGPT :-)

A tiny automation Addon for Blender. Most of the script was written by chatGPT. I only made the polish and the inputs. https://youtu.be/vfu5yzs_2EU importdepthmap_1.0.2.zip

I get a Error Message using the addon Report: Error Python: Traceback (most recent call last): File "CAUserslMetulski AppData1Roaming181ender Foundationeender13.01scripts\addonsNimportdepthmapLinit_.py", line 98, in execute sna_buildsetup_63DA10 File "CAUsergMetulski AppDataNRoamingeender Foundationeender13.0 \scripts1addonsNimportdepthmap1_init_.py", line 126, in sna_buildsetup_63DA1 bpy.ops.object.shade_smooth('INVOKE_DEFAULT, use_auto_smooth=True, auto_smooth_angle=math.radians(60.0)) File "E:1131ende63.01scriptslmoduleslbpylops.py", line 130, in call ret = _op_call(seltidname_pya C_dict, kw, C_exec, C_undo) TypeError: Converting py args to operator properties: : keyword "use_auto_smooth" unrecognized location: :-1

Can you share your settings like Blender Version; Python Version and Image Size. Thanks.

I will also ask ChatGPT :-)

You have to use Blender 3.3 or up

I will investigate deeper, have not had the time to thoroughly check it out yet. Good find.

I've just tested the Depthmap loader add-on for Blender written by @Ladypoly and it works very well.

I've added the link to the README too.

I'll try to investigate if there are special uses for your extension when combined with the new PR that allows the use of the 2.0 Depth model for IMG2IMG. I've been playing with it non-stop since I installed it a couple of hours ago - time flies when you have fun !
#5542

Updates

• v0.2.8 bugfix
  • boost now also able to run on cpu
  • res101 able to compute on cpu

midas 3.1 will be implemented asap to bring a permanent fix to the changes that came with it.. meanwhile the install script clones an older commit of the midas repo..

@thygate I'm just wondering how and with what you made this video. :) I would love to learn.

With depth maps you can create a 3d like experience from an image. This was shown off here, and now I believe @thygate has implemented it in the extension for us.

@mykeehu It was generated using boostinmonoculardepth, res101, and 3dphotoinpainting. All available in the extension as a one click operation.
@ClashSAN Exactly

I have updated the readme#Usage section with the required info.

I've been trying to document as much as possible in the readme, but it has gotten a bit dense indeed. The Changelog could be condensed and moved to a separate file. And I would like to add some additional example images later ..
Suggestions are welcome !

For the description, do you think "Depth Maps, Stereo Image, 3D Mesh and Video generator extension" is adequate ?

yes, I mean on this page: https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Extensions#depth-maps (It's all right though, you don't have to think about it)

ah ok, thanks, I will have to get back to you on that.

That's freaking amazing.
Do you think it is possible to use UVs and textures instead of vertex colors?

And could you separate the generate mesh and video option?

Do you think it is possible to use UVs and textures instead of vertex colors?

That would require UV unwrapping the new mesh and generating a new rgb image that includes the newly inpainted content .. I don't have any plans to do so at the moment, you could do it in blender manually for now ... If there's lasting interest I might consider figuring something out .. (perhaps chatgpt can help make some automation script ;)

And could you separate the generate mesh and video option?

The mesh and video generation are already separate, videos can be rendered from a mesh on the depth tab on the right below the depth output panel.

The mesh generation option does currently also generate 4 demo videos, I will disable these or make it an option to generate them with the mesh .. I'm not sure yet which .. Is that what you meant ?

Suggestions and feedback welcome.

That's freaking amazing. Do you think it is possible to use UVs and textures instead of vertex colors?

I do that in C4d by creating a new set of UV coordinates and baking the vertex colors into a new texture map based on those UV coordinates.

We could look into doing some sort of uv unwrap and project to the texture.

There are still a lot of 3dphotoinpainting parameters that might be tweakable, i'm still testing ..

Also forgot a setting that should be enabled and repeats some inpainting .. so new version later ..

I think some wiki pages might be in order soon, as the interface gets more and more dense with options, new users might be scared off..

The video generation is separate (screenshot: #4252 (reply in thread))

The extension generates 4 demo videos when generating a mesh ... I will likely disable these later.

There are some simple scripts to define the camera moves for those 4 clips if I remember correctly. I think the amplitude of the movement is defined in one place, and the trajectory (circle, forward, etc.) is defined in another.

@AugmentedRealityCat yes, that's what I meant, to make my own pan-and-zoom version in addition to the demo

Yes, the 4 demo video parameters are :

crop border for all: [0.03, 0.03, 0.05, 0.03],

names: ['dolly-zoom-in', 'zoom-in', 'circle', 'swing']
translate x: [0.00, 0.00, -0.015, -0.015],
translate y: [0.00, 0.00, -0.015, -0.00],
translate z: [-0.05, -0.05, -0.05, -0.05],
trajectory: ['double-straight-line', 'double-straight-line', 'circle', 'circle'],

@thygate thank you, now I understand better how to set the parameters!

This is no longer an extension. This just an amazing toolkit which found it's way into a stable diffusion web ui.

Can you explain what "pre-depth background removal" does?
I used it with and without and couldn't see any difference. I thought it might skip the depth map process but it was still generated before the background was removed.

Can you explain what "pre-depth background removal" does?

It removes the background on the input image for depth generation, so the whole depth range can be used for the foreground subject.

You should look at the asymetric tiling extension - it's old but it's gold !
That's how I manage tiling on the X-axis for the panoramas I'm generating.

https://github.com/tjm35/asymmetric-tiling-sd-webui

This is how i'm doing it currently:
https://imgur.com/a/PMDJF4S

So

• generate depthmap1 of original image
• swap two (horizontal) halves of image to create a second image and create depthmap2
• swap two (horizontal) halves in depthmap2
• blend depthmap1 and depthmap2

any more info on the blending ? Just 50/50 ?

Mostly I'm just using a gradient as mask with 100% opacity in the middle where the seam is and then 0% until the border.
https://imgur.com/a/TXMh2uU

I have set up a discussion with some proof of concept code to follow up on your technique :
thygate/stable-diffusion-webui-depthmap-script#183