Spatial Photo and Video Updates for October
I am trying to get the GoPro Stereo, Dual Lumix, and my rotating platform https://realityofvision.com/category/cameras/ more bulletproof for taking out in the field. I need to upgrade the GoPros to probably Hero12s.
I’ve been playing with ComfyUI and various stable diffusion models. Flux is the first model where I got great results with simple prompts. The initial tests were so good I used it to develop some storyboards and work with actors to do volumetric capture to combine with AI generated 3D backplates. I think immersive storytelling can be quite a bit more engaging than traditional 3D content, but it is different than 3D movies of the past. My favorite YouTubers are https://www.youtube.com/@FiveBelowFiveUK https://www.youtube.com/@TheFutureThinker https://www.youtube.com/@mickmumpitz and My most used tools are DepthAnything.
I continue to work on various WebXR ideas with BabylonJS https://realityofvision.com/killer-app/ . Unfortunately Safari on Vision Pro cannot show an immersive video. DeoVR had a hack that broke in the very recent 2.2 beta. This is keeping me from sharing video content on the web, a number of content creators are making their own apps which seems a bit overkill!
By request I created a couple short YouTube videos on spatial processing for Apple Vision Pro ( https://www.youtube.com/@rovrw ) Maybe with some practice they won’t be so terrible.
Top Fusion Ideas
Fusion continues to amaze me with what a versatile tool it is. I am still primarily on my Windows 11 box, Ryzen 9 12core 64GB, RTX3090 24GB. This is primarily to handle the very large spatial photos (32k) as well as Blender for 3D. My Mac mini M2 16GB is under powered, the new M4 with 64GB of unified memory looks very compelling!
• Fusion allows me to view 360over-under in my HP Reverb headset. This is really useful, but Windows MR is dead and fairly unreliable, so I also use one of the Fusion windows set to anaglyph and good old fashion 3D glasses to verify stereo. While I shoot mostly stereo 360, I think bandwidth limitation and Apple’s decision to ephasize spatial photo (stereo 3D) and immersive video (180VR) has me standardizing on 180VR, which is cumbersome to preview on my current setup. Some how getting a stereo window, and an immersive 180VR preview with the Apple Vision Pro will be a game changer.
• I find myself needing to make quick stereo text and graphics in Fusion. KartaVR’s kvrplane does this well, but it is Mac only.
• For my rotating single camera I am shooting up to 24 shots for a 360, and then stitching in PTGui Pro 13 Beta. A workflow to get this into PTGUI and stmaps back into Fusion would be great. I’ve never gotten the older KartaVR scripts to work on my Windows setup.
• The kvrwarpstitch ultra is a great node for those shooting dual circular 180 fisheyes, so much so I find myself rendering stereo fisheyes in Blender to use that node. Much of the same workflow is applicable to my dual GoPro setup, or any ultra wide stereo for that matter, a new tool or a modification to allow full frame image of various fields of view would be super useful.
• Incorporate Topaz Photo and Video AI. I use both these tools constantly, for upsizing ComfyUI Flux results which are often small, and for upsizing video from my Gopros to be acceptable on Apple Vision Pro.
• This video https://www.youtube.com/watch?v=SJgMH-d8oG4 got me excited about the possibility of using ComfyUI, Blender and Fusion to make immersive short form videos. Seems Nuke already has some ComfyUI integration along with this workflow. Getting Switchlight features (perhaps by just using the equivalent ComfyUI modules) along with Omniscient ( https://omniscient-app.com/ ) is something I’m going to look at next month.
• Waiting for Apple’s Vimeo play. I’m anticipating we’ll see the new fisheye projection https://blog.mikeswanson.com/apples-mysterious-fisheye-projection/ to enable 180 degree immersive video. I’m really seeing the downside of side-by-side equirectangular, and I think Apple has this problem licked with their internal pipeline. Hopefully Fusion will be able to support this format, and KartaVR will be able to help creators with conversion from multi-cameras (like my camera rotator), dual cameras (like for dual GoPros) and stereo fisheyes (like my dual Lumix fisheye) into the new viewing format!
****
