I've been wanting the option of a VR desktop for a while now. Last I looked, it wasn't ready for prime time, and that's still my impression now. I find a small 16:9 screen pretty unacceptable for coding, and the market currently offers very limited options for anything taller in a laptop.
I'm eager to see where this goes, and may need to start shopping for a head-mounted display.
The current generation of headsets is unsuited for showing large amounts of reasonably sized text. The display resolution is not there yet. If you map your desktop into a plane in VR space at the proper distance, your UI text shows up heavily aliased and barely readable due to the texture resampling that happens in that process. You need to increase resolution by another factor of two or three to make that actually convenient.
And then there is the issue with holding 3d pointing devices for prolonged amounts of time. This is taxing upper body strength if you do. The weight of thenheadset itself is also straining the neck muscles.
All of this taken together usually means that I want out of VR after about 2 hours. I barely have sessions that are longer.
> And then there is the issue with holding 3d pointing devices for prolonged amounts of time. This is taxing upper body strength if you do. The weight of the headset itself is also straining the neck muscles.
You could use a real mouse and keyboard, if you're presented with some kind of virtual workstation. And even for navigation outside the workstation, of your broader "environment", 3D video games have been using a mouse & keyboard to navigate complex interfaces & environments for a long time, with more than a little success. Granted that means you still need a surface of some sort, but I think non-touch-typing text input inside a VR space would be kinda hellish anyway, so you're at least gonna want a real keyboard regardless.
Navigation in VR is a barely solved problem. Most games have players teleport between positions instantly because a continous motion would create a sensation of conflict between balance reported by the inner ear and the perceived motion. That means instant nausea.
Positioning objects in 3D precisely and conveniently without a 3D pointing device is hard, too. The best option for that is a mouse and crutches like handles oriented along coordinate axes. But that only works because the mouse pointer has a reference plane it moves on (the projection plane for the 3D view). In VR, this plane is tracking the head precisely. You cannot reasonably put the mouse cursor on that plane. You would need a different reference plane and then you're back to all the problems that 3D input on a 2D screen has. And a positioning objects in 3D with a keyboard is even more tedious.
They wrap around your hand, so you can type (ish) and not take them off.
I'd think it would be fantastic to physically walk around to different monitor/window/server setups (in VR), though, and that'd require dragging around a rolling standup desk, or figuring out how to type based on finger positions from the controllers (which I'd imagine would be decidedly not great).
What's the resolution of the headset you've been using?
I don't think I'd like my desktop to have a lot of 3D features or require a 3D pointing device. What I'm imagining is essentially a giant 2D workspace where I can still use a Trackpoint.
I've used or tried most of the major headsets available in the market. None of them is remotely close to a resolution where reading reasonably sized text isn't straining.
The large screen concept is a bit problematic. A large planar screen distorts the portions you're not in front of because you're only seeing them at pretty extreme angles. A cylindrical screen distorts projected content, e.g. photos or 3d graphics displayed on it. A better solution would be to treat each window as a flat rectangle that can be freely positioned in 3d. So you can arrange the most important window around you in ways that you can conveniently look at them by turning your head and you'd still look at them dead on. For implementing that, a 3D pointer is by far the best option.
> I've used or tried most of the major headsets available in the market. None of them is remotely close to a resolution where reading reasonably sized text isn't straining.
I use DJI's 1080p drone goggles. With DIY shutter glasses, because DJI didn't expose independent panel inputs. :/ It looks like a 3D TV screen, not VR, but you get readable small fonts from corner-to-corner (almost, depending on whether you also need to use reading glasses with it). But with batteries, they indeed aren't light weight.
Nreal's AR glasses could be available in a couple of months.
> problematic [...] angles [...] positioned in 3d [...] arrange [...] around you
The skeuomorphic design phase with phones was a short-lived part of onboarding an untrained user population. For non-novice professional user interfaces in XR, I suggest it's... silly. :) But oddly reoccurring, perhaps because of game tooling's availability?
What I wrote has nothing to do with skeuomorphism, only with human anatomy, perception and pretty basic geometry. I am not proposing that windows get rendered with 3d modelled grips for grabbing and moving.
When an artist draws with a display-less graphics tablet, the hand is in one place, but they watch and use an abstraction of it on a display, displaced in space, orientation, and scale. It can take some getting used to. Touchpads and cursors have similar non-trivial relationships. And similarly, there's no reason for your hands to be positioned at the end of your arms (as with "throwy hands"). Nor, with non-novice users, need any other physical correlation be maintained if it's useful to ignore it.
When you use office apps, there's little attempt to maintain physical-world properties like say object permanence. Delete a word, and it "goes someplace, because it can't just vanish"? That would be silly. There's no need to emulate physical paper, or pens, or rubber erasers, or cut-and-paste with an xacto knife, paper fragments, and gooey paste from a jar. It can sometimes be useful or artistically fun to do so, but that's a UX choice.
Or say you're looking at a whole-desktop "window", placed uncomfortably close and large, to be readable despite current HMD's limited angular resolution. When you turn your head, how much does it move? With what velocity profile? And with what alterations in appearance? Skeuomorphism suggests it appear as a real physical object, nailed in space. I found I preferred a quantized 2-3x displacement, to reduce head motion, and enabled it by transiently increased transparency (including "comfort mode" clipping from fov), with slightly delayed motion and added jiggle, so other visual cues could dominate wrt balance. And the shape distorted to maintain pixel alignment, because given my weak gpu, alignment was worth more than temporal antialiasing when reading small fonts. YMMV, as just what combinations of cues are fine vs intolerable varies so much among people.
The common thread is aphysicallity. Skeuomorphism can be useful when onboarding novices, but is later often largely abandoned. As with phones. With the current focus on immersive games and novices, that second part tends to be neglected. I joke that while I use XR hardware, I'm not doing XR, because being focused on non-game non-novice UX, and using a custom non-gaming stack, I've no allegiance to the "R", to Reality. Resemblance to reality as UX design smell.
Skeuomorphism is the wrong word for what you are criticizing, I think.
You are talking about an simulated non-physical world that makes no semse at best and I'm certain that it is just a nausea and disorientation simulator for almost every single user if I understand your thoughts correctly. The thing with VR is that it messes with expectations about the physical world that people process subconsciously. Break them ever so so slightly and people react involuntarily to it. There is
no consciously controllable component to getting uncomfortable in VR.
Just wanting to point out, "...nausea and disorientation simulator for almost every single user" is quite the exaggeration. I have demo'd the Vive at several schools, businesses, among friends/family, and a significant portion of people had no problems with flying or "sliding" control schemes.
One curious personal observation: out of the ~10 Asian coworkers who tried the Vive, all experienced some degree of motion sickness related discomfort.
What about a flat rectangle that's automatically facing the user's face as if pinned to the inside of a sphere? Whatever window you're facing should have the right orientation.
Leaning in is not ergonomic over longer periods of time and may be very undesirable in constrained spaces. I find myself actually doing that a lot when interacting with the desktop in VR and it becomes physically straining after a few minutes.
Yes. If anyone knows of any existing code to support say 3D cross/parallel-eye stereo views, with z-axis displaced words, in an emacs buffer being actively edited, I'd like to hear of it. Teaching your window manager to be 3D is fun, but the next step is to teach your apps.
You are welcome to try, but I can tell you right now that a UI that places words in a text at different depths is a terrible idea. Depths is simulated by varying parallax and vergence. And forcing a permanent vergence adjustment is a strain on the eye muscles. Plus you get to enjoy the vergence-focus mismatch from the fixed headset focal distance to the fullest. This is a headache-inducing cue.
Which is why you keep your primary text plane at focus depth, and emphasize shallow 3D displacement. You can do a lot with a mere cm of depth around your laptop screen, or a similarly scaled annulus at an HMD's 2 meter focus distance.
Hmm. For VR HMDs, with RGB subpixels (non-PenTile), you can read a pixel-aligned 5 pt character in the center, but you don't get many of them (because of lens blur, and just VR lensing), and the pixels are a bit large. I've no idea how well SteamVR/Unreal/Unity currently let you use them. The lenses of Valve's Index were praised elsewhere here.
AR HMDs trade narrower FoV for angularly smaller pixels, and you can use "all" of them. But the pixels may be poorly aligned between eyes, as with Project North Star. Maybe Hololens 2 (with different a-v issues). I can't comment on Magic Leap. I look forward to trying next month's/year's Nreal glasses.
Media-viewing and drone HMDs offer a screen at some distance. Some are stereo 3D. Some are 1080p. Few have head tracking. You can add it with Intel t265, Zed, Structure Core (no linux), or DIY. Many have a "diopter" adjustment. They've varied build quality, and supported nose sizes.
Some months back, I failed to find a 3D 1080p HMD, and so currently use mono 1080p DJI drone goggles (discontinued cheaper white model), with anaglyph (crushing the color space to reduce eye strain) or DIY shutter (flickery 30 or 20 Hz) to recover 3D. :/ Usually with reading glasses, despite my old eyes being nearsighted, which costs some edge pixels. DIY optical tracking, so no temporal antialiasing. I'd not recommend it.
My current hope is for a Q4, 1080p, 50-ish degree FoV, 3D "screen" HMD (media-viewer or AR). With either head tracking built in, or COTS. It may need a Windows box in support, especially for hand tracking. I'll likely stay with a simple custom browser-as-compositor stack, for low-cost pixel-precise control, but a mainstream VR or AR stack might work too.
When one does software development, one might use a variety of media. Technical pen on paper for design-space and architecture graphs. Whiteboard. Whiteboard with colleague. Notes. Exploratory code. And so on. So one question to ask is, here comes consumer VR and AR, and what might these new media offer to justify their place in this tool collection? That takes exploration.
Another question to ask is, given hardware with 3D tracking and display, how might you usefully tweak your existing tools to leverage that?
Rather than reinventing wheels, languages, keyboards, etc, I like the approach of augmenting what exists. And I use emacs a lot.
TensorFlow posenet-based head tracking is easy.[1] And wiggle 3D, and anaglyph shaders, aren't hard. And making DIY shutter glasses (arduino, phototransistor, and Adafruit light valves) was surprisingly simple. So my generic thinkpad screen can do jittery, washed-out, or flickery 3D. And electron/node.js/v8 can serve up low-latency video of the desktop, to be sliced and diced. So an app can provide two or three eye views, in separate windows or time multiplexed, and it can be shown as 3D. When last I tried to do that separation in emacs, I timed out, still unsure if there was a viable path. Thus my query.
Another thing one might try is to overlay graphics. An emacs could have an arbitrary backchannel chat with the compositor, and draw 3D lines and such on itself. And of course control its own window positions and orientations, fragmentation and coalesce. ;)
If you have red-cyan anaglyph 3D glasses, you might enjoy toying with https://atom.io/themes/anaglyph-syntax - but trading color space for depth is a steep price.
> What would you use this for?
So basically for exploration. What might our tooling now look like, if we'd had 3D display and tracking in widespread use for the last two decades?
I care more about vertical resolution, and vertical height, more than about aspect ratio. So with 3k and 4k getting more popular on higher end laptops, the 16:9 doesn't bother me as much. I can keep a code window on one side and documentation (or program output) on the other side of the screen.
The only other drawback of 16:9 is that in order to get an acceptable height, it makes the laptop too wide. My solution is to use clip-on reading glasses over my regular distance prescription, when using a high res laptop screen.
I can imagine that solution working, but what I'd really like to see is some serious business laptops offering 3:2 screens, or 16:12 for that matter. The screen from the large iPad Pro would be perfect on a laptop.
I'm eager to see where this goes, and may need to start shopping for a head-mounted display.