Add XRInteractionSpace

[Manishearth]

Fixes #9

cc @klausw @bialpio @cabanier

r? @toji

[Manishearth]

Annoyingly there's no easy way to do this without doing some math on the projection matrix. Perhaps select events for "screen" target ray modes should also come with (x, y) positions?

[toji]

I wonder: Are standard pointer events still being processed during an AR session? If so developers could choose to listen for those instead in a "screen-space" interaction space. Not sure if that's the pattern we want to promote, but there's no way I can see to completely normalize these input modes anyway so if the logic needs to branch it may be better to have it switch to a more robust and proven input mechanism than for us to try and offer a partial substitute.

[Manishearth]

I feel like they won't be? cc @klausw

But if they are, then yes, that makes sense. Perhaps we should spec more about how handheld AR should work vis a vis inputs in this spec, then.

[klausw]

In the current ARCore-based implementation in Chrome, there are no DOM input events while a regular immersive-ar session is active.

Enabling DOM Overlay mode adds DOM events for the selected overlay element, including pointer events, along with the beforexrselect logic to make it easier for applications to deduplicate DOM events and XR input events.

FYI, in current Chrome Canary, the transient WebXR input source for screen touch already reports x/y axis values in its gamepad attribute. These are normalized -1..1 range values, not DOM coordinates.

Example:

> session.inputSources[0].gamepad
Gamepad
id: ""
index: -1
connected: true
timestamp: 15682728.100000007
mapping: ""
axes: (2) [-0.3270833492279053, 0.5138888359069824]
buttons: Array(3)
0: GamepadButton {pressed: false, touched: false, value: 0}
1: GamepadButton {pressed: false, touched: false, value: 0}
2: GamepadButton {pressed: false, touched: true, value: 1}

[Manishearth]

Hmm. It might be important to expose the size of the screen, then? This got discussed a bit in #9: it may be useful to expose the offsets of the screen rect from the viewer space or something.

I guess one question is: given that it's handheld AR, do we expect people to draw things in viewer space with an epsilon offset from the screen, or do we expect people to draw things directly to the output buffer? In the former case we need to know the near rect of the screen to know where the "edges" are so we can draw within them, in the latter case we need to know the near rect and framebuffer size (which you have already) so that we can map the input events to screen-space coordinates. Either way you need to know the near rect.

It can potentially be exposed as an offset of the viewer space, plus a width and height. Then you need to intersect the target ray with the resultant yz plane and see where on the screen it falls.

[Manishearth]

I think something like this is necessary, I'm not sure if this is the best option. It seemed to be the best after all the discussion on #9, though perhaps we may want a companion enum on the DOM overlay module that mentions if the overlay is screen-space or emulated world-space.

[toji]

This seems like a reasonable approach to the problem, so I support it. I'd like to hear from more WG members regarding this pattern before we commit to it, though.

[toji]

I wonder: Are standard pointer events still being processed during an AR session? If so developers could choose to listen for those instead in a "screen-space" interaction space. Not sure if that's the pattern we want to promote, but there's no way I can see to completely normalize these input modes anyway so if the logic needs to branch it may be better to have it switch to a more robust and proven input mechanism than for us to try and offer a partial substitute.

[cabanier]

Now that we have DOM Layers, is this feature needed?
Even a mobile device can have floating world-locked UI and a headset can have lazy headlocked ui.

[Manishearth]

Yes , both can have such a UI, but there's a clear preference.

The feature is still needed: DOM overlays are specifically for DOM UI only. Furthermore, the plan for headsets is to have the UA to pick some rect to place it in, which does not help if for headsets you want to design your own 3D UI.

Also, not everyone may implement DOM overlays, and this is a relatively lightweight boolean that enables frameworks and apps to know how best to do their own UIs.

A possible outcome of this is that in screen-space situations people will just use DOM overlay, whereas in world-space situations they may build their own.

[cabanier]

What I think you're trying to achieve is that the author can distinguish between immersive and handheld AR.
So instead of exposing it through some UI hint, just tell the author what type of device it is.

[klausw]

I think this looks good overall.

[klausw]

In the current ARCore-based implementation in Chrome, there are no DOM input events while a regular immersive-ar session is active.

Enabling DOM Overlay mode adds DOM events for the selected overlay element, including pointer events, along with the beforexrselect logic to make it easier for applications to deduplicate DOM events and XR input events.

FYI, in current Chrome Canary, the transient WebXR input source for screen touch already reports x/y axis values in its gamepad attribute. These are normalized -1..1 range values, not DOM coordinates.

Example:

> session.inputSources[0].gamepad
Gamepad
id: ""
index: -1
connected: true
timestamp: 15682728.100000007
mapping: ""
axes: (2) [-0.3270833492279053, 0.5138888359069824]
buttons: Array(3)
0: GamepadButton {pressed: false, touched: false, value: 0}
1: GamepadButton {pressed: false, touched: false, value: 0}
2: GamepadButton {pressed: false, touched: true, value: 1}

[klausw]

I think there's some ambiguity about what it means to draw directly on the screen. For example, a stereoscopic display may have separate left/right eye views, so applications would still need to respect the viewport split when drawing to the screen, but for projection purposes it would act as screen-space.

Maybe explicitly highlight that applications still need to respect the views array and viewports for rendering?

[Manishearth]

Will do!

[Manishearth]

Though, for a stereoscopic handheld display I suspect you'd actually want to draw directly to the canvas anyway? You don't want to have multiple versions of the same buttons.

(This is precisely the kind of distinction this API lets us make, that an API based on exposing form factor directly does not)

[Manishearth]

What I think you're trying to achieve is that the author can distinguish between immersive and handheld AR.

So instead of exposing it through some UI hint, just tell the author what type of device it is.

Yes. This has already been discussed a couple times (conclusions from last meeting it got discussed)

The UI hint is intended to be a safer abstraction. We don't necessarily want to directly expose the UI form factor (especially since new form factors may crop up), but instead we want to help answer the question of "where should I draw my UI?". In general it seems to be a pattern that if a piece of information is only going to be used for a single purpose, we should expose it tailored for that purpose instead. In practice, this enum is going to be roughly equivalent to a headworn/handheld enum.

[Manishearth]

/agenda to get wider input and maybe make a decision

[AdaRoseCannon]

Note to self fix the bot

[Manishearth]

(discussed in the call, nobody seems allergic to this. we'll make the space->mode change though)

[cabanier]

Can you reiterate what cases you see where "screen-space" is not a handheld device and "world-space" is not a headset?

[Manishearth]

@cabanier alternate devices, for example a CAVE system may have UI on the surface. Or even some form of transparent screen device that's not necessarily handheld.

[cabanier]

Does WebXR work on CAVE systems?
Do you envision a transparent screen that's like a cell phone but stuck on a stand?

[Manishearth]

Do you envision a transparent screen that's like a cell phone but stuck on a stand?

Yeah, basically. It's not necessarily "handheld".

[thetuvix]

Does WebXR work on CAVE systems?
Do you envision a transparent screen that's like a cell phone but stuck on a stand?

An interactionMode attribute is likely useful for non-handheld fixed displays outside of AR scenarios. In the last discussion, @klausw mentioned fixed head-tracked displays such as zSpace monitors. These displays are not handheld, but also may benefit from screen-space UI. It may even be that a zSpace-aware UA would want to give users the choice of whether to prefer screen-space UI for use with a mouse or world-space UI for use with motion controllers.

Such a UA could support the AR module without allowing the creation of "immersive-ar" sessions. That would allow the UA to expose the interactionMode attribute even for their "immersive-vr" sessions on such fixed displays. Even desktop UAs without no AR or zSpace support should ideally support this AR module attribute so that they can return "world-space" for desktop VR headsets.

[toji]

Merging since we believe that all outstanding feedback was addressed on the last call.

[thetuvix]

@Manishearth: It looks like the enum itself did not get renamed here.