Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Virtual Keyboard API - boundaries of docked overlay keyboard #507

Closed
kenchris opened this issue Apr 21, 2020 · 24 comments
Closed

Virtual Keyboard API - boundaries of docked overlay keyboard #507

kenchris opened this issue Apr 21, 2020 · 24 comments
Assignees
Labels
Resolution: ambivalent Review type: CG early review An early review of general direction from a Community Group Topic: editing Topic: Input Venue: Web Apps WG W3C Web Applications Working Group

Comments

@kenchris
Copy link

kenchris commented Apr 21, 2020

Hello TAG!

I'm requesting a TAG review of VirtualKeyboard API .

The Virtual Keyboard (VK) is the on-screen keyboard used for input in scenarios where a hardware keyboard may not be available. Unlike a hardware keyboard, a VK can adapt its shape to optimize for the expected type of input. Authors have control over the displayed shape of the VK through the inputmode attribute, but have limited control over when the VK is shown or hidden.
We propose a new VirtualKeyboard interface that has APIs to provide authors with more control over when the VK is shown or hidden and also exposes the size of the VK so the page can reflow its content if part of it is occluded by the VK.

Further details:

  • [X ] I have reviewed the TAG's API Design Principles
  • The group where the incubation/design work on this is being done (or is intended to be done in the future): Editing Task Force
  • The group where standardization of this work is intended to be done ("unknown" if not known):
    Editing Task Force
  • Existing major pieces of multi-stakeholder review or discussion of this design: inputPanelPolicy as a way to control software keyboard w3c/editing#225
  • Major unresolved issues with or opposition to this design: N/A
  • This work is being funded by: Microsoft

We'd prefer the TAG provide feedback as (please delete all but the desired option):

🐛 open issues in our GitHub repo for each point of feedback

@kenchris kenchris added Progress: untriaged Review type: CG early review An early review of general direction from a Community Group labels Apr 21, 2020
@kenchris
Copy link
Author

I am positive over the new overlaysContent + inset idea (I have suggested similar myself before).

As using an inset doesn't require JavaScript, I am wondering whether there will be another way to set this in addition to a JavaScript only API?

"overlaygeometrychange" sounds quite generic, and as there can be other overlays (other application windows, modal dialogs, file pickers etc) that might be a bit confusing. Are the values flipped (or should they be) in a right-to-left layout?

The boundingRect is currently on the event itself, but it might make sense to additionally also have it on the virtualKeyboard object like the generic sensors do. Or maybe only have it there (@anssiko)

@kenchris kenchris added this to the 2020-04-27-week milestone Apr 21, 2020
@BoCupp-Microsoft
Copy link

Are the values flipped (or should they be) in a right-to-left layout?

No they are not flipped. They are in client coordinates. I don't think they should be flipped as the location of the keyboard is not writing mode dependent.

@snianu
Copy link

snianu commented Apr 22, 2020

As using an inset doesn't require JavaScript, I am wondering whether there will be another way to set this in addition to a JavaScript only API?

Do you mean the virtualKeyboard geometry? You do have to register for VK updates so there has to be JS involved anyways, but if there is a way to do this without JS, then I would love to hear it.

"overlaygeometrychange" sounds quite generic, and as there can be other overlays

We are open to suggestions on naming this event.

The boundingRect is currently on the event itself, but it might make sense to additionally also have it on the virtualKeyboard object like the generic sensors

This sounds good to me. I'll open an issue to make this change.

@kenchris
Copy link
Author

Do you mean the virtualKeyboard geometry? You do have to register for VK updates so there has to be JS involved anyways, but if there is a way to do this without JS, then I would love to hear it.

The keyboard is docked so it should be possible to represent its position using CSS environmental variables, just like the window segments are with the CSS spanning proposal from Microsoft

@kenchris
Copy link
Author

We are open to suggestions on naming this event.

If it only affect the virtual keyboard we could just call it virtualkeyboardgeometrychange - or just geometrychange if it is an even on the virtual keyboard object

@snianu
Copy link

snianu commented Apr 23, 2020

The keyboard is docked so it should be possible to represent its position using CSS environmental variables, just like the window segments are with the CSS spanning proposal from Microsoft

Yes, this is definitely we can explore more and incorporate in VirtualKeyboard API. Thanks for clarifying.

If it only affect the virtual keyboard we could just call it virtualkeyboardgeometrychange - or just geometrychange if it is an even on the virtual keyboard object

geometrychange sounds good to me as this event is fired on the virtualkeyboard object.

@cynthia
Copy link
Member

cynthia commented May 12, 2020

Simple question - does this potentially expose extra bits of entropy for fingerprinting over what is already out there? (window resizing)

@BoCupp-Microsoft
Copy link

@cynthia there's no new information to be queried that reveals any aspect of the user's machine or its configuration. I suppose the boundingRect exposed in the geometrychanged event will provide a width for the keyboard instead of just the height (which could already be inferred from APIs like Visual Viewport). That information is only available when the user interacts with the page in a way that requires text input. I don't think there is any new privacy concern with this API.

@alice
Copy link

alice commented May 28, 2020

Possibly silly question: if the point of geometrychange is simply to allow restyling certain elements in the page to accommodate the keyboard, would it be possible to provide the necessary constraints via CSS environment variables or similar, rather than fire a whole event?

e.g.

.search-box {
   bottom: env(safe-keyboard-inset);
}

I see there is some discussion above about an inset - did something change in this area?

What other actions might an author take in response to that event?

@cynthia
Copy link
Member

cynthia commented May 28, 2020

So, @alice and I took a look at this during our VF2F and there is a huge meta-question about this in general - aside from the declarative bit noted by Alice above.

The proposal notifies web applications when something (in this case, a keyboard) obstructs the content, and communicates that through two scalar values (width/height). The meta-question is - this is simply something that defines content obstruction through a non-content entity, so why is namespaced in the virtual keyboard? There are multiple cases where such things can happen (e.g. picture-in-picture, browser chrome dialogs, etc) which could benefit from communicating this information back to the content.

Further questions:

  1. The geometry model assumes that the origin is fixed, and only has a single obstruction. How would it work for something like this? https://help.apple.com/assets/5E989062094622C539A9FD75/5E989065094622C539A9FD8A/en_US/d92415de0336bf743eb7c4b6d4540999.png
  2. Were multiple obstructions considered but insignificant or was that missed in the design?
  3. What would be a better namespace, if this is no longer scoped to virtual keyboards?

I might be digging the rabbit hole deeper than necessary, but I would like to hear your thoughts on this.

@cynthia
Copy link
Member

cynthia commented May 28, 2020

@cynthia there's no new information to be queried that reveals any aspect of the user's machine or its configuration.

@BoCupp-Microsoft thank you for the clarification.

@BoCupp-Microsoft
Copy link

Hi @alice and @cynthia,

Thanks for your questions.

@alice, we're in support of using css environment variables but haven't written a proposal for it yet. @snianu, could you make an update to the explainer? Note we think the geometrychange event is necessary even if we have them since an author could want to scroll an element into view, which can't be done with CSS alone.

@cynthia regarding your meta-question, we don't believe that authors want to respond in the same way to all content obstructions. In the [explainer] (https://github.com/MicrosoftEdge/MSEdgeExplainers/blob/master/VirtualKeyboardAPI/explainer.md) there are examples that position a searchbox above the keyboard. The searchbox should likely not be repositioned above a browser dialog or picture-in-picture UI. For that reason we want to explicitly describe changes in the visibility and position and size of the virtual keyboard.

Responses to your further questions below:

Regarding point 1, we propose that the author's ability to adjust the site in response to geometry changes for the virtual keyboard is limited to docked virtual keyboards. That prevents the site from trying to reposition its content while the user is trying to explicitly position the keyboard. In the image you referenced, I believe that is a floating keyboard. I don't think there's a docked version of it, is there?

Regarding point 2, related to the answer above, we don't have any known cases where docked keyboards result in multiple obstructions.

I do realize both of my answers to points 1 and 2 are basically dodging the question by claiming there aren't any current cases that require considering multiple obstructions. If we do decide we need to develop for that case, I think we could define the existing boundingRect property of the geometrychange event to be the rectangle that bounds all parts, and define an additional sequence of rects representing each part individually. I would expect sites developed against the current proposed API shape to behave well with a future docked, split keyboard, as there's no requirement to flow content through a split keyboard. And this future API extension would allow sites that want to be docked, split keyboard-aware to do something more complicated with the new rect sequence. So in short, we haven't designed for this case, but we see a clear path to solving that case if the need arises.

Regarding point 3, I think my answer to the meta question shows that we don't expect there to be one generalized API to handle all obstructions. If that was the approach we wanted to take then there is a Visual Viewport API which would be the best candidate.

Note the Virtual Keyboard API's relation to the the Visual Viewport API has been discussed here. I think at least @othermaciej feels differently, but my current thinking is that the API surface of the VirtualKeyboard interface is not limited to just describing the rectangle of how the feature intersects the content: there are hide and show APIs and a property that indicates whether the author has opted out of having the Visual Viewport resize when the virtual keyboard is shown. IMO its more natural to group these APIs together, under a namespace related to their common purpose, instead of putting APIs to control visibility on one interface, and events that generically describe change to the viewport geometry (that may not relate to the appearance of the virtual keyboard) on the Visual Viewport interface.

Hope that helps.

@kenchris
Copy link
Author

Extending this to other types of occlusions might make sense, but I don't think we should extend it to any occlusion.

I understand the use-cases behind virtual keyboard and picture-in-picture, but I am afraid that we will suddenly start to expose occlusions like "web bluetooth selector modal dialog" without actually having use-cases for adoption to that.

Also if we go all aboard and support occlusions by other OS windows, this might even add finger-printability, by knowing the default size of say a Skype window, that will pop on top when I to alt-tab cycling. Or knowing that there is a floating Facebook messenger head on top.

The only real use-case I can come up with for supporting generic occlusions is to modify the position of Ads to always be visible :-) which I am sure is a terrible idea that some surely will pursue.

That said, I am find with supporting keyboard and pip which are related to the web page itself, but they developer needs to know what occlusion belongs to what region to which of these.

@cynthia
Copy link
Member

cynthia commented Sep 22, 2020

Extending this to other types of occlusions might make sense, but I don't think we should extend it to any occlusion.

I don't think we implied that above, the intent was being able to find all occlusions in one place instead of a bajillion different places. The intent is to provide a shared repository of occlusions when a new spec needs to add one, not to add all existing occlusions into this.

pull bot pushed a commit to Mu-L/chromium that referenced this issue Sep 22, 2020
This change added width and height of the virtual keyboard rectangle
to the CSS environment variables reported in the geometrychange event
defined for VK.
This was a feedback from TAG review as well as from web devs who are
experimenting with VK APIs.
w3ctag/design-reviews#507

Bug:1127746

Change-Id: Ib3eaefa4f92b5e7482f30c0c6643ccd470728a7c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2422701
Reviewed-by: Anders Hartvoll Ruud <andruud@chromium.org>
Reviewed-by: Kinuko Yasuda <kinuko@chromium.org>
Commit-Queue: Anupam Snigdha <snianu@microsoft.com>
Cr-Commit-Position: refs/heads/master@{#809346}
pull bot pushed a commit to Mu-L/chromium that referenced this issue Oct 13, 2020
Removed the keyboard bounding rectangle from geometrychange event.
The keyboard rectangle is now a property of virtualKeyboard object.
It gets updated when the geometrychange event is fired. The keyboard
rectangle can also be captured from CSS env variables:
https://github.com/MicrosoftEdge/MSEdgeExplainers/blob/main/VirtualKeyboardAPI/explainer.md#virtual-keyboard-visibility-change-css-environment-variables
This was also a feedback from TAG: w3ctag/design-reviews#507 (comment)

Bug: 1127749

Change-Id: Id7f19b0e091b820fa45ba4181d482559846d40ba
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2466402
Commit-Queue: Anupam Snigdha <snianu@microsoft.com>
Reviewed-by: Kent Tamura <tkent@chromium.org>
Reviewed-by: Avi Drissman <avi@chromium.org>
Cr-Commit-Position: refs/heads/master@{#816395}
@plinss plinss removed this from the 2020-09-21-F2F-Cork milestone Oct 14, 2020
@cynthia
Copy link
Member

cynthia commented Jan 26, 2021

@snianu @BoCupp-Microsoft @gked Is this still on-going work? We haven't heard back from you and we noticed that there might have been a misunderstanding - we closed and considered the other VK API issue as good to go, but we think there still is room for improvement here.

The response here MicrosoftEdge/MSEdgeExplainers#389 starts with:

The VisualViewport only solves the problem of knowing when the visual viewport is changing. It is not a general purpose reporter of occlusions and provides no information today about occlusions

But what we see is the actual occlusion still inside the virtualKeyboard object. (which does not make it general purpose, since it's impossible to extend unless we implement a common interface that enforces boundingRect. With that it's still unclear how the user is supposed to enumerate these occlusions.) I think it's fine that there is a pointer to it from the object, but we still think that there should be a canonical place to look for when it comes to occlusions, and we don't see that addressed. @alice and I discussed this today, and was wondering if this functionality could be added to the Visual Viewport API. (even if it has shipped, that doesn't mean adding features is impossible.)

@BoCupp-Microsoft
Copy link

Yes, the work continues. We have an implementation behind a flag in Chromium browsers which matches our explainer and will turn our explainer into a spec sometime soon.

I reopened this issue (MicrosoftEdge/MSEdgeExplainers#389) for now while our discussion continues. Apologies for closing it prematurely.

But what we see is the actual occlusion still inside the virtualKeyboard object.

Yes. We describe how a docked, overlaid virtual keyboard object occludes the layout viewport via the VirtualKeyboard interface.

I made an argument in the issue linked above as to why the virtual keyboard seems like a special case when compared with other examples like picture-in-picture or windows that overlap with the browser in a desktop OS. Here's the relevant bit:

In general I don't think we want to report all the things layered over top of the visual viewport. I wouldn't want web pages updating their layout as I ALT+TAB to another application that overlaps my browser, or have a web page fight me as a drag around a floating keyboard or a picture-in-picture window.

The "docked" virtual keyboard is a special case. When it appears, users do expect to be able to see what they're typing, but authors may not want the experience we offer by default. L-shaped viewports aside, adjusting the visual viewport means we'll be creating the "slop" I describe above and jumping part of the page out of the visual viewport to ensure the user can see the blinking caret. If I'm an author that has created a web application whose UI fits in 100vw x 100vh, I might rather adjust the height of one of my subscrollers rather than "unglue" my UI so that the user can start panning it around.

So with regards to this comment:

it's still unclear how the user is supposed to enumerate these occlusions.

My answer is that I don't want authors to be able to enumerate other arbitrary occlusions. If there are some specific occlusions that make sense, then I'd like to consider those when we have the specifics. I'm not aware of any customers asking to relayout their UI while other application windows are being moved around or the user moves the floating keyboard or the user repositions picture-in-picture. If there were such requests, I would still be skeptical whether exposing that information would really enable better user experiences.

@snianu
Copy link

snianu commented Mar 4, 2021

@cynthia Could you please let us know if the above answers address your concerns or not? This is currently blocking spec review & shipping in Chromium as based on the outcome of this discussion it may lead to significant changes in the API structure/code. We had this discussion at the TPAC meeting and we came to a consensus that VisualViewport is not the right place to fire geometrychange event caused by virtual keyboard show/hide. One of the reasons is with overlaysContent flag set, the VK doesn't change the VisualViewport i.e. the keyboard is overlaid on top of the content and doesn't "resize" or change the VisualViewport in any way. These discussions were buried in the show/hide policy thread which was opened separately. The discussions about VisualViewport APIs start from here. The MoM of the TPAC breakout session discussion is here
Thank you for all the feedback so far!

@plinss plinss added this to the 2021-03-22-week milestone Mar 16, 2021
@cynthia cynthia added Progress: propose closing we think it should be closed but are waiting on some feedback or consensus and removed Progress: needs consensus Progress: pending external feedback The TAG is waiting on response to comments/questions asked by the TAG during the review labels Mar 23, 2021
@kenchris
Copy link
Author

kenchris commented Mar 23, 2021

This is my personal take as there is not full consensus in the TAG at this point.

I believe virtual keyboards are special enough to warrant a specific API instead of a generic occlusion API. If such a generic API could be created it would be perfect, but I am wondering if there are strong enough use-cases to drive the development of such an API. Example occlusions:

  • Other windows
  • Picture in picture
  • Emoji picker
  • Virtual keyboards (even split ones)

I am happy that you included CSS environmental variables as that makes it much easier to use this feature from CSS.

I am also fine with this not being integrated with the Virtual Viewport API given its state of development.

I am a little bit concerned about navigator.virtualKeyboard.overlaysContent as it might not be clear that it is a setter, but that is nitpicking. I am more concerned that I as a developer can set it to false, but if the user is using a split keyboard, it is going to overlay content anyways. Maybe it should be an enum like preferredMode: overlay | docked.

As overlay keyboards are used on existing platforms, supporting them out of the box seems quite important and that is also what I am hearing from @cynthia - This requires the supporting at least two bounding boxes. This also requires additions to the CSS support @atanassov

@cynthia
Copy link
Member

cynthia commented Mar 23, 2021

Hi, apologies for the delayed response - this fell through the cracks (never got a proper milestone update - our bad) and luckily was noticed by @plinss.

Could you please let us know if the above answers address your concerns or not?

To be honest, I'm afraid not - but given this:

This is currently blocking spec review & shipping in Chromium as based on the outcome of this discussion it may lead to significant changes in the API structure/code.

We also don't think we should block work - the core concerns have been raised long enough ago, but we don't think it was addressed in a timely manner. However, given that there is a deadline and we don't think there is much else for us to do here.

That said, we have also noticed that representatives from different implementors have raised [1] [2] similar concerns and we do not think the concerns have been properly addressed. For these reasons, we plan to close this during our plenary.

[1] w3c/editing#225 (comment)
[2] w3c/editing#225 (comment)

@torgo
Copy link
Member

torgo commented Mar 24, 2021

Hi folks – thanks again for sending this review request. We are going to close this with an 'ambivalent' status. As indicated above, some of the feedback has been actioned but we feel there are some important points that haven't been addressed. More importantly there is an issue with multi-implementer support and considering the work is going in in webapps, you will need to gain implementer support before advancing this spec to the next stage there. We hope that happens.

@torgo torgo closed this as completed Mar 24, 2021
@torgo torgo added Resolution: ambivalent and removed Progress: propose closing we think it should be closed but are waiting on some feedback or consensus labels Mar 24, 2021
@jspurlin
Copy link

jspurlin commented Apr 3, 2021

...I'm a little late to the party on this, but I want to share my thoughts...

As a web app developer, the virtual keyboard and it occluding content is unique/specific enough to warrant the functionality to be scoped to its own API, and navigator.virtualKeyboard/navigator.virtualKeyboard.overlaysContent makes sense for implementing associated scenarios in a web application.

In my mind, attempting to lump the functionality into with Visual Viewport does nothing but over complicate the the mental model for how to think about and approach scenarios related to virtual keyboard (which are very targeted and different from any other Visual Viewport interactions). Additionally, I think we also need be mindful of not trying to come up with a "kitchen sink" solution that solves for unrelated scenarios that don't exist today... doing this could lead to an over-engineered solution that ends up being too complicated for practical purposes.

As it relates to occluding content, I feel the virtualKeyboard API should encapsulate allowing web developers to have knowledge about its presence/dimensions (overlaygeometrychange and boundingRect) and be able to tell the browser to automatically occlude content with the virtual keyboard (overlaysContent).

I'm happy to discuss this in more detail.

mjfroman pushed a commit to mjfroman/moz-libwebrtc-third-party that referenced this issue Oct 14, 2022
Below were some feedbacks from TAG
(w3ctag/design-reviews#507):
1. Changed overlaygeometrychange event to geometrychange as it is fired
on the virtualKeyboard object.
2. Added boundingRect to virtualkeyboard object so web authors can query
whenever they want to know the last VK geometry values that was reported.

Bug: 856269

Change-Id: I7408133aa907fbf11d375c89417364059f2b3fc0
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2240458
Reviewed-by: Avi Drissman <avi@chromium.org>
Reviewed-by: Kent Tamura <tkent@chromium.org>
Commit-Queue: Anupam Snigdha <snianu@microsoft.com>
Cr-Original-Commit-Position: refs/heads/master@{#777563}
Cr-Mirrored-From: https://chromium.googlesource.com/chromium/src
Cr-Mirrored-Commit: e197190317b3d881bd953e151b00e066203ba331
mjfroman pushed a commit to mjfroman/moz-libwebrtc-third-party that referenced this issue Oct 14, 2022
This patch contains the CSS environment variables for the virtual
keyboard rectangle that are updated when geometry change occurs.
This was a feedback from TAG review:
(w3ctag/design-reviews#507)

Bug: 856269

Change-Id: I1cf5bd234f388fbab0083398f46f70ed7a1482a2
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2244196
Reviewed-by: Anders Hartvoll Ruud <andruud@chromium.org>
Reviewed-by: Kent Tamura <tkent@chromium.org>
Reviewed-by: Kinuko Yasuda <kinuko@chromium.org>
Commit-Queue: Anupam Snigdha <snianu@microsoft.com>
Cr-Original-Commit-Position: refs/heads/master@{#779069}
Cr-Mirrored-From: https://chromium.googlesource.com/chromium/src
Cr-Mirrored-Commit: 715a8e8d6225934050e3651fbe39ade598d29924
mjfroman pushed a commit to mjfroman/moz-libwebrtc-third-party that referenced this issue Oct 14, 2022
This change added width and height of the virtual keyboard rectangle
to the CSS environment variables reported in the geometrychange event
defined for VK.
This was a feedback from TAG review as well as from web devs who are
experimenting with VK APIs.
w3ctag/design-reviews#507

Bug:1127746

Change-Id: Ib3eaefa4f92b5e7482f30c0c6643ccd470728a7c
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2422701
Reviewed-by: Anders Hartvoll Ruud <andruud@chromium.org>
Reviewed-by: Kinuko Yasuda <kinuko@chromium.org>
Commit-Queue: Anupam Snigdha <snianu@microsoft.com>
Cr-Commit-Position: refs/heads/master@{#809346}
GitOrigin-RevId: 6320d4b57043252f1b4260a97ef493d2414492c4
mjfroman pushed a commit to mjfroman/moz-libwebrtc-third-party that referenced this issue Oct 14, 2022
Removed the keyboard bounding rectangle from geometrychange event.
The keyboard rectangle is now a property of virtualKeyboard object.
It gets updated when the geometrychange event is fired. The keyboard
rectangle can also be captured from CSS env variables:
https://github.com/MicrosoftEdge/MSEdgeExplainers/blob/main/VirtualKeyboardAPI/explainer.md#virtual-keyboard-visibility-change-css-environment-variables
This was also a feedback from TAG: w3ctag/design-reviews#507 (comment)

Bug: 1127749

Change-Id: Id7f19b0e091b820fa45ba4181d482559846d40ba
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2466402
Commit-Queue: Anupam Snigdha <snianu@microsoft.com>
Reviewed-by: Kent Tamura <tkent@chromium.org>
Reviewed-by: Avi Drissman <avi@chromium.org>
Cr-Commit-Position: refs/heads/master@{#816395}
GitOrigin-RevId: f2c5241fa9f1b8563d1c8ca08d80481618980b80
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Resolution: ambivalent Review type: CG early review An early review of general direction from a Community Group Topic: editing Topic: Input Venue: Web Apps WG W3C Web Applications Working Group
Projects
None yet
Development

No branches or pull requests

8 participants