The discussions at w3c/selection-api#37 are relevant to this issue.

Historically, Gecko respects each OS manner which is behavior in native OS widget or apps created by the OS vendor. E.g., a lot of detail of editing behavior is based on Notepad, Wordpad and Word on Windows, based on native widget on macOS and Linux (GTK), but there is no concrete rules to forcibly take exactly same behavior. So, if there is reasonable reason, Gecko may change the default behavior, but anyway, Gecko will make a pref for that users can restore traditional behavior.

This is expected. The definition of a word boundary in some languages like Japanese requires heuristic. There is no way for us to define them precisely. There is also a difference between how trailing or leading whitespace is treated in each operating system, and it's important for us to respect each platform's convention.

Correct me if I've misunderstood, but I think what you're saying @masayuki-nakano and @rniwa is that we generally want to mirror a platform's convention, but that ‘platform’ in this case refers to the underlying OS and not the browser. That is to say that we expect Firefox, Chrome, and Safari to behave the same way, and furthermore we expect them to behave like the underlying OS (eg. Mac OS, Windows, or Linux).

If we agree on that, then we can also agree that that's not the situation today :) If you consider just Mac OS, you'll see that word segmentation behavior is different in Safari than it is in Chrome than it is in Firefox. This affects all move by word (Option-Left/RightArrow), delete by word (Option-Delete and Fn-Option-Delete), and double-click-to-select-word operations.

Could you provide an example that does not require heuristic to detect a word boundary?

I agree with @rniwa that we should respect platform (i.e., OS) conventions, but if someone wanted to go through the effort of defining those more exactly, including any heuristics, I would not be opposed. In fact, that seems great.

ccing @makotokato

Correct me if I've misunderstood, but I think what you're saying @masayuki-nakano and @rniwa is that we generally want to mirror a platform's convention, but that ‘platform’ in this case refers to the underlying OS and not the browser.

Yes, I intended so.

But I misunderstood this issue. I was thinking that this was a request for same behavior for moving to word boundaries across OS (Windows have a different manner about this). However, this issue is for consistency of the word boundary detection across browsers. As @rniwa -san said, some languages require heuristic to consider word boundaries. Mozilla has a plan to change the logic, though, I think that it's not scope of Editing API.

So, I don't think that this should be standardized.

Why should it not be standardized?

Why should it not be standardized?

I think that it should not be limited by the standards that which way (native API on specific OS, specific library, etc) should be used on each platform. Vendors may not like standardized way against binary size, performance, and also the behavior in some languages.

But that's not what anyone is asking for right? It's about getting agreement on the observable behavior of that code. Same way it doesn't really matter how you implement GIF, as long as you decode it in the same way as others...

I feel we could at least try implementing what UAX29 says (per the tweet thread in OP) and leave the part that still needs heuristics?

And perhaps Selection.modify could explicitly be specified to use that behavior.

For user view moving a caret must have the same behaviour in rich text field, text area and simple input element (at least if the text can also be used in theses two text fields). To achieve it in standardized way, i think the rich editor should be able to ask the web browser.

I don't think there is a good way to standardize this behavior considering the fact that each language can have different set of word breaking rules (See the Notes section in UAX#29 that talks about the complexities around these rules). There is also OS specific behavior and at least on Windows we don't want to change the existing behavior as there are screen readers that depend on these platform specific word breaking rules. Changing this would have accessibility implications.
e.g. abc| def where | represents the caret's location, when user presses ctrl+right arrow on Windows it is supposed to move the caret to |def, but on Mac when user tries to move right it always places the caret at the end of the word def| (which is also what I would expect as a mac user). Chromium had some weird behaviors in the past, but I fixed most of them at least on Windows to adhere to the platform word breaking rules.

I agree that standardizing across all browsers/OSs/languages will not work. But where do we ideally want the decision on what is the right caret movement behavior to sit? Is it in the browser or in the JavaScript app? If it is in the browser, as seems to be argued here, then it would be good if:

A) we could get browsers to try to make sure that they follow the conventions of the OS and script they are dealing with and to try to put resources aside for fixing bugs when they are encountered so that the JS editors don't end up with a lot of unfixable selection bugs against them, and

B) in those cases where JavaScript is taking on a greater part of the job of an editing app, for example using Input Events and/or EditContext, we need to make sure that it is always compatible with the native selection and that there are few or no situations in which JavaScript needs to intervene and move a selection manually because it ended up in a place where the caret cannot be displayed or similar.

I can see that we can do something for B by making sure those specifications are set up for such a situation. I don't know if A is something we can do anything about here other than to talk about it.

@Naheulf : Are you requesting an event with a target range that is executed before selection changes? Or are you saying you need an API to be able to query "If the selection was in place A and the user executed a command to go one word forward, where would the selection end up?" Or maybe something entirely different?

I thought of the second solution: "If the selection was in place A and the user executed a command to go one word forward, where would the selection end up?".
However I'm nether web developer nor professional developer and I only follow this topic as hobby. So this is only a suggestion, not a request.

Would it be helpful for web authors to have something like a beforeselectionchange event (similar to beforeinput), which would inform authors where the selection is about to move and give them a way to change the default behavior?

I could imagine a selection change event similar to InputEvent that fires before updating the selection, and would contain a string similar to InputEvent.inputType that gives authors a hint as to what kind of selection modification the user is trying to perform (e.g. "MoveWordBackwards"), as well as the default resulting selection range or caret.

In general, this loosely reminds me of Input Events Level 2, in that we're looking for a way for the browser to communicate default behavior to the page, and let the page "fix up" or influence the default behavior if needed. However, whereas input events are just about text content modification (bolding, deleting, inserting, etc.), this would be limited to modifying the selection.

That would be w3c/selection-api#56

I was tasked by the I18N WG with adding our thoughts to this issue.

Word-based cursoring/selection is complicated in that it not only involves the platform interface, but is influenced by language. Many languages can use fairly simple heuristics to accomplish this (when words are separated by spaces and/or certain punctuation), but other languages present multiple difficulties. UAX#29 enumerates a few of these and outlines a default implementation. Of specific interest are these two notes:

• It is not possible to provide a uniform set of rules that resolves all issues across languages or that handles all ambiguous situations within a given language. The goal for the specification presented in this annex is to provide a workable default; tailored implementations can be more sophisticated.
• For Thai, Lao, Khmer, Myanmar, and other scripts that do not typically use spaces between words, a good implementation should not depend on the default word boundary specification. It should use a more sophisticated mechanism, as is also required for line breaking. Ideographic scripts such as Japanese and Chinese are even more complex. Where Hangul text is written without spaces, the same applies. However, in the absence of a more sophisticated mechanism, the rules specified in this annex supply a well-defined default.

I will note that a number of non-browser implementations (such as a certain ebook reading application I am familiar with) have implemented improvements to word-based selection/navigation for many of these languages. The I18N WG wouldn't want to forcibly standardize selection behavior in a way that prevents this sort of improvement, but to various people's point, it would be sensible to describe consistent behavior or patterns of behavior regarding e.g. cursor placement.