Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Float16 type support #26

Closed
BenjaminPoulain opened this issue Sep 7, 2019 · 5 comments
Closed

Float16 type support #26

BenjaminPoulain opened this issue Sep 7, 2019 · 5 comments

Comments

@BenjaminPoulain
Copy link

The current spec defines the tensor types "float32", "int32" and "8 bits quantized".

We should consider "float16" for a few reasons:

  • Certain hardware are significantly slower handling float32 compared to float16. This includes but is not limited to GPUs.
  • Storing the intermediate values in float16 can be beneficial to reduce memory footprint and memory bandwidth. This is true even if all the operations are done on float32 in the FPUs.
  • Supporting constant tensors in float16 would allow clients to half the size of the model's parameters. This can reduce models size in memory and reduce network bandwidth when loading the page.

Implementations without native float16 support should be allowed to use float32 internally. This would help maximizing performance on float32 hardware while keeping things simple for the authors.
@huningxin
Copy link
Contributor

Thanks for proposing this @BenjaminPoulain . Adding "float16" support makes sense to me.

@anssiko
Copy link
Member

anssiko commented Jan 9, 2020

Resolution from https://www.w3.org/2020/01/09-webmachinelearning-minutes.html#x04

Add "float16" to OperandType enum in WebNN API and define a way for the API to respond when float16 is not natively supported

@huningxin feel free to craft a PR for the group to review. Thanks!

@wchao1115
Copy link
Collaborator

wchao1115 commented Jan 10, 2020 •
edited

Agreed with @BenjaminPoulain on the stated benefits of supporting float16 in the API. Models with float16 weights and features have become increasingly popular in real-world use cases especially on the GPU e.g. Nvidia TensorCore in the Pascal or newer generation is specifically float16-based. Float16 is also a primary native data type on most DSP-based AI hardware chips e.g. Intel Movidius.

That said, in absence of native support in the underlying hardware, internal simulation such as up-casting the data type on the fly could lead to severe performance penalty and sometimes unexpected computation errors. The API should be designed such that it could fail the operation with unsupported error to allow the client an opportunity to gracefully fallback either by switching to a different target computing device, or to a model variation that simply processes float32 weights and feature sets.

@huningxin huningxin mentioned this issue Jan 10, 2020
Merged
@huningxin
Copy link
Contributor

@huningxin feel free to craft a PR for the group to review. Thanks!

Put together #35. @BenjaminPoulain @wchao1115 @anssiko , please take a look. Thanks.

@huningxin huningxin mentioned this issue Jan 15, 2020
Closed
@BenjaminPoulain
Copy link
Author

Closing. @huningxin resolved the issue in #35.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@anssiko @BenjaminPoulain @huningxin @wchao1115