Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JSON modules #4315

Closed
MylesBorins opened this issue Jan 22, 2019 · 65 comments · Fixed by #4407
Closed

JSON modules #4315

MylesBorins opened this issue Jan 22, 2019 · 65 comments · Fixed by #4407
Labels
addition/proposal New features or enhancements topic: script

Comments

@MylesBorins
Copy link
Member

Hey All,

I'd like to explore support for importing json, similar to how Node.js currently support require('./some.json')

Expected behavior:

imported json would export an object or array with the content from the provided json file.

Why:

Currently the was to get JSON is via fetch + import.meta.url... and requires a bit of back and forth... the eventual result is a promise to resolve to the json object. Being able to statically import and parse json would allow us to import specific symbols, export symbols, and asynchronously get resources during the fetch phase.

What are the next steps?:

I'm assuming a spec change in the HTML spec, but unsure what else we would need to do. Thoughts?

@domenic
Copy link
Member

domenic commented Jan 22, 2019

So concretely, for those not familiar with Node.js's support here, this would mean that if you import something with a JSON mime type, you would get back the result as if JSON.parse were applied to the files contents (decoded as UTF-8). If it doesn't parse as JSON, then the module ends up in the error state, similar to JavaScript parse errors.

As for next steps, the main one is implementer interest. /cc @whatwg/modules; does this seem like a reasonable addition?

@bmeck
Copy link

bmeck commented Jan 22, 2019

I think it is reasonable, but would note that this is for static data, whereas fetch() would get new data each time it is used; import() will not get new modules for each use. As long as this is well understood it seems good for a variety of usages that we see in bundler usages.

@tabatkins
Copy link
Collaborator

I'm happy with certain well-known and commonly-used content-types getting good module support, even if it's just a "trivial" wrapper around a constructor or function call. JSON, CSS, etc are all good.

@annevk
Copy link
Member

annevk commented Jan 23, 2019

This came up at the last W3C TPAC too, I raised WICG/webcomponents#770 at the time. If the complexity is indeed branching on a JSON MIME type and then using parse JSON from bytes I think Mozilla would be supportive. If we also want to advertise support to servers in some manner the discussion might become a little more involved. (It seems apart from an object or array this could also return a primitive, by the way.)

@annevk annevk added addition/proposal New features or enhancements needs implementer interest Moving the issue forward requires implementers to express interest topic: script needs concrete proposal Moving the issue forward requires someone to figure out a detailed plan labels Jan 23, 2019
@thw0rted
Copy link

I don't know if it matters to the standards process, but there is some discussion of the topic already. If you look at the answers on that SO question (and at linked/related questions), you'll see there is some confusion as well. Because many (most?) people writing in ES6 or later end up transpiling via Babel -- which means using a module loader provided by the transpiler, rather than the browser-native one -- they can write import { x } from "foo.json" today, and as far as they are concerned, it works. Presumably this is because the Babel runtime loader is actually more advanced than the browser native implementation.

@bmeck
Copy link

bmeck commented Jan 23, 2019

There are some complexities if we are wishing to exactly match tooling as it exists today.

To note, tools supporting JSON as an importable module type do allow both named and default exports. The JSON provided always assigns the result value to the default and then "picks" the relevant Object properties to expose as named exports e.g.

{"x": {}}

~=

export default {x: {}}
// export x = default.x //

This kind of aliasing where updating default.x updates x isn't available to Source Text Module Records, but that is probably fine.

Additionally, for non-Identifier properties, the only way to access values would be through the default export due to being unable to import non-Identifier bindings.

import locales from './locales.json';
locales[navigator.language];

I think the simplest solution of not supporting named exports would be fine for now and encourage a more uniform usage.

@ljharb
Copy link
Contributor

ljharb commented Jan 23, 2019

Conceptually a json file is a single thing; that babel allows named imports from it is merely a product of it conflating destructuring with named imports. I definitely agree json should be confined to default import.

@MylesBorins
Copy link
Member Author

What are the next steps for getting this off the ground? Spec Text? Implementor interest?

I'm happy to help drive this but need some mentorship on the process

@domenic
Copy link
Member

domenic commented Jan 23, 2019

Implementer interest would be ideal, otherwise spec text might be wasted effort. But, you could start spec and test work ahead of implementer interest, if you're OK taking the risk.

I'm happy to mentor on spec/test work when you're ready; let's discuss in a more high-bandwidth medium like IRC or offline.

@MylesBorins
Copy link
Member Author

MylesBorins commented Jan 23, 2019 via email

@jhnns
Copy link

jhnns commented Feb 4, 2019

Conceptually a json file is a single thing; that babel allows named imports from it is merely a product of it conflating destructuring with named imports. I definitely agree json should be confined to default import.

As background information why this complexity was introduced: aliasing the top-level JSON keys to named exports makes the JSON file tree-shakable on the first level. So if only x is used in the application, the rest can be removed. This can make a notable difference if the JSON file is big enough.

@guybedford
Copy link

Tree shaking isn't really a good argument for named exports anymore. Build tools can check statically analyzable member expressions and determine if an object binding escapes the static analysis.

@robpalme
Copy link

robpalme commented Feb 5, 2019

Let's consider multiple modules that depend on the same remote source of static JSON. With today's approach of having each user fetch & JSON.parse the result, they get their own object which is pristine.

If the module loader is now holding a shared mutable copy of the data, how can a single user/callsite confidently access the original data?

Will there be a special way to request the untainted copy? Or should we consider JSON modules being deeply frozen by default?

@thw0rted
Copy link

thw0rted commented Feb 5, 2019

If anybody can come up with a reasonable use case for mutable JSON imports, I'd like to hear it, but otherwise freezing seems like a simple solution.

@jhnns
Copy link

jhnns commented Feb 5, 2019

Tree shaking isn't really a good argument for named exports anymore. Build tools can check statically analyzable member expressions and determine if an object binding escapes the static analysis.

These checks are very difficult and I think there are a lot of situations where the object binding would escape. Using named imports is certainly a lot easier to statically analyze. But I also see that allowing named imports of first-level JSON keys kind of abuses named imports for the sake of static analysis, so I'm not a strong advocate of this feature. Just wanted to give some background information.

Will there be a special way to request the untainted copy? Or should we consider JSON modules being deeply frozen by default?

I also thought about this. I would definitely prefer freezing it but I also don't see a strong reason why the host environment should enforce this. Maybe security reasons?

@guybedford
Copy link

Freezing sounds very sensible. Note that freezing would also help static analysis because then even in the case of a reference escaping the analysis, member expressions can still be inlined.

@ljharb
Copy link
Contributor

ljharb commented Feb 5, 2019

an alternative to recursively freezing might be if the import was a thunk - a function that returns a new mutable object.

@domenic
Copy link
Member

domenic commented Feb 5, 2019

I don't think this is the place to introduce any kind of freezing or recursive freezing into the web platform. The platform is full of shared mutable objects, e.g. window, or JS modules' namespace objects. There's no reason to treat JSON modules' namespace objects specially.

As always, if you want to create a frozen copy of one of these shared mutable objects, your code needs to run first, and do the freezing itself.

@guybedford
Copy link

@domenic your argument seems to be nothing more than to state the status quo as canon. I'd have hoped for a more convincing point here.

@domenic
Copy link
Member

domenic commented Feb 5, 2019

Consistency is valuable. JSON modules should not depart from JS modules.

@guybedford
Copy link

Perhaps this pushes the argument back towards being for named exports then, since that would ensure the consistency guarantees of JS modules? Although it does mean introducing a valid identifier filter unfortunately.

@domenic
Copy link
Member

domenic commented Feb 5, 2019

No, JS modules that represent a single thing do use default exports, so the consistency is perfect there.

MylesBorins pushed a commit to MylesBorins/node that referenced this issue Mar 27, 2019
This PR updates the current `--experimental-modules` implementation
based on the work of the modules team  and reflects Phase 2 of our
new modules plan.

The largest differences from the current implementation include

* `packge.type` which can be either `module` or `commonjs`
  - `type: "commonjs"`:
    - `.js` is parsed as commonjs
    - default for entry point without an extension is commonjs
  - `type: "module"`:
    - `.js` is parsed as esm
    - does not support loading JSON or Native Module by default
    - default for entry point without an extension is esm
* `--entry-type=[mode]`
  - allows you set the type on entry point.
* A new file extension `.cjs`.
  - this is specifically to support importing commonjs in the
    `module` mode.
  - this is only in the esm loader, the commonjs loader remains
    untouched, but the extension will work in the old loader if you use
    the full file path.
* `--es-module-specifier-resolution=[type]`
  - options are `explicit` (default) and `node`
  - by default our loader will not allow for optional extensions in
    the import, the path for a module must include the extension if
    there is one
  - by default our loader will not allow for importing directories that
    have an index file
  - developers can use `--es-module-specifier-resolution=node` to
    enable the commonjs specifier resolution algorithm
  - This is not a “feature” but rather an implementation for
    experimentation. It is expected to change before the flag is
    removed
* `--experimental-json-loader`
  - the only way to import json when `"type": "module"`
  - when enable all `import 'thing.json'` will go through the
    experimental loader independent of mode
  - based on whatwg/html#4315
* You can use `package.main` to set an entry point for a module
  - the file extensions used in main will be resolved based on the
    `type` of the module

Refs: https://github.com/nodejs/modules/blob/master/doc/plan-for-new-modules-implementation.md
Refs: https://github.com/GeoffreyBooth/node-import-file-specifier-resolution-proposal
Refs: nodejs/modules#180
Refs: nodejs/ecmascript-modules#6
Refs: nodejs/ecmascript-modules#12
Refs: nodejs/ecmascript-modules#28
Refs: nodejs/modules#255
Refs: whatwg/html#4315
Refs: WICG/webcomponents#770
Co-authored-by: Myles Borins <MylesBorins@google.com>
Co-authored-by: John-David Dalton <john.david.dalton@gmail.com>
Co-authored-by: Evan Plaice <evanplaice@gmail.com>
Co-authored-by: Geoffrey Booth <webmaster@geoffreybooth.com>
Co-authored-by: Michaël Zasso <targos@protonmail.com>

PR-URL: nodejs#26745
Reviewed-By: Gus Caplan <me@gus.host>
Reviewed-By: Guy Bedford <guybedford@gmail.com>
Reviewed-By: Ben Coe <bencoe@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: Сковорода Никита Андреевич <chalkerx@gmail.com>
@Jessidhia
Copy link

FWIW, a new feature of webpack@4 was support for JSON-as-modules.

When the JSON document is an object, properties are plucked (when they are valid identifiers) and made available as independent exports, with the default export being guaranteed to be the entire unmodified JSON.parse result (which may not even be an object). If you do actually use only named imports (other than default), it is capable of deleting the provably-unused parts of the object from the bundle.

BethGriggs pushed a commit to nodejs/node that referenced this issue Apr 5, 2019
This PR updates the current `--experimental-modules` implementation
based on the work of the modules team  and reflects Phase 2 of our
new modules plan.

The largest differences from the current implementation include

* `packge.type` which can be either `module` or `commonjs`
  - `type: "commonjs"`:
    - `.js` is parsed as commonjs
    - default for entry point without an extension is commonjs
  - `type: "module"`:
    - `.js` is parsed as esm
    - does not support loading JSON or Native Module by default
    - default for entry point without an extension is esm
* `--entry-type=[mode]`
  - allows you set the type on entry point.
* A new file extension `.cjs`.
  - this is specifically to support importing commonjs in the
    `module` mode.
  - this is only in the esm loader, the commonjs loader remains
    untouched, but the extension will work in the old loader if you use
    the full file path.
* `--es-module-specifier-resolution=[type]`
  - options are `explicit` (default) and `node`
  - by default our loader will not allow for optional extensions in
    the import, the path for a module must include the extension if
    there is one
  - by default our loader will not allow for importing directories that
    have an index file
  - developers can use `--es-module-specifier-resolution=node` to
    enable the commonjs specifier resolution algorithm
  - This is not a “feature” but rather an implementation for
    experimentation. It is expected to change before the flag is
    removed
* `--experimental-json-loader`
  - the only way to import json when `"type": "module"`
  - when enable all `import 'thing.json'` will go through the
    experimental loader independent of mode
  - based on whatwg/html#4315
* You can use `package.main` to set an entry point for a module
  - the file extensions used in main will be resolved based on the
    `type` of the module

Refs: https://github.com/nodejs/modules/blob/master/doc/plan-for-new-modules-implementation.md
Refs: https://github.com/GeoffreyBooth/node-import-file-specifier-resolution-proposal
Refs: nodejs/modules#180
Refs: nodejs/ecmascript-modules#6
Refs: nodejs/ecmascript-modules#12
Refs: nodejs/ecmascript-modules#28
Refs: nodejs/modules#255
Refs: whatwg/html#4315
Refs: WICG/webcomponents#770
Co-authored-by: Myles Borins <MylesBorins@google.com>
Co-authored-by: John-David Dalton <john.david.dalton@gmail.com>
Co-authored-by: Evan Plaice <evanplaice@gmail.com>
Co-authored-by: Geoffrey Booth <webmaster@geoffreybooth.com>
Co-authored-by: Michaël Zasso <targos@protonmail.com>

PR-URL: #26745
Reviewed-By: Gus Caplan <me@gus.host>
Reviewed-By: Guy Bedford <guybedford@gmail.com>
Reviewed-By: Ben Coe <bencoe@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: Сковорода Никита Андреевич <chalkerx@gmail.com>
@domenic
Copy link
Member

domenic commented May 4, 2019

I realize this feature never seems to have gotten concrete signs of implementer interest. Blink is interested in implementing; any thoughts from Gecko or WebKit? @annevk @jonco3 / @rniwa @Constellation.

Concretely, #4407 has the pull request.

@annevk annevk removed needs concrete proposal Moving the issue forward requires someone to figure out a detailed plan needs implementer interest Moving the issue forward requires implementers to express interest labels May 6, 2019
@annevk
Copy link
Member

annevk commented May 6, 2019

Mozilla is interested.

@dandclark
Copy link
Contributor

Microsoft is on board for implementing this in Blink.

domenic pushed a commit to littledan/html that referenced this issue May 7, 2019
This patch provides JSON modules as a single default export, with
parse errors checked before instantiating the module graph.

Note, editorially, it's unclear whether JSON modules should be
considered a type of "module script", with a settings object, fetch
options, base URL, etc or not. This patch considers them "module
scripts", but leaves those record fields unset (as they are unused).

This patch is based on
tc39/proposal-built-in-modules#44
which hasn't landed yet, so the references are a bit awkward, and
this patch should not land until that one does.

Closes whatwg#4315
domenic added a commit to littledan/html that referenced this issue May 7, 2019
- Tweaks to <script> element stuff
- Add example
- Add Myles to acks for raising whatwg#4315.
- Tighten definition of JS and JSON module script a bit
domenic added a commit to littledan/html that referenced this issue May 7, 2019
- Tweaks to <script> element stuff
- Add example
- Add Myles to acks for raising whatwg#4315.
- Tighten definition of JS and JSON module script a bit
domenic pushed a commit that referenced this issue May 17, 2019
This commit adds JSON modules as a single default export, with parse
errors checked before instantiating the module graph.

As infrastructure, this divides the "module script" concept into
"JavaScript module scripts" and "JSON module scripts". Most of the
spec's existing uses of "module script" become "JavaScript module
script".

JSON module scripts are fetched in the same way as JavaScript module
scripts, e.g. with the "cors" mode and using strict MIME type checking.
They use the Synthetic Module Record defined in
whatwg/webidl#722.

Closes #4315.
Closes WICG/webcomponents#770.
Ms2ger pushed a commit to littledan/html that referenced this issue May 20, 2019
This patch provides CSS modules as a single default export, of
a CSSStyleSheet object, which is not added to the document.

Edge cases which I didn't see discussed elsewhere:
- @imports are recursively fetched together with the module graph,
  blocking script execution. Network errors reached prevent the
  execution of the entire module graph.
- Any MIME type whose essence is "text/css" is accepted; this appears
  to be weaker checking than elsewhere in the specification.
- Although the Constructable Stylesheet Objects proposal is used for
  infrastructure, the resulting CSSStyleSheet object acts as if it
  were not constructed (i.e., you can't call the replace() method).

Note, the Constructable Stylesheet Objects proposal makes important steps
to specifying loading of @import, but there may still be room for more
precise plumbing with HTML. This text ensures that style sheet module scripts
have a base URL and fetch options, which might be referenced by the definition
of @import in the future.

This patch is based on
tc39/proposal-built-in-modules#44

Closes whatwg#4315
Closes WICG/webcomponents#759
@TimvdLippe
Copy link
Contributor

Should this issue be reopened since #4943 reverted #4407 which closed this issue?

@annevk
Copy link
Member

annevk commented Nov 7, 2019

Once someone takes the time to resolve WICG/webcomponents#839 I'm sure the conversation will find its way back to this repository somehow. Not sure we need a new tracking issue or reopen this one.

@trusktr
Copy link

trusktr commented Mar 21, 2020

Seems like treating JSON file content as a default export is the only way to have consistency across all JSON features.

With JSON files containing content other than objects, for example

true
null
"foo"
123
[1, "2", true]

named imports don't work (obviously).

The only way to keep usage consistent with JSON features is a default export, then in their code people can choose to detect whether the import is a boolean, an object, or something else, or what properties they want to pluck from the default import in case it is an object.

import pkg from './package.json'
import child from 'child_process'

child.exec(pkg.scripts['build:dev'], ...)

If we don't mind being less consistent across JSON features, then the next best option is for default to work as above, but with named imports also available from JSON files that have object values (the same as @Jessidhia last mentioned).

But the question is, is consistency across the JSON features important? Most JSON values are objects, and most keys in those objects are valid identifiers.

bixycler referenced this issue in bixycler/PhapCu-Dhammapada May 18, 2021
- The pseudo JSON files are renamed to ".json.js"
- There's another way to get JSON:  use JS module's `import` directive. But the module cannot be loaded from `localhost` (due to CORS). So we keep the good old non-module JS so that we can run the ebook locally.
domenic pushed a commit that referenced this issue Jul 29, 2021
Reland JSON modules (#4315), originally found in db03474 but removed in a530f6f. Importing a JSON module now requires the module type to be specified at every import site with an import assertion, addressing the security concern that led to the revert.

Additionally, some of the infrastructure now lives in the TC39 proposal at https://github.com/tc39/proposal-json-modules, and 3d45584 introduced the import assertions integration, so overall this patch is pretty small.
mfreed7 pushed a commit to mfreed7/html that referenced this issue Jun 3, 2022
Reland JSON modules (whatwg#4315), originally found in db03474 but removed in a530f6f. Importing a JSON module now requires the module type to be specified at every import site with an import assertion, addressing the security concern that led to the revert.

Additionally, some of the infrastructure now lives in the TC39 proposal at https://github.com/tc39/proposal-json-modules, and 3d45584 introduced the import assertions integration, so overall this patch is pretty small.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
addition/proposal New features or enhancements topic: script
Development

Successfully merging a pull request may close this issue.

17 participants