One good example there might be RED (https://tools.ietf.org/html/rfc2198).
IIUIC, it is integrated as a codec similarly to Opus, the transform being applied on the result of the RED encoder.

RED encoder is using an underlying encoder (say Opus) to compress data.
It will then generate content by taking this compressed data and by appending previously encoded data to it until a byte limit is reached. Ideally, the content will be serialised as one packet.

If the transform increases the size of the content by too much, things might actually break.

Correct, the audio RED encoder is wrapping the opus one, stores a packet (or two, three, four, ...) and then essentially concatenates them.

Since audio typically isn't split into multiple packets that could create a problem with insertable streams but I think we have a lot of margin here. The audio encoder does know the transport overhead which is calculated from RTP header size or RTP header extensions currently (see here for details)

Generic problem is transforms where the input is very different from the output.
"Support RED as a transform" may be a separate topic - this would require that the transform has available the desired packet size from the packetizer. Also gets into the whole issue of how an app that inserts a transform can also do the correct SDP actions (preferably without having to munge everything in SDP).

If we switch to using EncodedVideoChunk, which is immutable, the problem becomes a different problem.

I actually ran into this again with a audio/RED encoder. Since audio packets don't get packetized they are size-limited (1200 bytes - transport overhead). This ended up being dropped silently deep down in libwebrtc.
Should
1/ enqueue throw for some definition of "too large audioframe"
2/ the original frame specify a maximum size?

@alvestrand, @aboba, I was initially thinking we could add an API for the web page to expose the target bitrate overhead of the transform.
Rethinking about this issue though, in case transforms add a lot of overhead, I wonder now whether we need to provide any API or whether this could be an unexpected consequence of transforms that implementations need to deal with.

1/ enqueue throw for some definition of "too large audioframe"

If write is failing, it usually means the WritableStream will be errored, which we do not want.
I guess an error event in RTCRtpScriptTransformer is a possibility if we want the web app to get that info fast, or a stat otherwise.

2/ the original frame specify a maximum size?

That is one possibility, another possibility is to put it at RTCRtpScriptTransformer level:

partial interface RTCRtpScriptTransformer {
    readonly attribute unsigned maxAudioFrameSize;
}

with possibly an event in case the value changes.
One question though is how to compute this value (MTU - estimated RTP packetization overhead?)

One question though is how to compute this value (MTU - estimated RTP packetization overhead?)

The question is what the MTU is even (and we have no PMTU). For video a common assumption in libWebRTC is 1200 but as https://bugs.chromium.org/p/chromium/issues/detail?id=1470254& has shown, stereo opus at 510kbps can result in an RTP payload of 1275 bytes. Which was lower than the (arbitrary) limit of 1000. It is unclear if what works on all networks even without adding more bytes.

Right, given it might be hard for UA to give a good value and the web app could probe this info on its own, it does not seem worth exposing such a value.

The comment about audio frame size limitation came up in a different context today.
I think the proper venue for that is AVTCORE in the IETF - if we want to add a packetizer to an audio codec (and I think there are cases where we need it), I think that's the place it should be defined.

Moving the arbitrary limit from 1000 to 1200 is probably pretty safe.