The "memory" in question can be a GPU buffer for some sources (e.g. zero copy from a camera to a GPU buffer).

The approach pioneered by WebCodec is that the frame has a destroy() method, and an API requirement that destroy() has to be called before the frame is released; if the frame is passed to a MediaStreamTrackGenerator, this will be handled by the generator.

The practical limit on outstanding frames in the Chrome implementation seems to be somewhere larger than 3, but less than 10.

I think the WebCodecs VideoFrame is quite flexible in this regard, as @alvestrand points out.
cc @chcunningham of the WebCodecs team, who might be able to provide additional input.

For the case where apps are reading fast enough and simply want to drop some of the frames, Harald and Guido gave a good summary on how to release the frame. The method is now called close() (was destroy()). w3c/webrtc-encoded-transform#99 (comment) has some additional details.

Also, for the case where apps are not reading fast enough, the MediaStreamTrackProcessorInit now includes a maxBufferSize argument that gives the app control over the frame dropping behavior. The queue lives inside the MediaStreamTrackProcessor and it's oldest item is dequeued to satisfy read requests from the ReadableStream. If the arrival of a new frame would cause the queue to exceed maxBufferSize, the oldest item in the queue is dropped prior to enqueueing the newest arrival.