User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36

Steps to reproduce:

As WebGL resources cannot be shared between several WebGL contexts / canvas, a common approach to implement a multi view 3D web application is to render content in an offscreen shared canvas and to blit the result into several onscreen canvas using drawImage.

https://github.com/greggman/virtual-webgl

As stated by the author, the drawImage call introduces a performance drop, thas OffscreenCanvas and ImageBitmapRenderingContext new APIs are expected to solve : Extract from the OffscreenCanvas.transferToImageBitmap documentation :

"This method returns an ImageBitmap object, which can be used in a variety of Web APIs and also in a second canvas without creating a transfer copy."

Actual results:

Moving from Canvas2D.drawImage to OffscreenCanvas.transferToImageBitmap / ImageBitmapRenderingContext does not improve performances wit current Firefox implementation (Nightly 106)

From the profiling it seems that a copy is performed (an an IPC call ?).

Expected results:

OffscreenCanvas / ImageBitmapRenderingContext should improve performances.

The Bugbug bot thinks this bug should belong to the 'Core::Graphics: CanvasWebGL' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

The severity field is not set for this bug.
:jgilbert, could you have a look please?

For more information, please visit auto_nag documentation.

Right now ImageBitmaps in Firefox are always basically CPU-side ImageData objects.
Ideally, we will eventually have a better GPU-side Surface API and will be able to transferToImageBitmap efficiently.

I assume https://bugzilla.mozilla.org/show_bug.cgi?id=1189634 is also tied into that limitation, and would rely on the same underlying fix? createImageBitmap with a canvas likely has many of the same use cases, but also works on Safari where OffscreenCanvas still isn't available.

(In addition to 3D multi-views, it also seems like this would be the best way to handle interlacing HTML and WebGL-rendered content when you have arbitrary stacking between the two and want to allow for transparency)

I think the severity of this bug should be reconsidered by both Mozilla and Apple. It's been years since Google chrome can blit a canvas content from a WebGL canvas content to a 2D canvas without any performance penalty. That's mandatory for sharing the same scene in several views. This topic is not progressing anymore on Mozilla and Apple side, despite the marketing annoucement of OffscreenCanvas.

Another workaround I considered was Canvas2D with drawImage. Appears to have the same issue there as well: https://bugzilla.mozilla.org/show_bug.cgi?id=1163426

Please consider re-evaluating the severity of this bug. We also have web apps which share GL resources by using a single OffscreenCanvas and calling transferToImageBitmap and transferFromImageBitmap to several bitmaprenderer canvases. On Firefox, this operation is very slow. This jsfiddle shows that transferToImageBitmap is up to 10x slower than on Chrome because of creating the copy: https://jsfiddle.net/harryterkelsen/6knzmscg/21/

This issue adversely affects all Flutter apps on the web. Our top use-cases include:

• Interlacing HTML and WebGL content, as jonathan describes in https://bugzilla.mozilla.org/show_bug.cgi?id=1788206#c4.
• Rendering into multiple canvases simultaneously (a.k.a. multi-view).

With the old-style WebGL our only option was to create a WebGL context for every overlay canvas, but that becomes expensive quickly when developers want to composite a lot of individual pieces of HTML in amongst WebGL content. Additionally, that approach was plagued by bugs caused by the complexity in cross-context GPU resource management.

This is the problem that OffscreenCanvas.transerToImageBitmap was meant to solve, by taking the ownership of the offscreen texture by reference, then calling transferFromImageBitmap to pass the ownership to the target canvas (also by reference), at minimal cost. However, the cost in Firefox' implementation makes it prohibitively expensive.

I'm improving the performance of similar scenarios where we do readback and reupload with WebGL/accelerated canvas/D2D canvas. I will look into this.

Note that we can get ImageBitmap objects through a few different sources, including created from VideoFrame and creating a VideoFrame via an ImageBitmap. So this doesn't just impact canvas sources, but also WebCodecs. We are aiming to improve performance on these codepaths in H1 2024.

Thank you! Looking forward to testing it when it's ready.

In the meantime, we'd appreciate any available workaround we could try. So far we tried the following:

• OffscreenCanvas.transerToImageBitmap + ImageBitmapRenderingContext.transferFromImageBitmap (which is what this issue is about).
• Pass OffscreenCanvas to drawImage.
• Pass OffscreenCanvas to textImage2D.
• Pass WebGL2RenderingContext to textImage2D.
• Pass WebGL2RenderingContext to drawImage.

In all cases we see Firefox performing expensive texture readbacks.

(correction, we were passing the HTMLCanvasElement backed by WebGL2RenderingContext to drawImage/texImage2D, not the context object)

Yegor, is there an example Flutter app that shows off this problem?

If I am understanding this correctly, we actually may need a bunch of related but independent changes to solve this:

1. Create/implement GPUCanvasImage, which works similarly to GPUVideoImage in that it is often backed by a platform-specific hardware texture, except the compositor process remote canvases are the producers, and PCanvasManager would facilitate readback of the pixels for the content process. We would need to teach the display pipeline to handle this as well, again, similar to GPUVideoImage.

2. Make WebGL produce a GPUCanvasImage for back buffer readbacks, which is how ImageBitmap is produced from a WebGL context today.

3. Make WebGL handle SurfaceDescriptor for GPUCanvasImage on its texture upload path.

4. Make WebGL handle ImageBitmap without readback if possible, using the same SurfaceDescriptor plumbing we have today. This will improve performance for platform-specific hardware texture backed ImageBitmap objects with WebGL, including the new GPUCanvasImage.

5. Override nsICanvasRenderingContextInternal::GetFrontBuffer for ImageBitmapRenderingContext to allow us to present using a SurfaceDescriptor for when it is backed by GPUCanvasImage (or anything else supporting SurfaceDescriptor for that matter).

Actually, I wonder if I can reuse GPUVideoImage or the remote texture plumbing here instead.

This affects my library's performance quite a bit.

If anyone needs a profile demonstrating this:
https://profiler.firefox.com/public/nd5146ndp0jw7tv00k6qwt2wzggc73cef5zgt7g/calltree/?globalTrackOrder=01&invertCallstack&thread=2&v=10

I guess my question is why can't a reference counter and reference be used to track the texture on the GPU. We should not be incurring this copy.

(In reply to Patrick Martin from comment #16)

This affects my library's performance quite a bit.

If anyone needs a profile demonstrating this:
https://profiler.firefox.com/public/nd5146ndp0jw7tv00k6qwt2wzggc73cef5zgt7g/calltree/?globalTrackOrder=01&invertCallstack&thread=2&v=10

I guess my question is why can't a reference counter and reference be used to track the texture on the GPU. We should not be incurring this copy.

Comment #14 details some of the complexity. It's a problem that is far easier to solve on a napkin than in code.

Sorry, I missed the request for a sample app. Do you need a full-blown Flutter app, or an isolated repro example? I altered Harry's JSFiddle a bit to make is very obvious when comparing Firefox against Chromium: https://jsfiddle.net/yjbanov/0vurdt42/5/. Chromium can transfer bitmaps to 10 canvases with no issues on my laptop, but Firefox visibly struggles with it.

A full Flutter app will have a bunch of unrelated stuff in it, but if you find it useful we could provide a build that demonstrates the issue.

A full Flutter app would be great. It will help us better understand how Flutter is using transferToImageBitmap.

I added a built Flutter Web app. This is the standard example app. This line shows how we use createImageBitmap to rasterize to a canvas using OffscreenCanvas: https://github.com/flutter/engine/blob/f3ecbb58a01de7d34b22e2c75514ee16b1008118/lib/web_ui/lib/src/engine/canvaskit/surface.dart#L149

(In reply to Jeff Muizelaar [:jrmuizel] from comment #19)

A full Flutter app would be great. It will help us better understand how Flutter is using transferToImageBitmap.

Hello! I am the maintainer and developer of https://github.com/GlitterWare/Passy-Browser-Extension, a password management extension built on Flutter. I have just finished adding WASM support on this browser extension's main branch and it is working correctly on my end. I tested it with Edge 123.0.2420.65 and Firefox 124.0.2. My system is Ubuntu 22.04 x86_64, using KDE Plasma 5.24.7 on X11.

I am compiling under Flutter 3.19.3 with flutter build web --web-renderer canvaskit --csp --pwa-strategy=none --dart-define=FLUTTER_WEB_CANVASKIT_URL=./canvaskit/ --wasm.

Mostly the app works fine, except for images, some of which don't work without format conversion. I am currently working around that by converting all images I'm fetching from the web into bmp format using encodeBmp(decodeImage(image)!) with https://pub.dev/packages/image (see at https://github.com/GlitterWare/Passy-Browser-Extension/blob/18c15c4d7b7b9ef868b347270d4155edaae6cd83/lib/passy_flutter/widgets/favicon_image.dart#L131).

Although probably unrelated, I was experiencing some issues with https://pub.dev/packages/flutter_svg as well but I managed to get it to render images by precompiling them using vector_graphics_compiler, as specified in their readme.

Another thing to note during testing is that the native messaging connection seems to fail in some specific circumstances. My users don't seem to be experiencing this issue, but I am only able to debug the extension if I launch Firefox via terminal. It is most probably an issue on my end.

Forgot to note that the default manifest file is specific to Chromium browsers, you can either manually delete the resulting manifest.json file and replace it with manifest_firefox.json, which is available in build folder, or otherwise use the build.sh build script, which will store the Firefox-compatible version under build/firefox

@gleam.around.the.world This issue is specifically about the browser's ImageBitmap API. Your issue looks more appropriate for Flutter's issue tracker: https://github.com/flutter/flutter/issues/new/choose

@yegor.jbanov apologies, Flutter's https://flutter.dev/wasm page currently states that Firefox is blocking WASM compatibility and links here, which, since that's not the case on my end, got me confused and brought me to this tracker.

--web-renderer canvaskit does not use ImageBitmap on Firefox, so you should be good (as in, if anything fails, it's probably not due to this issue, but due to something else).

Having said that, we mostly optimize WasmGC support for the multi-threaded renderer (--web-renderer skwasm) that uses a web worker for rasterization, and that part does use ImageBitmap. This is why the Flutter team is highly interested in seeing this issue fixed.

Sorry for dragging my feet on a demo Flutter app. I put one here:

https://image-bitmap-stress-test.web.app/

This performs very well in any Chromium browser, even on a mobile phone. However, something in Firefox falls of a cliff.

Are you sure the problem is only with this method? For example, I can't update large textures smoothly without lag in Firefox, while it works fine in all other browsers, even phones. The cause of the problem seems to be the slow "Image decoding" process, because if I use JS in the worker to decode the image (using 3rd method described here - https://webglfundamentals.org/webgl/lessons/webgl-qna-how-to-load-images-in-the-background-with-no-jank.html), the lag disappears.

(In reply to Andrew from comment #27)

Are you sure the problem is only with this method? For example, I can't update large textures smoothly without lag in Firefox, while it works fine in all other browsers, even phones. The cause of the problem seems to be the slow "Image decoding" process, because if I use JS in the worker to decode the image (using 3rd method described here - https://webglfundamentals.org/webgl/lessons/webgl-qna-how-to-load-images-in-the-background-with-no-jank.html), the lag disappears.

Andew, do have some example web content that shows the problem that you're seeing?

My code is based on samples from Webglfundamentals and you can find them on GitHub, example - https://github.com/gfxfundamentals/webgl-fundamentals/blob/master/webgl/webgl-qna-how-to-load-images-in-the-background-with-no-jank-example-3.html But use larger images to observe the problem. For example, Chrome and Safari (including older iPhones) have no problem using 5000x4000 or larger images, but there are issues using Firefox. I've spent a few days trying to figure out the cause of the problem and tried various workarounds and tricks, but I can't seem to find a way to get Firefox to work without a jank. It seems that FireFox simply has a problem with "Image decoding" process blocking the main thread, even from code coming from the worker. What's worse is that this is an old and well-known problem that doesn't seem to be being addressed.

I don't see any jank when loading https://live.staticflickr.com/65535/53733822610_d229cb9092_6k.jpg and https://live.staticflickr.com/65535/53730261387_3a31a31a2d_5k.jpg on webgl-qna-how-to-load-images-in-the-background-with-no-jank-example-3.html.

Can you try getting a profile using profiler.firefox.com with the graphics preset and sharing the result here?

And that is the whole point - why do we need a hacky solution that decodes images using some JS library, just because Firefox is not able to that in the worker without blocking everything??? Also, users are still reporting problems with this solution to us, I suspect that the JS decoding requires too much memory for some devices.

(In reply to Andrew from comment #31)

And that is the whole point - why do we need a hacky solution that decodes images using some JS library, just because Firefox is not able to that in the worker without blocking everything??? Also, users are still reporting problems with this solution to us, I suspect that the JS decoding requires too much memory for some devices.

Ah, sorry, I think there was a miscommunication. I'm looking for an example that shows the jank. You don't see jank with webgl-qna-how-to-load-images-in-the-background-with-no-jank-example-3.html correct? Can you point me at an example that shows the jank?

Please, check out this article - https://webglfundamentals.org/webgl/lessons/webgl-qna-how-to-load-images-in-the-background-with-no-jank.html First two solution have lag when using large enough images. Sadly, users report "jank" with the 3rd solution, too. But, like I said, I suspect that is due to the fact that processing large images (over 6mb, for example) consume too much memory when decoding using JS library. Works fine on smaller images. Works fine on other browsers on any image size without decoding using JS.

If you're still looking for examples affected by this, my app has an "eraser" that transfers the canvas to a bitmap, does an inverse clip on said canvas (eg: https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API/Tutorial/Compositing#inverse_clipping_path) to erase a circular section, and then draws bitmap back (and "erased" region is clipped out).

The method is repeated as the mouse moves (dragging the eraser) and performance is hindered on FF. Its the eraseBrush method => https://github.com/micahg/tbltp/blob/main/packages/mui/src/utils/contentworker.ts#L277-L298.

too bad the inefficient implementation slows down flutter, I thought now with wasmgc support in firefox i am finally able to use a high-level gc-backed language for developing wasm apps.

2025 WebCompat: Graphics::WebGL

ImageBitmap implementation always handles local data and the data is handled by cpu. Then when original data exists in GPU, transfers to ImageBitmap always cause readback.

If we want to avoid the readback, we need to add fast path like Bug 1891675, Bug 1889275 and Bug 1887818.

And ImageBitmap is used by WebCodec. If WebCodec with ImageBitmap needs good performance, it also needs fast path for WebCodec.

To add fast path for remote Canvas rendering, RemoteTextureMap also needs to be modified. Since RemoteTextureMap works as a queue. It does not have a capability to use individual remote texture out of the queue. Then it might be easier to support a fast path for ImageBitmap with video like ImageBitmap::CreateInternal() at first.