Machine learning

Running background removal in the browser without a server

Most background removers upload your photo, run a model on hardware you don't control, and meter you per image. erasebg.dev does the same job inside the browser tab, which gets rid of the upload and the cost but replaces them with a different set of problems.

The segmentation itself is handled by @imgly/background-removal, which is a good library and not the part I want to write about. What took the actual work was everything around it: persuading a static host to allow WebAssembly threads, deciding how much resolution to give up before the model sees anything, keeping a phone from running out of memory, and building an editor that can put back whatever the model gets wrong.

The shape of the thing

There's no build step and no framework. The page is static HTML, the model runs through a library imported as an ES module, and the weights are served from the same origin as everything else:

import {
  removeBackground,
  preload,
} from "https://cdn.jsdelivr.net/npm/@imgly/background-removal@1.7.0/+esm";

const ASSETS_PATH = new URL("imgly-assets/", window.location.href).href;

That second line matters more than it looks. By default the library fetches its model files from a CDN, and pointing publicPath at a local directory instead means the weights come from my own origin. A tool whose entire premise is that nothing leaves your device shouldn't be quietly reaching out to a third party for a 180 MB download, and self hosting also means the tool keeps working when that CDN doesn't.

Getting WebAssembly threads on a static host

Inference on the CPU is much faster with multiple threads, and threads in WebAssembly need SharedArrayBuffer. Since Spectre, browsers only hand that over to a page that is cross origin isolated, which normally means serving two response headers:

Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Embedder-Policy: require-corp

If you control the server, you set them and move on. On a static host you often can't, which is the reason there's a coi-serviceworker.js in the page. It registers a service worker that intercepts requests and adds those headers on the way back, so the page ends up isolated without any server configuration at all.

Three models, and a default that depends on the device

The library offers three variants of the same architecture at different numeric precisions, and the download sizes line up with what you'd expect from 8 bit, 16 bit and 32 bit weights:

A 180 MB download is not something to hand a phone on mobile data, so the default is chosen from the device rather than being the same everywhere. Desktop visitors get the quality model, and anything that looks like a phone gets the small one. Both choices are overridable in settings and stored in localStorage, and changing either needs a reload, because the model is already loaded by the time you open that panel.

The detection is deliberately generous about what counts as mobile, since the cost of guessing wrong in one direction is a slightly softer mask and the cost of guessing wrong in the other is a 180 MB download over a cellular connection:

function isMobileDevice() {
  return (
    window.innerWidth < 768 ||
    /Android|webOS|iPhone|iPad|iPod|BlackBerry|IEMobile|Opera Mini/i.test(
      navigator.userAgent,
    ) ||
    ("ontouchstart" in window && navigator.maxTouchPoints > 0)
  );
}

Loading the model before anyone asks for it

Whatever the size, that download has to finish before the first image can be processed, and if it starts when somebody hits upload then their first experience of the tool is a long unexplained wait. So the page calls preload as soon as it loads and shows a small status pill in the nav while it works, which turns green when the model is ready. Most people spend long enough choosing a file that the download finishes before they need it.

try {
  document.getElementById("ai-status-text").textContent = "Warming up...";
  await preload({
    publicPath: ASSETS_PATH,
    model: state.config.model,
    device: state.config.device,
  });
  setStatus("AI Ready");
} catch (e) {
  // warmup is an optimisation, not a requirement. removeBackground will
  // fetch whatever it needs on demand, so a failure here is not fatal.
  setStatus("Ready");
}

Because those files land in the browser cache, the second visit skips the download entirely, and the tool works with the network disconnected. That falls out of the architecture rather than being a feature I had to build.

The image gets smaller before the model sees it

This is the single most important thing in the pipeline, and it's the one that keeps phones alive. A recent phone camera produces images around 4000 pixels on the long edge, and decoding one of those into a canvas, running inference on it, and holding both the input and the output in memory at once is enough to have the tab killed on a mid range device.

So every image is measured first, and anything larger than the ceiling for that device is drawn into a smaller canvas and re-encoded before it goes anywhere near the model. The ceiling is 1024 pixels on mobile and 2048 on desktop, and desktop also resizes anything over 5 MB regardless of its dimensions.

async function resizeImage(file, maxDimension = 1500) {
  return new Promise((resolve) => {
    const img = new Image();
    const url = URL.createObjectURL(file);
    const cleanup = () => URL.revokeObjectURL(url);

    img.onload = () => {
      let { width, height } = img;

      // already small enough, so leave it completely alone
      if (width <= maxDimension && height <= maxDimension) {
        cleanup();
        resolve(file);
        return;
      }

      if (width > height) {
        height = Math.round(height * (maxDimension / width));
        width = maxDimension;
      } else {
        width = Math.round(width * (maxDimension / height));
        height = maxDimension;
      }

      const canvas = document.createElement("canvas");
      canvas.width = width;
      canvas.height = height;
      canvas.getContext("2d").drawImage(img, 0, 0, width, height);

      canvas.toBlob(
        (blob) => {
          cleanup();
          resolve(blob ? new File([blob], file.name) : file);
        },
        "image/jpeg",
        0.85,
      );
    };

    // a file we cannot decode is still worth handing to the model,
    // which may well manage it
    img.onerror = () => {
      cleanup();
      resolve(file);
    };
  });
}

Three details in there earn their place. The early return means a small image is passed through untouched instead of being pointlessly re-encoded, which would cost quality for nothing. cleanup runs on both the success and the failure path, because an object URL that's never revoked keeps the whole blob in memory for the lifetime of the page, and this function runs once per file in a batch. And every failure resolves with the original file rather than rejecting, so a format the canvas can't decode still gets a chance further down the pipeline.

The cost is real and worth being straight about. Downscaling caps the resolution of what you can download, so a 4000 pixel photo comes back at 2048, and re-encoding to JPEG at 0.85 gives up a little fidelity exactly where matting quality is most visible, which is the boundary between subject and background. On a desktop that tradeoff is arguable. On a phone the alternative is the tab dying, so there isn't much of an argument.

Working through a batch

Files are processed one at a time rather than in parallel, since running two inference sessions at once on the same device makes both of them slower and doubles the peak memory. A progress ring counts through the queue, each result is timed with performance.now() so the status bar can show how long it took, and the first finished image loads into the workspace immediately so there's something to look at while the rest of the queue finishes.

Failures are handled per file. One image that the model chokes on shouldn't discard the nine that worked, so the error is reported in a toast and the loop continues. The exception is a failure on the very first file with nothing queued behind it, which sends you back to the upload screen rather than into an empty editor.

Erasing with the compositor, restoring from a second canvas

No segmentation model is right every time. It'll take a bite out of a dark jacket against a dark background, or keep a strip of wall that happened to match somebody's shirt, and a tool without a way to fix that is a tool people use once. So the result goes onto a canvas with two brushes over it.

Erasing is the easy half. Setting globalCompositeOperation to destination-out turns every subsequent fill into a hole punch, so filling a circle deletes a circle of pixels instead of painting one.

Restoring is more interesting, because those pixels are gone and the canvas can't give them back. What makes it possible is a second canvas, created when the image loads and never drawn on, holding the untouched original. The restore brush clips to a circle and then draws that whole canvas through the clip, which copies exactly the pixels under the brush and nothing else.

function drawAt(x, y) {
  const r = editor.brushSize / 2;

  if (editor.brushType === "erase") {
    editor.ctx.globalCompositeOperation = "destination-out";
    editor.ctx.beginPath();
    editor.ctx.arc(x, y, r, 0, Math.PI * 2);
    editor.ctx.fill();
  } else {
    editor.ctx.globalCompositeOperation = "source-over";
    editor.ctx.save();
    editor.ctx.beginPath();
    editor.ctx.arc(x, y, r, 0, Math.PI * 2);
    editor.ctx.clip();
    // the pristine original, drawn only inside the circle
    editor.ctx.drawImage(editor.originalCanvas, 0, 0);
    editor.ctx.restore();
  }
}

The save and restore pair around the clip is not optional. A clipping region stays on the context until the state is popped, so forgetting the restore leaves every later operation confined to one circle somewhere in the middle of the image, which is a confusing thing to debug.

Making the brush behave

Stamping a circle wherever the pointer happens to be reported gives you a dotted line, because pointer events arrive at something like 60 to 120 per second and a fast drag covers a lot of pixels between two of them. The fix is to treat each pair of positions as a segment and stamp along it, closely enough that consecutive circles overlap:

function drawLine(x1, y1, x2, y2) {
  const dist = Math.hypot(x2 - x1, y2 - y1);
  const step = Math.max(1, editor.brushSize / 4); // quarter of a brush overlaps
  const steps = Math.ceil(dist / step);

  if (steps <= 1) {
    drawAt(x2, y2);
    return;
  }

  for (let i = 0; i <= steps; i++) {
    const t = i / steps;
    drawAt(x1 + (x2 - x1) * t, y1 + (y2 - y1) * t);
  }
}

Stepping by a quarter of the brush width scales the work to the brush rather than to a fixed number, so a 200 pixel brush doesn't do fifty times more work than it needs to.

The cursor is its own small problem. A brush that operates in image coordinates while you're zoomed to 300 percent covers three times as much screen as it does image, so a fixed size cursor lies about where the stroke will land. The circle you see is a positioned div sized at brushSize * scale, recomputed whenever either value changes, and it's red for erase and green for restore so there's never a question about which brush is armed.

Zoom is anchored under the pointer, which is the difference between a zoom that feels like a map and one that feels like it's fighting you. Convert the cursor position into image space before changing the scale, then move the canvas so that same image point is still under the cursor afterwards:

const imgX = (mouseX - editor.offsetX) / editor.scale;
const imgY = (mouseY - editor.offsetY) / editor.scale;

editor.scale = newScale;

editor.offsetX = mouseX - imgX * editor.scale;
editor.offsetY = mouseY - imgY * editor.scale;

Undo is expensive, so it's capped

Undo works by keeping snapshots. At the end of every stroke the canvas is read into an ImageData and pushed onto a stack, and undoing pops back to the previous entry. It's the simplest approach that's correct, and it's also the most memory hungry, because an ImageData holds four bytes per pixel with no compression whatsoever.

function saveHistory() {
  const maxHistory = isMobileDevice() ? 5 : 10;

  if (editor.history.length >= maxHistory) {
    editor.history.shift(); // drop the oldest state
  }

  editor.history.push(
    editor.ctx.getImageData(0, 0, editor.canvas.width, editor.canvas.height),
  );
}

Reading pixels back off a canvas is also slow if the browser has put the backing store on the GPU, since it has to stall the pipeline and copy everything down. Because every stroke ends in a getImageData, the context is created with a flag that asks for the software path up front:

canvas.getContext("2d", { willReadFrequently: true });

A tile based undo, storing only the region a stroke touched, would cut the memory by an order of magnitude. It's the obvious next thing here, and it's the sort of change that's easy to justify and easy to get subtly wrong, so it hasn't happened yet.

The before and after slider is two images and a clip path

The comparison view looks like it needs canvas work, and it needs none. Two images sit stacked in the same box, the result underneath and the original on top, and the original is clipped from the right by a percentage:

#compare-original {
  clip-path: inset(0 50% 0 0);
}

Dragging the handle rewrites that one percentage. Nothing is redrawn, nothing is composited in JavaScript, and clip-path is cheap enough that it tracks the pointer without any help. What actually drives it is a native range input stretched across the whole stage at zero opacity, which is worth more than it sounds: keyboard control, touch dragging and the correct pointer behaviour all arrive for free instead of being reimplemented in mouse event handlers.

The one thing to remember is that the compare view holds an image and the editor holds a canvas, so switching back to compare after editing has to refresh the result from the canvas or you're looking at a stale copy of your own work.

What it buys and what it costs

The privacy claim is the easy one to make because it isn't really a claim. An image that's never transmitted can't be retained, logged or used as training data, and there's no policy to read because there's no transfer to write a policy about.

The more interesting consequence is that the economics invert. A hosted model costs its operator money on every request, which is why every one of them counts your images and eventually asks for a card. A model running on the visitor's own hardware costs me one static file transfer, cached after the first visit, so there's no reason to cap how many images anyone processes and I don't.

What I give up is control of the hardware. Somebody on a five year old laptop has a slower experience than somebody with a discrete GPU, a phone waits several seconds where a desktop waits under one, and there's no server I can scale up to fix that for them. Every optimisation in this post exists because of that constraint, and for a tool like this I'd still make the same trade.

The tool is live at erasebg.dev, and the rest of what I've built is on my projects page. If you want the reasoning behind the other things on this blog, the peer to peer file transfer post covers a similar idea applied to networking rather than inference.