Skip to the main content
mkfifo.com

Running on BEAM.com

Updated planned discovered changed fixed

mkfifo.com is one of the first real apps on BEAM.com. These are the rough edges that I found and reported, and the numbers of a stress test: speed, memory, and the limits of one Durable Object.

Running on BEAM.com

mkfifo.com runs on BEAM.com: Erlang/OTP and Elixir in one file, and the VM in WebAssembly in a Durable Object. It is one of the first real apps on it, so I find the rough edges first, and report them.

What I found

  • exqlite. beam.com always uses its own NIF of exqlite, so the project must use the same version (0.41.0).
  • Protocols. beam.com could not consolidate a protocol of Elixir again after a change: the compile stopped with :missing_chunk. So an envelope has no String.Chars; FIFO.Envelope.text/1 writes it.
  • Tests. At first, there was no coverage tool, and the NIF of lazy_html did not load for the tests of LiveView. For a few hours I ran the tests on a standard Erlang/OTP of the same version. 0.1.0-rc.1 had both, and I went back to beam.com only.
  • Hosts behind Cloudflare. With rc.1, the VM opened its connections with connect() of the Worker, which cannot reach the addresses of Cloudflare. A webhook to such a host, and the API of Cloudflare Email Service, failed on the live site but not in wrangler dev.
  • The scheme. The Cloudflare host of rc.1 did not give the app x-forwarded-proto, so force_ssl redirected each request. The Worker set the header itself.

rc.2

BEAM.com 0.1.0-rc.2 fixed the last two. HTTP and HTTPS go through fetch() of the Worker, and the host sets x-forwarded-proto from the URL. So I removed the code that set the header. The tools of rc.2 also put an escript in PATH for rebar3, so the build script does not make one now.

The tests, the build, wrangler dev, and a webhook to a host behind Cloudflare on the live site all work with rc.2.

The snapshot of the build, a mistake

rc.2 can write a snapshot of the VM at the build (npx beam.com --snapshot). The first VM of a deploy restores it in place of a boot. mkfifo.com keeps its data in SQLite in the Durable Object, so it can use only a snapshot at the boot point: the modules are loaded, and the app does not run yet.

It made the first request after a deploy faster: 4.3 s to 3.4 s on the live site, and 2.5 s to 1.4 s in wrangler dev. I added it, and I removed it again the same day.

First, the deploy did not go out. A Worker can be at most 64 MiB (not compressed), and the upload was 67 MiB: app.com is 40 MB and the snapshot 21 MB. But 25 MB of app.com is the native program (the same file runs on Linux, macOS and Windows), and the Worker never reads it. The runtime reads only lib/, releases/ and .wasm/ of the zip at the end of the file. So the build now writes edge.com (scripts/edge-com.mjs): a zip with only these entries, with the same bytes at new offsets, 14 MB. The Worker imports it in place of app.com.

Then the site reset many times with "exceeded its memory limit" while I read the blog. An isolate has 128 MB for the VM, the modules of the Worker and the JavaScript. I measured the memory of the VM after the first requests:

  • a VM that boots: 42 MB;
  • a VM from the snapshot of the build: 72 MB.

The tool that makes the snapshot does not get the variables of the Worker, so its VM ran without -Mea min, the flag that turns off the allocators of ERTS. The restored VM kept those allocators. And the 21 MB of the snapshot stay in the isolate as a module. Together it was more than 128 MB. wrangler dev does not apply the limit, so all my local tests passed.

With the flag (--env "BEAM_ERL_FLAGS=-Mea min"), the snapshot is 15 MB, and the VM 42 MB again. But the snapshot helps only the first VM of a deploy. After a reset, the object restores the snapshot that it made itself, in 0.2 to 0.5 s. One second for each deploy is not worth 15 MB in each isolate. So there is no snapshot of the build now, and edge.com stays: the modules of the Worker are 26 MB smaller than with app.com.

Now I measure the memory of the VM after the first requests for each change of the Worker.

rc.3 to rc.5

I tested rc.3 and rc.4 on a branch, with all the checks and the stress test, and kept rc.2 on the live site: each had a problem with the bodies of requests, and BEAM.com fixed them. mkfifo.com runs on rc.5 now.

  • beam.com --target wasm32 writes only the part of a release that the WebAssembly runtime reads. It replaces my own script and its tests, and the Worker still gets edge.com.
  • The body of a request goes to the VM in parts, not at one time. When the app answers before the end of a body, for example 413 for a body that is too large, the host reads the rest and drops it. The next request works.
  • A body with Expect: 100-continue (curl sends it above 1 MiB) gets its answer in wrangler dev too.
  • The log gives the memory of the VM: beam: memory 40 MB (a new peak of this VM) after the first pages.

The checks: the 1,500 tests with 100% coverage, the TLA+ models, the 137,203 cases of the builtins against the tools of the shell, and 1,710 random bodies of up to 1 MB from 32 clients, each with the correct hash. Large bodies take more time than with rc.2, so the gate below stays for now.

Email through a port

Until now, the app sent each email the way it would call any email service: an HTTPS request with a token to sendmail.mkfifo.com, a small route of the same Worker that gives the email to Cloudflare Email Service. The email left the Durable Object and came back in, with TLS at both ends.

The next release of BEAM.com adds ports to bindings: Port.open({:spawn_executable, "/env/NAME"}, ...) in Elixir calls a JavaScript object of the Worker, with the bytes of the port as a stream. So the mailer now writes the JSON of the email to the port /env/SENDMAIL. The Durable Object gives the VM a small object with a port() function, which reads the email and sends it with the Email binding. There is no token, no URL and no TLS, and only the VM can open the port. It is not a new binding in wrangler.jsonc, so it needs no migration. The port and the HTTPS route share one function, so they give the same results and errors.

In wrangler dev, a sign-in email took 24 ms of CPU time through the port. Through the HTTPS route, it took 28 ms in the VM and 8 to 10 ms more in the route.

Where the port does not exist (on an older runtime, or in the tests on my computer), Port.open fails at once, and the mailer uses the route. I checked both in wrangler dev. When the port works on the live site, the token and the route can go.

Less CPU time for each request

I profiled one small request in the VM, with the profiler of the isolate and with tprof of Erlang. Two things stood out.

The first was the names of header fields. Each request compares about 90 of them, without regard to case, and String.downcase/1 reads the Unicode tables for each character. Most names are ASCII, so the app now compares the bytes, and uses String.downcase/1 only for a name with a byte above 127. A character such as the Kelvin sign K still matches k, as before. Property tests compare the two ways.

The second was the log. Phoenix wrote two lines for each request, and each line costs CPU time in the VM: about 7% of a small request. Now the log has one line only for a request that fails, with a status of 400 or more, such as POST /my-movies 404, with no query. The other requests are in no log at all, because the request logs of Cloudflare Workers are off too.

Together, a small answer went from 16.5 ms to 13.4 ms of CPU time in wrangler dev.

Downloads that skip the VM

A download through a FIFO used to pass each byte through the VM. This was cat of a file from S3, or any reply with Reply-To: -. The Worker read the reply, gave it to the VM in pieces, and the VM gave the pieces back to the Worker for the client. For 1 MiB, that took about 100 ms of CPU time.

Now the VM only decides. It checks the envelope and builds the request, as before. Then it answers with a header that names the request, and no body. The Worker sends the request and gives the reply to the client as it is, so the bytes stay inside the runtime and never enter WebAssembly. In wrangler dev, the same download of 1 MiB took about 30 ms of CPU time, and the median time went from 144 ms to 88 ms. This is the idea of X-Accel-Redirect in nginx, and it needs only the Fetch API.

It applies to a GET with no body, and to cat of one file with nothing piped after it. A pipe such as cat FILE | secretbox -d still goes through the VM, because the VM changes the bytes. An envelope with Errors-To also goes through the VM, because it needs the result. The Worker waits at most 10 seconds for the reply to start, and does not follow redirects. One thing changes: when such a reply is not a success, the editor shows it as an answer, because the VM no longer sees the reply.

Emails with a length

An email comes to the Worker from Cloudflare Email Routing as a stream, and the Worker gave it to the VM with no length. With no length, the bridge of BEAM.com reads a body in parts of 32 KiB. Email Routing gives the size of each email, so now the Worker passes it on as Content-Length, and the bridge reads the email in large parts. In wrangler dev, an email of 4.3 MB with a file took 291 ms of CPU time, down from 433 ms.

A stress test of BEAM.com

mkfifo.com is one of the first real apps on BEAM.com, so it is a good stress test for it. I wanted numbers before BEAM.com 0.1.0: how fast one Durable Object is, how much memory the VM takes, and where it stops.

The test

scripts/stress.mjs is a Node script with no packages. Each scenario sends requests from 1 to 64 clients at the same time, for some seconds, and gives the requests each second, the latency (p50, p90, p99, max), the status codes and the errors. The scenarios are a static file, the home page, a Request with an answer, the editor, a JSON body and an HTML body with selector lines, an email with an attachment, a save, Requests to 100 FIFOs, an HTTPS egress, and open editors on WebSockets.

For the memory, GET /_/stats gives erlang:memory/0 and the counts of the VM as JSON. It answers only when the variable MKFIFO_STATS is 1, so only in wrangler dev. The live site does not set it.

I ran it in three places with the same release: wrangler dev, the release on a computer with the native beam.com (JIT, 4 schedulers), and the live site, through a proxy about 70 ms from Cloudflare.

Numbers

With one client, in wrangler dev:

  • A static file takes 7 ms. The front of the Worker serves it, with no VM.
  • The home page takes 14 ms, a Request with an answer 11 ms, the editor 19 ms, a save 22 ms.
  • A JSON body of 256 KiB with three selector lines takes 86 ms. The same with an HTML body and a CSS selector (lazy_html, a NIF in WebAssembly) takes 25 ms.

The native VM is 2 to 7 times faster for the same request. Erlang code that uses the CPU is the slowest in WebAssembly: my JSON tokenizer takes 79 ms against 15 ms. The NIF of lazy_html is near native (18 ms against 11 ms), so most of the cost is the interpreter of Erlang, not WebAssembly. FIFO.JSON can be faster, and that is on my list.

On the live site, one object gave about 190 Requests each second with an answer, and 125 home pages each second, from 64 clients. The VM is one thread, so more clients give a longer queue, not much more work.

The memory: the idle VM takes 32 MB, with 492 processes. A FIFO process takes about 60 KB, until it stops after 5 minutes with no Request. An open editor takes about 0.7 MB. 100 of the 492 processes are the acceptors of Thousand Island, and they wait on the one socket that the bridge of BEAM.com gives.

The start

The first request after a deploy takes 4.3 s: the first VM boots, and makes its snapshot at the boot point. After Cloudflare resets the object, the VM comes back from that snapshot in 0.2 to 0.5 s. A snapshot of the build made the first request 3.4 s, but it cost too much memory (see "Running on BEAM.com").

Fixed

  • When the object could not take a request, the Worker threw, and the client got the page 1101 of Cloudflare ("Worker threw exception"). Now handler.js catches it, and answers 503 with the cause and Retry-After: 1.
  • The snapshot of the build made the Worker larger than 64 MiB, and then it made the object go over its memory. I removed it, and the Worker now gets only the release part of app.com (see "Running on BEAM.com").

Two limits, fixed

The live site has the real 128 MB of an isolate, so it found two limits that wrangler dev did not.

Open editors. About 75 open editors made the object go over its memory. I expected the LiveViews, and planned to hibernate them and keep fewer assigns. The numbers said something else. LiveView already hibernates an idle process after 15 s, and the live data of one editor is 42 KB. Most of the memory was garbage on the heaps of the processes of each WebSocket: the handler of Bandit (about 330 KB) and the socket of BEAM.com (about 280 KB). A process that waits does not collect its garbage, and the WebAssembly memory of the VM does not get smaller. One garbage collection took 50 editors from 32 MB to 4 MB.

So MkFIFO.Collector now collects, each 2 s, each process that is larger than 64 KiB and two times larger than after its last collection. 100 open editors: the VM memory went from 129 MB to 80 MB. 100 editors that open at one time still made too much garbage between two passes, so each LiveView that joins now asks for a pass 250 ms later: the peak went from 87 MB to 70 MB. I also cut the acceptors of Thousand Island from 100 to 4: the bridge has one listener, so 96 of them only waited.

Large bodies at one time. In my first test, most of eight emails of 2.7 MiB at one time failed. I was wrong about the cause: later, curl sent sixteen at one time with no error, and the errors came from my Node.js client, through the proxy of my test computer. But the risk is real. The runtime reads a body into JavaScript, makes a copy with the head, and then gives all of it to the VM, so sixteen such emails are about 90 MB of JavaScript, and the memory of the VM does not show it. So the object has a gate (gate.js): a body larger than 64 KiB waits until the large bodies in the VM are at most 8 MiB, in the order of arrival, for at most 20 s. An email waits too, because a Worker cannot ask the sender to send it again later. A body larger than 8 MiB gets 413 for now: alone, it could take too much memory. Since rc.3, BEAM.com gives a body to the VM in pieces, but the gate and its limit stay until I measure them on Cloudflare.

There is one more limit, and it is in my notes for BEAM.com 0.1.0. I will write about it here when it is fixed.