Skip to the main content
mkfifo.com

A pipeline across FIFOs

planned discovered changed fixed

A pipeline needs a FIFO that sends a Request on, and a different FIFO that examines the reply. The fields of a form came first, then a graph of nodes and edges in the database, then Reply-To and Errors-To.

Fields of a form

Most Requests to a FIFO come from a form of HTML, or bring JSON. Their data is in the body, not in the header fields. But an ingress envelope could examine the body only with one pattern for the full body. So my templates had lines such as (?m)^email: (?<email>.+)$.

A body of fields

Now Content-Type can tell how to read the body:

Content-Type: form

name: (?<name>\S.*)
email: ^(?<email>[^\s@]+@[^\s@]+)$
website: ^$

Content-Type is still a pattern. mkfifo.com tries it on samples: application/x-www-form-urlencoded and multipart/form-data for a form, application/json for JSON. When it agrees with one, each body line is NAME: PATTERN. form agrees with both types of a form, and ^application/ agrees with a form and with JSON.

For JSON, a name is a key of the top object, or a JSON path: $.order.id: ^\d+$. A missing field is an empty value, as a missing header field, so website: ^$ is still a honeypot.

The text of a form is not the form

A form already became text in the body: one line name: value for each field. I thought of reading the fields back from that text. Then a <textarea> with the line email: x would add a field email. So the Request now keeps the fields of the form, with their names and their values, and a field line reads them, not the text.

Try sends a real form

"Try" put the fields of its letter in header fields. So a letter could agree with an envelope that a real form of HTML never agreed with. Now the letter of an envelope with a body of fields sends a form, the same Request as the form on your site.

A field called name

In a check in Chromium, ?form&name=Ann put the name of the FIFO in the field name. The editor read the query from the parameters of the page, and the path /:name has a parameter name too, which wins. Now the first render reads the query of the request, and the browser sends the query of the page when it connects.

What did not change

The FIFOs of the site, such as /auth, keep their header lines for now. The start of mkfifo.com puts a seed only in a FIFO of the site with no envelopes, and a letter of mkfifo.com still sends header fields to them.

This is the first of three steps. The next step is Reply-To: a FIFO sends a Request on, and a different FIFO examines the reply. Then a pipeline goes across FIFOs.

A graph of nodes and edges

The next step is a pipeline across FIFOs: a FIFO sends a Request on, and a different FIFO examines the reply. Then one envelope links to more than one FIFO in more than one way. The table messages had a column for each link: to, home, in_reply_to, and one more row for each mirror. A pipeline needs more links, not more columns.

Two tables

Now the database is a graph: nodes and edges. The columns have the names of the property graphs of SQL (SQL/PGQ), which PostgreSQL 19 brings: label, source, destination and properties, as JSON.

An envelope, a note and a FIFO are nodes. Each line of an envelope that names a FIFO or a different envelope is an edge. I first planned a label for each type of edge. But the node and the line already tell the purpose: a To edge of an ingress envelope is where its Requests come from, and a To edge of an egress envelope is where it sends. So each edge has the label link, and its property field is the line: To, home, Cc, Bcc, call or In-Reply-To.

A FIFO is a node now. My first design said "A FIFO is not a record in the database". With a node, a FIFO can later have a description, decorations and its own "Try" form.

The check of the loops reads no text

Before, the check of the loops parsed the text of each envelope on the path, and opened each sealed To again. Now the save opens a sealed To one time, and writes its edge. The check reads only the To and call edges.

Writes with no rollback

Each statement commits alone. In a Durable Object, a rollback does not undo. So a save adds before it removes: the FIFO nodes, the new edges, the node, then the old edges. A delete removes the nodes first, then their edges. A stop between two statements leaves only edges that no read uses, and the start of mkfifo.com removes them.

A node of a FIFO can go too early. A FIFO node goes away with its last edge. A save puts in the node of a FIFO, then adds its edge. A different save can remove that node between the two statements, because it has no edge yet. I wrote a TLA+ model of the writes with stops (specs/Edges.tla). Without one lock for all the writes, TLC finds an edge to no node. With the lock, it finds no error. Before, only the saves with an edge to a FIFO took the lock. Now each save and each delete takes it.

100 parameters. A Durable Object accepts at most 100 bound parameters in one statement. The old load of the mirrors of a FIFO put the id of each mirrored envelope in one list. Now a long list is a subquery, or more than one statement. A test counts the parameters of each statement.

A sealed To uses a name. Before, a name was free when no To had it as plain text. Now the edge of a sealed To also uses the FIFO.

The data stays

The migration moved each envelope of messages to the graph at the start, and opened each sealed To for its edge. I checked it in wrangler dev: I saved FIFOs with the old version, a mirror, a hold, a note and a sealed To, then started the new version on the same storage. The loops, the mirror and the hold worked as before.

Pipes in the graph

The first version of the graph had one label for all the edges: link. It told nothing. Web Streams has two words for the one thing that a pipeline needs to know, "does the answer come back?": pipeTo and pipeThrough. So the edges got these labels.

Edges in the direction of the data

  • pipe_to: the data goes to the destination, and stops there. A Request to a FIFO goes to its ingress envelopes. The captures go to the egress envelope. The egress envelope sends to a different FIFO.
  • pipe_through: the data goes through the destination, and its answer comes back. {{email | totp}} is a pipe_through edge to totp. In the next step, To: b with Reply-To: c is pipe_through to b, then pipe_to to c, as pipeThrough(b).pipeTo(c).
  • has: no data. A FIFO has the envelopes and the notes that you edit there.

Before, an edge went from the line to the thing that the line names. Now it goes in the direction of the data. Then a pipeline is a path, and a loop of FIFOs is a cycle of pipe edges. The check of the loops reads all the pipe edges that the data of a FIFO can follow in one recursive statement. Before, it read the edges of each FIFO on the path in two statements. The recursion keeps each node one time, so it stops also on a cycle.

JSONB

SQLite has JSONB: JSON in a binary format, in a BLOB. The JSON functions read it with no parse, and properties ->> '$.name' can have an index. The :map type of Ecto writes JSON text, so I wrote an Ecto type that writes and reads the bytes of JSONB itself. SQLite is the reference in its tests: a property test gives random JSON values to the type, and SQLite must read each one. A second property test gives the JSONB of SQLite to the type, and the type must read it.

Before I wrote it, I checked JSONB in a Durable Object with wrangler dev. jsonb(), ->>, the unique index on the name of a FIFO and WITH RECURSIVE work. sqlite_version() does not: the Durable Object does not permit that function.

A new version removed the edges of others

With the new directions, the edges of an ingress envelope also include edges of other envelopes: the In-Reply-To of its egress envelopes and of its notes. A new version that removes all the old edges of its node would remove these too, and the egress envelopes would go from the Requests. So a write changes only the edges of its own lines, and a test saves a new version of an ingress envelope and keeps the edges of its egress envelope and its note.

A new database

mkfifo.com is not live yet. So I removed the move of the old table. One migration makes the graph, and the Durable Object has a new name: it starts with an empty database, and the start puts the FIFOs of the site in it again.

Reply-To

A FIFO could match a Request and do something, but it could not examine what came back. An HTTP request gave its status as a result, and the response went nowhere. Now Reply-To tells where the reply goes.

Through, then to

To: https://api.shop.example/orders
Reply-To: orders

The data goes through the API, and its response goes to the FIFO orders, as pipeThrough(api).pipeTo(orders) in Web Streams. orders gets a Request with Status, Content-Type and the body of the response. Its ingress envelopes examine them with the fields of the body, so a FIFO can match on $.id of a JSON response and do the next step. A pipeline goes across FIFOs, and each FIFO stays one ingress and its egress envelopes.

A reply can be large, so the HTTP adapter reads a response as a stream, and stops after 1 MiB.

Reply-To: - gives the reply to the client. With a FIFO in To, its answer comes back: the same thing as a call, but for the whole message.

The first answer wins

I let the FIFO of a reply answer the first client. A client gets one answer, so the first answer wins, and a later answer egress fails, as before. The state of the answer now goes with the Request of a reply: the client waits, has its answer already, or does not exist (an email, or a Request from a different FIFO).

Programs

To: program:totp gave the answer of the program to the client, as a special case. Now a program is a destination like the others, and its answer is a reply. So the seeds of totp and markdown have Reply-To: -, and a program with no Reply-To is an error at the save.

In the graph

With Reply-To, the To of a FIFO is a pipe_through edge, because its answer comes back, and the Reply-To is a pipe_to edge. The check of the loops reads them as the other pipe edges.

Reply-To was taken

Reply-To already had a meaning: the contact seed sends an email with Reply-To: {{from}}, the address for a reply to the email. So for an email, Reply-To keeps that meaning. A copy of /contact sends its email as an HTTP request to an example URL, and there Reply-To: {{from}} is not valid. The first test of a copy failed on it. Now a copy keeps only Reply-To: - of an HTTP request.

Old seeds in a new build

After the deploy of Reply-To, the new code was on mkfifo.com, but /totp and /markdown gave no answer. In wrangler dev, with the same commit, they answered.

What was different

The two FIFOs need Reply-To: - now, and the seeds have it. The editor of /markdown on mkfifo.com showed the egress envelope with no Reply-To. The Durable Object was new, so I first thought that it was an old database. But the Message-ID of a seeded envelope is a UUID of version 7, and its first 48 bits are a time: 15:15:12, after the deploy. So the new object saved an old seed, and the release had the old seeds.

Why

FIFO.Site reads the seeds at the compile, so the release needs no file of priv/repo/. Its __mix_recompile__?/0 told Mix to compile it again only when a file of the seeds came or went. For a changed file, Mix compares times. A build that keeps old files of _build can give them newer times than the files of the commit, and then a changed seed does not compile the module again. The blog has the same code, and the new post was there: a new file comes, so the blog compiled again.

The fix

  • FIFO.Site, FIFO.Gallery and MkFIFO.Blog now keep the SHA-256 of their files, and compare it in __mix_recompile__?/0. A file that comes, goes or changes compiles the module again.
  • The node of each FIFO of the site keeps the SHA-256 of its seed. At the start, a FIFO whose seed changed loses its envelopes and gets its new seed. Before, a new seed needed a new database: five times since the start of the project. Nobody can change a FIFO of the site, so nothing of a person goes away.

Errors-To

An egress can fail: the service answers 500, the next FIFO has no envelope, or the Request goes through too many FIFOs. The client gets 502, and an open editor shows why. But a webhook has nobody at the other end, and the editor is usually closed. So nobody knew.

A second output

A process in a shell has two outputs. Reply-To was the standard output. Errors-To is the standard error, as 2>:

In-Reply-To: <ID@mkfifo.com>
To: https://api.shop.example/orders
Errors-To: shop-alerts

When this egress fails, shop-alerts gets a new Request:

Error: The destination answered 500.
Envelope: <0199a7c3-...@mkfifo.com>

shop-alerts is a usual FIFO. It can match Error: (?<error>.+) and send $error to a chat. The name comes from email: Errors-To is an old header field for the address of the bounces.

The error is one more hop of the Request, so an error of an error cannot go on forever. Errors-To is an edge of the graph, so the save refuses a loop. And the egress stays failed: the error does not change the result.

No name in the error

The Request of the error does not tell the name of the FIFO that failed. A name is the key to change a FIFO, and shop-alerts can belong to a different person. Envelope tells which egress failed, and a Message-ID alone opens nothing.

Reply-To leaked a name

That rule made me look at Reply-To again, and I found a leak. The HTTP adapter sent each header line that it did not know as a header field of the request, and Reply-To was not on its list to skip. So Reply-To: orders went to the service, and the service learned the name orders. A Request to the next FIFO had it too. Now no adapter sends Reply-To or Errors-To, and a test examines what an HTTP service and the next FIFO get.