A call of a FIFO in an envelope started as a filter, in the syntax of Liquid. In one day it became the syntax of a shell, with builtins that have the names and the options of the commands of a shell, and limits for their work.
A filter is a FIFO
In Unix, a FIFO is a named pipe between processes. A variable such as {{email | totp}} already has a pipe in it, but totp was code in the parser of the variables. Now it is a FIFO: a process that takes a Request and gives an answer.
A call
{{email | totp}} gives the capture to the FIFO totp as the body of a Request, and the body of its answer is the value of the variable. It is $(...) of a shell. {{a | f | g}} is a pipe: the answer of f goes to g.
Each FIFO with an answer egress can be a filter, also your FIFO. For example, a FIFO countries with one ingress envelope for each code and an answer with the name:
X-HTTP-Method-Override: ^POST$
^FR$To: -
FranceThen {{country | countries}} in a different envelope gives France.
How do you make totp with envelopes?
You cannot do all of it. A one-time code needs a secret key that nobody reads, a hash, and the memory of a used code. An envelope has none of them, and that is on purpose: mkfifo.com keeps no Request. Unix has the same split: a shell script reads its input, then runs a compiled program for the hard part.
So /totp is a FIFO of the site with two envelopes. The ingress envelope is the input: a value in the body, and maybe a code to check. The egress envelope is To: program:totp: it gives the Request to a small program of mkfifo.com, and the answer of the program is the answer of the FIFO. /markdown is the same. Open them and push "Try". Only a FIFO of the site can have a program.
The check is one step
In a pattern, Code: ^{{email | totp}}$ cannot only ask for the code and compare it. The code must also be used, so that it agrees one time, and the key of the code changes at that use. A call to ask and a second call to use could let two Requests use one code. So a call from a pattern also sends the value under test, in the field X-Test. totp checks the code and uses it in one step, and answers it. A wrong code gets the status 422, and the line does not agree.
The key of a code has the FIFO of the envelope that calls totp. mkfifo.com gives it to the program, not in a field of the Request: the name of a FIFO is the key to change it. A code that you get from /totp itself is valid only for totp.
A call is an edge
A call is an edge of the graph of FIFOs, as an egress envelope to a different FIFO is. A save that makes a loop of calls gets 422. The TLA+ model of the loops had one target for each save. A save with calls can add edges to more FIFOs at one time, so the model has that now, and TLC finds no loop.
A sealed pattern could also have a call in it, and the check of the loops does not open a seal. A sealed value could not have a variable, but the save of an ingress envelope did not examine it. Now it does.
No report from totp
A Request to totp has a code in X-Test. The editor of a FIFO of the site did not show its Requests, but mkfifo.com still sent the report to it. Now a FIFO of the site gets no report.
The syntax of a shell
A FIFO is a process: a Request goes in, and an answer comes out. So a call of a FIFO should look like a call in a shell. The envelopes said {{email | totp}}, the syntax of Liquid. Now they say $(totp $email), or $email | totp.
Four forms and a pipe
$nameputs a capture in the text.${size}pxneeds the braces.$(fifo word ...)calls a FIFO, and puts its answer in the text.$a | f | ggives$atof, and the answer offtog. It is$(g $(f $a)).\$is a$, and\|is a|.
mkfifo.com does not run a shell. It reads only these forms: no ${name:-default}, and no $((1 + 2)).
I left the pipe out, and that was wrong
My first version had no | at all. A call in a word already did the job, $(g $(f $a)), and I wanted one way to write it. But the pipe is the first thing that a person who knows a shell looks for. It also reads from left to right, in the direction of the data. So the pipe is back, after a variable or a call.
Inside the parentheses it stays an error. $(f $a | g) could mean two things, and the error tells you to write $(f $a) | g.
The $ and the | of a pattern
An ingress envelope has regular expressions, and they use $ for the end of a line and | for "or". So a $ is a variable only before a lowercase letter, _, { or (: $5 stays a price, and ^$(totp $email)$ keeps its last $. A pipe needs a space at each side and the name of a FIFO after it, so ^($a|$b)$ and $a | Paid stay patterns. No pattern of the templates changed.
Words are fields
In a shell, the words of a command are its arguments. Here, the words of a call are the fields of a form: 1, 2, ... in their sequence, and --name=word is the field name. A pipe gives its value as the field 1.
$(markdown $note)
$(greet $name --lang=fr)The Request of a call is a form. So the FIFO examines the words as it examines a form of HTML, and a FIFO of a person needs nothing new to get a call:
Content-Type: form
1: \SBugs on the way
- The ingress envelope of
totpwas\Son the body: a value that is not empty. But the body of a call is a form now, and the form of a space,1=+, is not empty. Sototpgave a code for a space. Now it examines the field:1: \S. - The check of a URL put a sample in place of each variable, with a regular expression of the old syntax. So
https://example.com/$(totp $x)was not a valid URL. The check uses the parser for its sample now, so the two cannot disagree. - A cookie name can have
$(RFC 9110), so$name=1was a cookie name with a variable in it. A cookie name cannot have$now. - A call with no
)took the rest of the envelope with it. A variable and a call stay on one line now, and an error stops at the line break. - A body of Markdown uses
\|for a pipe in a cell of a table. If the template made it a|first, the pipe would cut the cell in two. So a body of Markdown keeps\|, and the converter of Markdown makes it a|.
Builtins: the commands of a shell in an envelope
An envelope could call a FIFO with $(fifo word ...), and the FIFOs of the site /totp and /markdown gave their Request to a program of mkfifo.com with To: program:NAME. It worked, but it was not a shell. totp was a FIFO with a program behind it, and each word of a call was a field 1, 2, ... of a form. Now an envelope has builtins, as a shell has commands, and a pipe gives data to a builtin or to a FIFO.
Two kinds of commands
$(name word ...) runs a builtin. The words are its arguments, and its output goes in the text:
To: https://api.example/hooks
X-Request-Id: $(uuidgen)
X-Date: $(date -R)
X-Signature: $($body | hmac <seal:KEY>)
$bodyAfter a pipe comes a builtin or a FIFO. A builtin gets the value as its input. A FIFO gets the value as the body of its Request, and its answer comes back: $($name | lower) with your FIFO lower. A FIFO takes no words: its input is all its data.
Only a builtin can start $( ). $(lower $name) is an error, and the error says: "To send to a FIFO, write $value | lower." Thus the first word in $( ) has one meaning, and each place where data goes to a FIFO has a |.
The same names, the same options
The builtins are printf, tr, cut, head, tail, wc, base64, sha256sum, md5sum, hmac, random, uuidgen, date, cat and tee, and also markdown and totp. Each one has the name and the options of the command of a shell. If you know tr A-Z a-z, you know the builtin. An option that a builtin does not have is an error, not a different behavior: head -q gives "head: the option -q is not supported."
tee sends its input to FIFOs, as tee of a shell writes to files: a FIFO of mkfifo.com is a named pipe. A FIFO after a pipe, and a FIFO of tee, is an edge of the graph, so the save refuses a loop. A builtin is not an edge.
The names of the builtins are names of the site. No FIFO can have one, so the name after a pipe has one meaning.
totp is a command now
The sign-in did not change: Code: ^$(totp $email)$. In a pattern, a builtin with no input gets the value that the line examines. Thus totp gets the code of the field Code, checks it, and uses it in one step. The key of a code has the home of the envelope, as before.
/totp and /markdown went away. The start of mkfifo.com now removes a FIFO of the site when its seed goes away: it deletes the envelopes and the notes, and the node forgets the seed. /fonts reads its files with cat:
To: -
Content-Type: font/woff2
Cache-Control: public, max-age=31536000, immutable
$(cat fonts/$file)A key that only hmac opens
A signature needs a key that nobody reads. A word <seal:KEY> in $( ) is sealed at the save, as a sealed field is. Only hmac opens it, in the FIFO that sealed it. The other builtins see <sealed:TOKEN>.
Limits first
A builtin runs in the Durable Object with all the other FIFOs. Thus a small envelope must not make a large work.
- No builtin reaches the network or writes a file.
catreads only the public files of the site, andteesends only to FIFOs. Onlytotpkeeps data: hashes, in memory, for 15 minutes. - The input, the words and the output of a builtin are each at most 1 MiB.
- A Request runs at most 1024 builtins, and their inputs, words and outputs are at most 16 MiB in all, also in the FIFOs after its pipes.
My first versions of printf and cat made all their output, and then compared its size with 1 MiB. printf with a width of 65536 and 5000 words makes more than 300 MiB before that check. Now they stop at 1 MiB. tr, cut, head, tail, wc and base64 made a list of each character or each line. In a test, such a list for 1 MiB of text used 56 MiB of memory. Now they read the input one time, with no list.
A property test runs each builtin with random words and random bytes. It found three exceptions, and I fixed them: a tr range across the surrogates of UTF-16, an option with a byte that is not UTF-8, and markdown of bytes that are not UTF-8.
The same review found a limit that each template needed, also before the builtins: $body 2000 times in one envelope makes a message of almost 2 GiB. Now the values in one message are at most 2 MiB, with each use of a value. In a pattern, the values of one line are at most 64 KiB: PCRE cannot compile a larger pattern, and the escape and the compile take time before that error.
The size of a word
The words of a builtin are at most 1 MiB, and printf refused a larger word. But the check came too late: it came when the builtin started, after the words had their values.
How a word grows
A word joins its parts with no space: $first.$last, --subject="Order $id". A variable can come many times in one word. An envelope has at most 16 KiB, so one word can have about 8,000 copies of $body, and a body has at most 1 MiB. A word of 8 GiB is much more memory than the VM has.
I found it when I read the code of the words for the next branch, and a test showed it: 200 copies of a value of 1 MiB made a word of 200 MiB, and only then printf said that the words were too large.
The fix
Now the evaluation of the words counts their bytes, and stops at 1 MiB, before the builtin runs. The error is the same as before: printf: the words are larger than 1 MiB. The limit is for all the words of one command, and a command in a word stops at its own words.
The values in a message (2 MiB) and in a line of a pattern (64 KiB) already had a check before the values went in. The words were the one place with no check before.