Before mkfifo.com, I wrote rfetch: rclone on the APIs of the Web, where each remote is a fetch() that takes a request and gives a response. A FIFO has the same shape: a Request goes in, and its answer comes out. So a FIFO can be a remote of rclone. This post tells the first step: a FIFO lists its files to other programs, and the commands of rclone ask a FIFO for its files.
A method for the list
rfetch lists a folder with a GET, and reads the links of its HTML page. A FIFO already answers a GET with a page, so the list needs a different method. A new method, as LIST, cannot work: the edge of Cloudflare answers 501 to a method that is not standard, and the Worker never sees it. PROPFIND of WebDAV gets through, and other programs know it:
curl -X PROPFIND -H 'Depth: 1' https://mkfifo.com/blog/2026A PROPFIND is a Request with the path of the URL, the header field Depth (the number of levels, as --max-depth of rclone), and the query as the fields of a form. As with a GET, only an envelope that asks for the method agrees with it. Its answer is the JSON of rclone lsjson.
A FIFO with no envelope for PROPFIND gives an empty list. The files of a FIFO are routes of its holder, and a route can be a secret, as the name of a FIFO. So the holder decides what other people see.
The commands are clients
ls, lsl, lsd, lsf and lsjson still list the files of their own FIFO with no Request. For a different FIFO, they now send a PROPFIND, read the JSON of the answer, and write the format of rclone. The query after a ? goes with it, as the flags of rclone go in the URL of rfetch: lsjson crypt/docs?remote=vault. cat NAME/PATH reads a file with a GET, as rclone cat.
The first test found a loop. The envelope for PROPFIND of the holder runs lsjson of the same FIFO. When that lsjson sent a PROPFIND, the Request came back to the same envelope, and again, until the limit of 8 hops. Now a FIFO lists its own files itself.
JSON in JSON
An answer of application/json gives each value the escapes of a JSON string, so that a capture cannot change the structure of the body. But then the list of lsjson came out as one long string. The output of markdown already goes into HTML as it is, because a builtin wrote it. Now the output of lsjson, and of jq with no -r and no -j, goes into JSON as it is. The holder of blog can make its list public with two envelopes:
To: blog/:path*
X-HTTP-Method-Override: ^PROPFIND$
Depth: ^1$To: -
Content-Type: application/json
$(lsjson -M blog/$path)The crypt of rclone, as tools
The first backend is crypt: it encrypts the names and the data of a different remote. My first version had builtins of crypt, cryptname, cryptdata and cryptlist, with the flags of rclone. They agreed with rclone to the byte. But they hid the logic of crypt in Elixir, and they kept the keys of a Request in memory, so that a list and a read made one key. I removed them before the merge.
Now there are tools, as the commands of a shell. Each one does one thing, and a pipeline joins them, as the streams of rfetch:
scryptmakes a key from the password in its input, as RFC 7914.emeencrypts a short text with EME and AES-256: a segment of a name.secretboxencrypts data with the secretbox of NaCl. With-b, it makes a box for each block, with the next nonce: the format of the files of rclone.basencisbasencof GNU coreutils 9.4, andbase32768is the encoding of rclone for long names.obscureandrevealarerclone obscureandrclone reveal.
The keys come from the password, and cut takes the key of the data, the key of the names and the tweak from them:
key=$(printf %s "$(reveal -- "$password")" | scrypt -l 80 -S a80df43a8fbd0308a7cab83e581f86b1)A name of rclone is a pipeline, and the data of a file is one more:
printf %s "$name" | eme -K "$namekey" -T "$tweak" | basenc --base32hex -w 0 | tr -d = | tr A-Z a-z
cat vault/$path | secretbox -d -K "$datakey" -M 52434c4f4e450000 -b 65536No data stays: the key comes from the password of the Request, and it goes away with the answer. The same tools read a configuration file of rclone with a password (sha256sum, basenc --base64 -d and secretbox -d), so a FIFO can keep the configuration of its remotes encrypted, and only a Request with the password opens it.
basenc -d needs the padding of base32, as GNU basenc 9.4: with no padding, GNU writes the data but exits with 1. rclone writes no padding, so the pipeline adds it with sed first. I read the source of basenc and of gnulib for the rules, and a script compares the builtin with GNU basenc on random options and inputs. It showed one surprise: --base32hex -d reads W to Z as base32, because GNU changes the alphabet before it decodes. The builtin does the same.
A second script runs the pipelines in the shell of mkfifo.com, and compares them with rclone 1.75.1 in the two directions: the names in each encoding, the files, a bad block, and the configuration files. It also compares scrypt with Python. The key of rclone takes 16 MiB of memory and about 100 ms in V8, but 1.7 s in the interpreter of my tests: it counts 4,194,304 steps of a Request.
Universal files
A FIFO is a file: a GET gives its bytes. So a backend of rclone is a FIFO too, as a folder of files, and the builtins of files work with it as with each FIFO: ls and lsjson list it, and cat reads it. The FIFO rclone of the site holds the backends, one FIFO for each: s3 now, crypt and alias later. Each backend has all its envelopes, with no FIFO for its parts.
cat is a stream now, as cat of a shell. An answer whose body is one $(cat FILE...) sends each file to the client as it comes, and the reply of an HTTP request with Reply-To: - does the same. The runtime of BEAM.com gives the body of a fetch() to the VM in pieces, with a window, and sends a chunked answer of the VM to the client in pieces. Req reads the next piece in the process of the client, only after the client has the last one. So a file of each size goes through, and the VM keeps only one piece of it. A pipe (cat FILE | gzip) still takes all the value, at most 1 MiB.
cat NAME reads a FIFO itself, as < NAME. An error of cat shows the path of a file, not its query: the query can have a key.
The first backend: s3, for R2
The first backend is not crypt, but s3: the buckets of Cloudflare R2, with the API of Amazon S3. crypt needs a remote under it, and s3 is such a remote:
export R2='-H RCLONE_S3_ACCESS_KEY_ID:ID -H RCLONE_S3_SECRET_ACCESS_KEY:SECRET -H RCLONE_S3_ENDPOINT:https://ACCOUNT_ID.r2.cloudflarestorage.com'
curl $R2 -X PROPFIND -H 'Depth: 1' https://mkfifo.com/s3/BUCKET/docs
curl $R2 'https://mkfifo.com/s3/BUCKET?list-type=2&prefix=docs/'
curl $R2 -L https://mkfifo.com/s3/BUCKET/docs/a.txtThe configuration follows rclone: the header fields have the names of the environment of rclone, and a field of the query takes the place of its header field, as a flag. mkfifo.com keeps no key and no file. A GET from a client gets a redirect to a presigned URL of R2, and a PUT too. A pipe or a FIFO gets the bytes, so a FIFO can give a film from a bucket with one line: $(cat "s3/BUCKET/movie.mp4?..."). GET /s3/BUCKET?list-type=2 gives the list of S3 as it is, and the answer of a PROPFIND reads that list with cat and changes its XML into the JSON of rclone lsjson with jq.
My first version had four FIFOs: s3 for the configuration, s3api for the requests, sigv4 for the signature, and s3list for the lists. But rclone must hold only backends, as a folder holds folders. Now each envelope of s3 does its own steps. Two features of the envelopes made this possible:
- A Request must agree with only one envelope, but the key can come from a header field or from the query. A line of
jqin a pattern fails when the configuration is not valid (section 7.8), so the envelope of a valid configuration and the envelope of the400never agree with one Request. - The signature is a chain of HMACs: the key of each step is the output of the step before it.
hmactook its key as text, so I added-K, which takes the key in hexadecimal, asopenssl. Now each envelope signs withsha256sumandhmac -Kinsh -c, with no builtin for S3:
key=$(printf %s "$day" | hmac "AWS4$secret"); key=$(printf %s "$region" | hmac -K "$key")The envelopes repeat the configuration and the signature, so a script writes them from one source.
The URL of R2 has the account in its host. Before, a URL could have no variable in its host, so that a value could not send a request to a different site. Now a variable can be one full label, before two fixed labels or more: https://$account.r2.cloudflarestorage.com/. When the egress sends, the value must be one label of letters, digits and -. A value such as evil.com/x makes the egress fail.
A PUT and a DELETE of /NAME/PATH got 404 before. Now they are Requests, with their method in X-HTTP-Method-Override, and the client cannot change it. PUT /NAME and DELETE /NAME stay the save and the delete of an envelope.
I have no bucket of R2 for the tests. So a stub of R2 in the tests examines each signature with its own code, and a script sends the same Requests to rclone serve s3, the S3 server of rclone, which examines each signature with the code of gofakes3, also for a file of 3 MiB through the stream. The keys of the tests have a space, & and <, because the XML of a list gives them as entities. One question stays: a value in a URL gets percent-encoding, so a / of a key is %2F in the URL, and the signature has the key with /. The server of rclone reads %2F as /, and so must R2. I could not check this on R2 itself before the merge.
Keys in a FIFO, and the next page
With the key in each Request, each client must know the key. A FIFO can keep it instead: its egress envelope goes to s3, with the key in sealed fields. Only the FIFO opens them, and s3 gets them as header fields. The To of a FIFO egress has no variable, because the save examines the graph of the FIFOs, so the path comes in a header field:
To: s3
X-HTTP-Method-Override: GET
Path: photos/2026/$file
RCLONE_S3_ACCESS_KEY_ID: <sealed:...>
RCLONE_S3_SECRET_ACCESS_KEY: <sealed:...>
RCLONE_S3_ENDPOINT: https://ACCOUNT_ID.r2.cloudflarestorage.com
Reply-To: -My first envelopes for Path did not agree with any Request. They were copies of the envelopes of the routes, and those read the key from the query too, with the line Content-Type: form. A Request from a FIFO egress has no query and no form, so that line did not agree. The copies now read the key only from the header fields.
A list of S3 has at most 1,000 keys. A list with more has NextContinuationToken, and GET /s3/BUCKET?list-type=2&continuation-token=TOKEN now gives the next page, as in S3. The query of the signature has its keys in order, and a token has +, / and =, so the test gives a token with them.
A Request could cause at most 64 egress messages, with its hops and tees. A FIFO that sends a request for each key of a list needs more, and a Worker of Workers Paid can send 10,000 subrequests for each request. The limit is 1,000 now.
A real bucket, and no list of all the levels
Then I got a test bucket of R2, and sent the Requests of s3 to it through mkfifo.com: the lists, the next page, the presigned links of GET and PUT, the bytes of a file of 3 MiB, a name with a space, & and <, DELETE, and the keys in sealed fields of a FIFO. All of them worked. R2 reads %2F in a path as /, as the server of rclone does, so the question above has its answer.
One answer was not correct. A PROPFIND with Depth: infinity of the bucket gave 1,000 of its 1,105 keys, because one list of R2 has at most 1,000 keys. A list of all the levels can be very large, and rclone does not ask for one: it lists one directory at a time, and goes into each directory, to the depth of --max-depth. WebDAV lets a server refuse infinity (RFC 4918). So Depth is now that number, from 1 to 1000. Depth: infinity, and a PROPFIND with no Depth, get 403 with the error propfind-finite-depth.
The commands do as rclone does. lsf --max-depth 2 NAME sends Depth: 2. ls, lsl and -R with no --max-depth walk: a PROPFIND with Depth: 1 for the path, then one for each directory in it. s3 lists one level with the delimiter /, and a larger Depth with no delimiter. An envelope can give the Depth to lsjson:
To: blog/:path*
X-HTTP-Method-Override: ^PROPFIND$
Depth: ^(?<depth>[0-9]+)$To: -
Content-Type: application/json
$(lsjson -M --max-depth $depth blog/$path)Each PROPFIND and its answer are two messages of the Request, so a walk of more than about 500 directories stops at the limit of 1,000 messages.
Pages
A number in Depth stops the large list of all the levels, but one folder can also have more than 1,000 keys, and one list of R2 has at most 1,000. So the answer of a PROPFIND can now be one page, and it gives the next page in the header field Link, as the APIs of the Web do (RFC 8288):
Link: <?continuation-token=TOKEN>; rel="next"The FIFO selects the size of a page. For s3, it is the list chunk of rclone, RCLONE_S3_LIST_CHUNK or list_chunk in the query, and the token is the NextContinuationToken of R2. The commands follow each page to the same path, with the fields of their query. A link to a different path is an error, so a key in a query goes only to the same FIFO. An answer can give Link only for the next page, on the same site, because a different link could make a browser load a different site.
The header field and the body both need the list of R2, and a template has no variable that two lines share. So in one envelope, cat of the same files now reads them one time. A GET does not change its file, so the second cat gets the same bytes, and R2 gets one request for each page.
The S3 server of rclone found one more problem. With the delimiter / and pages of one key, its token of the third page gave the first page again, and ls followed the loop until the limit of 1,000 messages. Now ls stops when a link comes back to a page before. R2 gives its pages in the correct sequence: a folder of 1,100 keys in the bucket of the test gave two pages.
crypt
crypt now reads a crypt remote of rclone: PROPFIND /crypt/PATH lists it, and GET /crypt/PATH reads a file. The configuration has the names of rclone again. RCLONE_CRYPT_REMOTE is the remote under it, with the configuration of that remote in its query, and RCLONE_CRYPT_PASSWORD is the password as rclone obscure gives it:
curl -X PROPFIND -H 'Depth: 1' -H 'RCLONE_CRYPT_REMOTE: s3/BUCKET/secret?access_key_id=...' -H 'RCLONE_CRYPT_PASSWORD: ...' https://mkfifo.com/crypt/docsThe envelopes are the pipelines above. For a list, a script encrypts each segment of the path, lists the remote with lsjson --max-depth N, and then makes one more script with jq: for each entry, a pipeline that decrypts each segment of its name. eval runs it. A name that does not decrypt is not in the list, as rclone skips it. The size of a file comes from its size in the remote: less the header of 32 bytes, and less 16 bytes for each block of 64 KiB.
Each name takes about 4 builtins, and a Request could run 1,024 builtins: a list of about 250 names. I did not want a tool only for the names of crypt, so the limit is now 8,192 runs for one Request. The other limits stay: 16 MiB of work and 10,000,000 steps.
Two problems came up. The body of an envelope reads each line alone, so a script in single quotes cannot have more lines: each script of sh -c is one line now. Then the list of the remote gave the envelopes of rclone, not the bucket. A FIFO lists the files of a FIFO that it holds with no Request, and rclone holds s3. A script of sh has the home ~, but its lists still had the home of the envelope. Now a list in a script is a Request to the FIFO, as for a client, and a script does not see the envelopes of the holder.
The tests use files that rclone wrote. scripts/s3.sh copies files into crypt remotes with rclone 1.75.1: names in base32 and in base64, the names of the directories encrypted and not, a second password, an empty file, and a file of two blocks. Then a test reads them through crypt and the S3 server of rclone. The name that the builtins make for big.bin is the name that rclone wrote.
crypt only reads now. PUT and a sealed password in a FIFO come next. Each backend streams the data to its destination, and keeps nothing.
Large files of crypt
At first, crypt gave only files of at most 1 MiB, because secretbox -d took all its input as one value. Now cat and secretbox stream together. In an answer, $(cat FILE | secretbox -d -K KEY -M MAGIC -b 65536) sends each box of 64 KiB to the client when it opens. The pieces from R2 do not have the size of a box, so the stream keeps the rest of a piece for the next one. The nonce of a box is the first nonce plus the number of the boxes before it. The NIF did not change: it opens a list of full boxes, as before.
The GET of crypt needs the key for two things: the name of the file in the remote, and its data. Two runs of scrypt take 0.4 s, and most of the steps of a Request. So the GET makes the keys one time, and its answer is cat of a different file of crypt: crypt?crypt_file=FILE&crypt_key=KEY. That envelope has no route, and its answer is the stream. The key goes from one envelope to the other in the same Request.
The test of rclone has a file of 3 MiB too. rclone writes it with crypt, and the test reads it back through crypt, a box at a time.
Remotes over remotes
rclone has backends that read other remotes. alias gives a different name to a remote, or to a path in it. combine puts each remote in a directory of its own. union puts the files of all its remotes in one tree. Now they are FIFOs of rclone too. Each remote under them is NAME/PATH?QUERY, with its configuration in its query, as the remote under crypt. Thus union can join two buckets, or a bucket and a crypt remote.
union lists the path in each remote, and keeps one entry for each path. rclone selects that entry with a search policy: ff takes the first remote that has the path, and newest takes the newest file. The jq of mkfifo.com has no def, so the program of newest has all the steps that change a time into a number: the days of the date, the seconds, and the offset. The policies that need the free space of each remote get an error for now: R2 does not tell its free space.
This time the test compares with rclone itself. The test of rclone starts rclone serve s3, then runs rclone lsjson and rclone cat with the same remotes, for each depth. The first run found a difference that the old tests did not see: rclone gives the size 0 to a directory of S3, and -1 only to a bucket. s3 gave -1 to each directory. Now it gives 0, and the test compares crypt with rclone too.
Compressed files
The backend compress of rclone keeps each file compressed, with gzip or with zstd. Its names have the size of the data: a.txt.BgAAAAAAAAA.gz is a.txt, with 6 bytes, as a number of 8 bytes in base64url. The jq of mkfifo.com has @base64d, but it gives text, not bytes. So the envelope decodes the 11 characters itself: 6 bits for each character, then the 8 bytes.
A compressed file can be large, so gunzip and a new builtin, zstd -d, stream after cat, as secretbox does. zstd is the zstd tool 1.5.7: Erlang/OTP has the same library, and the tests compare the bytes with the tool for each level from 1 to 19. The library did not do two things that the tool does. A frame that stops too soon gave no error, so the builtin reads the header of each frame and of each block, which has the size of the block. And for an empty input, the tool writes the size 0 in the frame, so the builtin does too.
The stream of gunzip was harder. zlib in Erlang does not tell where the data of a member end: it says finished also when its input ends. In a stream, the input ends only with the last piece. So the stream decompresses the first member, and expects its trailer in the last 8 bytes. A small input gets more checks: two members can have the same trailer, and the first property test found exactly that, with two copies of one byte.
rclone writes zstd that a reader can seek: frames of 1 MiB, and at the end a skippable frame with the table of the frames. The test of rclone reads these files and the files of gzip with rclone cat and with mkfifo.com, and they agree.
Files in chunks
The backend chunker of rclone keeps a large file in chunks: big.iso.rclone_chunk.001, big.iso.rclone_chunk.002, and the next numbers, and a small JSON object in big.iso with the size and the number of the chunks. The list must join the chunks again. rclone does this in processEntries: it sorts the names of a directory, so the metadata comes before its chunks, and it puts each chunk in its file. The envelope does the same in one reduce of jq. The number of a chunk is a key of an object, not an index of an array: a name with the number 9999999 would make a large array.
The GET was the hard part. The answer must give the chunks one after the other, as a stream. cat takes more files, but each word of cat is one file, and the envelope does not know the number of the chunks before the list. A FIFO for each chunk, one after the other, stops after 8 hops. The shell has a tool for this work: xargs. Now $(LIST | xargs -d '\n' cat) streams too: LIST runs first, then cat of all the files streams to the client.
The test of rclone writes files in chunks of 1 MiB, with the metadata and with no metadata, and compares rclone lsjson and rclone cat with mkfifo.com. With no metadata, rclone keeps a small file as one chunk too: top.txt.part0.
crypt writes
Until now, crypt only read. A file had to come from rclone itself. Now PUT /crypt/PATH encrypts the body and DELETE /crypt/PATH removes the file, so an app can keep private files in R2 with no key of its own on disk: the password comes with each request, and mkfifo.com forgets it.
The pieces were there. secretbox already writes the format of rclone: the magic bytes, a random nonce, and a box for each 64 KiB. eme and basenc already encrypt names for the GET. So the envelope of the PUT makes the keys with scrypt, encrypts the path and the body, and sends the bytes to s3.
Two things were missing. First, s3 gave a redirect for each PUT, also from a FIFO: the client then sends the plain body to that URL, and the file in R2 is not encrypted. Now a FIFO egress with Proxy: true sends its body to R2 itself, signed with UNSIGNED-PAYLOAD, as proxy=true already gave the bytes of a GET. With no Proxy, a PUT still gets the redirect, for a large file.
Second, no envelope could take a body of bytes. A body line does not agree with a body that is not UTF-8, so a photo did not agree with any envelope. I did not want text patterns to find matches in bytes by chance, so only the line of the full body, (?s)(?<body>.*), takes such a body now, with all its bytes.
Bytes still cannot go where only text fits. A JSON string, a header field and an email hold only UTF-8 text, and the editor gets each report as JSON. So a capture of bytes in a JSON body, in a header field or in an email makes that egress fail, with an error that says to encode it first, for example with $($body | base64 -w 0). A header field also refuses a control character such as NUL. The editor shows the size of such a capture, not its bytes.
A write is one request, so the body is at most 1 MiB, and the output of secretbox too: the data are at most 1,048,288 bytes, the 1 MiB less the header of 32 bytes and 16 bytes for each box. The PUT makes the keys two times, for the path and for the data. That is about 0.4 s in the Worker, and fine for a photo.
The test writes files with mkfifo.com, then reads them with rclone cat: a file of two boxes with random bytes and a line break at its end, an empty file, a file of exactly 1,048,288 bytes, and the names in base32 and in base64.
The backends move to rfetch
Everything above is the work of rclone, done again with builtins in envelopes: the signatures of S3, the names and the data of crypt, the lists of the backends over backends. It works, but each new backend was a lot of jq, and each fix had to happen twice: once in rfetch, my port of rclone to JavaScript, and once in the envelopes. rfetch now runs as a Worker of its own at rfetch.pages.dev, with its own tests against rclone, and it already has these seven backends and more.
So the envelopes of rclone now send each request to rfetch, and give its answer to the client. mkfifo.com still keeps no file and no key, and rfetch keeps nothing between requests either.
The options were the hard part. My first try named each option in each envelope, in three places, RFETCH_S3_ACCESS_KEY_ID, RCLONE_S3_ACCESS_KEY_ID and access_key_id in the query, and sent each one on under the same name. It was explicit, but a FIFO is for the people who call it, and they know the options of rclone, not my list. An option that I forgot could not get through, and a remote inside crypt could not get its own options at all.
So the options now pass through as they come. Two parts of that are new in mkfifo.com:
- A route can match the query too, the way
URLPatterndoes in a browser:To: s3/:path*\?:query*captures the whole query in$query, and the egress sends?$queryto rfetch. I check these routes againstURLPatternof Node.js with thousands of random routes and queries. - An egress header line like
RFETCH_*:copies each header field of the request whose name starts withRFETCH_. Fields of Cloudflare, of mkfifo.com and of the connection, and cookies, never get copied.
Secrets still go in header fields, and the other options in the query. Nothing else from the request goes to rfetch.
Two more small changes made this work:
- A route like
s3/:path*captures a path with/in it. In the URL of an HTTP egress, each variable got percent-encoding, soa/b.txtbecamea%2Fb.txt. Now a capture of the route keeps its/in the path of the URL, and a capture of the query keeps its&and=in the query. They are still encoded everywhere else. - The answer of a reply kept only
Content-TypeandLocation. Now it also keepsContent-Range,Accept-Ranges,Last-ModifiedandETag. Socurl -r 0-99gets the first 100 bytes of a file, and a download can resume.
Some things went away. s3 gives no redirect to a presigned URL now: the bytes go through rfetch, so a PUT is at most 1 MiB for now. A PROPFIND is one list from rfetch, with no pages. The HTTP egress cannot send PROPFIND, so the envelope sends a GET with Accept: application/json and the Depth, and rfetch answers it with the same list. A remote under crypt or union is now a remote of rclone, like :s3,access_key_id=...:bucket, not the path of a FIFO.
rfetch reads only the default names of crypt: base32, with the names of the directories encrypted. With no check, a request for names in base64 would read and write the wrong names. So crypt answers 501 to each other name option, and the request goes nowhere.
The 102 files of the seed and their three scripts became 60 files from one small script. With the options passing through, a new backend is one more line in that script, so the other backends of rfetch are FIFOs now too, in 114 files: archive, azureblob, doi, drive, ftp, http, sftp, tiles and webdav. Only local and memory stay out: the Worker has no local disk, and a memory remote would put the files of different people in one place.