senro › guides

file uploads

a form with a file input posts its fields as multipart/form-data, and the file arrives as bytes that are rarely text. senro reads every request as bytes, turns a multipart body into the same form an action already reads with web/form, writes each file to disk under a random key, and serves the file back by that key. this is what Active Storage's disk service does for a Rails app. this guide covers reading the request, the form an action sees, where the files go, and how they come back.

reading a request

a read from a socket answers whatever has arrived, and a browser often sends a request's head in one packet and its body in the next. the server reads until it has the head, works out from content-length how long the whole request is, and reads again until it has that many bytes or the client stops sending.

# A request read as bytes, and a form sent as multipart/form-data, which is how
# a browser sends a file. Each file is written to the app's storage directory
# under a key of 32 random hex digits, and the form the action reads holds
# the key where the file was, as Active Storage's disk service keeps a blob.
# /senro/blobs/<key> serves it back.
#
# The most a request's body may hold, 10 MiB. A request that promises more
# is answered 413 before its body is read.
body_limit = 10485760

# A read is one segment of what the client sent, and a browser often sends a
# request's head in one packet and its body in the next. The server reads
# again until the body has the bytes content-length promised, or the client
# stops sending. `wanted` is the whole request's length once its head is in,
# and -1 until then.
fn gathered c had wanted
  net/read_bytes c .> (more -> gathered_after c had wanted more)

fn gathered_after c had wanted more
  return effect had if length more == 0
  raw = text/append had more
  want = wanted_of raw wanted
  return effect raw if whole? raw want
  return continued c raw want if wanted < 0 and want >= 0 and expects? raw
  gathered c raw want

fn wanted_of raw -1
  head_end raw (found raw blank_line 1)

fn wanted_of _ w
  w

fn head_end _ none
  -1

# A body larger than the limit is not waited for: the request is whole at
# its head, and `opened` refuses it.
fn head_end raw at
  head = ascii_of (some_bytes (text/slice raw 1 (at - 1)))
  promised = length_promised (header_in head "content-length")
  return at + 3 if promised > body_limit
  at + 3 + promised

fn whole? raw w
  w >= 0 and length raw >= w

a body longer than 10 MiB is not waited for. the request counts as whole once its head is in, and the server answers 413 Content Too Large without reading the body. curl and other HTTP clients send Expect: 100-continue before a large upload and wait to be told to go on, so the server says so once the head is in:

# A client that sends `Expect: 100-continue` waits to hear that the server
# will take its body before it sends it, as curl does for a large upload.
fn continued c raw want
  net/write c "HTTP/1.1 100 Continue\r\n\r\n" .> (_ -> gathered c raw want)

fn expects? raw
  at = found raw blank_line 1
  continue_asked? raw at

the head's content-length and content-type are read from its own lines, with each byte past ASCII read as a question mark, so a head that is not text is still measured rather than refused halfway through.

the form an action sees

a body that is not multipart is decoded as UTF-8, and one that is not UTF-8 is answered 400. a multipart body is cut along its boundary into parts. a part with no filename is a text field, and its value is its text. a part with a filename is a file: senro writes it to disk and the field's value becomes the key it was written under. a file input left empty sends a part with an empty filename, and its value is "".

fn stored_as p none
  text_or_blank p.content .> (v -> effect [(field_name p.name) v])

fn stored_as p ""
  effect [(field_name p.name) ""]

fn stored_as p filename
  os/random_bytes 16
    .> (r -> written (hexed r) p filename)
    .> (key -> effect [(field_name p.name) key])

fn written key p filename
  storage_dir
    .> (dir -> os/make_dir dir .> (_ -> effect dir))
    .> (dir -> os/write_file "{dir}/{key}" p.content .> (_ -> effect dir))
    .> (dir -> os/write_file "{dir}/{key}.meta" "{p.kind}\n{filename}")
    .> (_ -> effect key)

# Where files are kept: SENRO_STORAGE, or storage/ beside the app.
storage_dir = os/env "SENRO_STORAGE" .> (d -> effect (dir_or d))

the fields are then written out the way a browser writes an ordinary form, title=a%20cat&photo=4f1c…, so an action reads them with web/form and web/value as it reads any other. it stores the key in a column the way it stores a title.

import "../../../web"
import "std/list"
import "std/net/http"
import "std/text"

# What a form with two files and a title reads as: the title as typed, and
# each file as the key it was stored under.
pub fn create req _
  f = web/form req.body
  said = list/map ["title" "photo" "notes"] (n -> web/value f n)
  effect (http/text_page (text/join (list/to_list said) "\n"))

a scaffold field declared name:attachment writes all of this: a text column for the key, a file input in a form sent as multipart/form-data, and an update that keeps the stored file when the input is left empty. the generators guide shows the notebook's cover.

files go to storage/ beside the app, or to the directory SENRO_STORAGE names. each key is 32 hex digits from 16 random bytes, and beside the file a .meta file keeps the content type and the filename the browser sent.

serving a file

a view links a stored file with web/blob_path key, which names /senro/blobs/<key>. a key that is not 32 hex digits is answered 404 without looking for a file. an image or a PDF is sent as its own type and shown in the page. anything else is sent as application/octet-stream and as a download, so an HTML file someone uploaded never runs as a page of the app. every answer carries x-content-type-options: nosniff.

fn blob_headers kind filename
  shown = list/any? inline_kinds (k -> k == kind)
  [
    "content-type: {shown_kind shown kind}"
    "content-disposition: {disposition shown filename}"
    "x-content-type-options: nosniff"
    "cache-control: private, max-age=31536000, immutable"
  ]

inline_kinds = [
  "image/png"
  "image/jpeg"
  "image/gif"
  "image/webp"
  "image/avif"
  "application/pdf"
]

fn shown_kind true kind
  kind

fn shown_kind false _
  "application/octet-stream"

fn disposition true _
  "inline"

fn disposition false filename
  bare = without (without filename "\"") "\\"
  "attachment; filename=\"{bare}\""

a key is unguessable, so a file is as private as its link. an app that must keep a file from someone who has the link should serve it from an action that checks the viewer instead.

what the spec checks

senro's spec serves a small app and posts to it the way a browser does, the head first and the multipart body a moment later. the form holds a title, a PNG whose bytes include a NUL, a 255, a line break and the boundary's own dashes, and an HTML page. it then asks for each file by its key, sends a body that promises 20,000,000 bytes, asks for two paths that are not keys, and repeats the upload with Expect: 100-continue:

title: a cat & a hat
keys are 32 hex digits: true
photo: HTTP/1.1 200 OK, content-type: image/png, content-disposition: inline, x-content-type-options: nosniff, same bytes: true
notes: HTTP/1.1 200 OK, content-type: application/octet-stream, content-disposition: attachment; filename="x.html", x-content-type-options: nosniff, same bytes: true
HTTP/1.1 413 Content Too Large
HTTP/1.1 404 Not Found
HTTP/1.1 404 Not Found
asked to continue: HTTP/1.1 100 Continue, then HTTP/1.1 200 OK

both engines print the same lines. the photo comes back byte for byte and inline, and the HTML page comes back as a download under its own name.

limits

files are kept on the server's disk only, and an app on more than one machine needs them somewhere shared. the boundary search is written in kanso: about 0.1 seconds a megabyte natively and about a second interpreted, which the 10 MiB limit keeps bounded.