a json library

part i taught the pieces. part ii spends them. the repository ships lib/json—a complete json decoder and encoder, escapes and \uXXXX and int-versus-float and failure positions included—written to prove the language can earn a living. it is six files and four hundred and change lines, and this chapter reads all of it, byte by byte. every panel below quotes the real files, and the executed samples either run the library itself or replay one of its moves small enough to fit on a page. first, what it feels like to use:

import "using"

ok = decode "\{\"name\": \"kanso\", \"tags\": [\"fast\", \"simple\"]}"
bad = decode "\{\"name\": nope}"
print (describe ok)
>> print "recoded: {encode ok}"
>> print (describe bad)

decoded: { "name": "kanso" "tags": ["fast" "simple"] }
recoded: {"name":"kanso","tags":["fast","simple"]}
parse failed at byte 10: invalid literal

the module behind this panel is the library itself—the sample directory links the real lib/json files next to that main. three chapters shake hands in twelve lines: decode returns a value or an err, dispatch pulls the failure apart in describe, and the happy path never checks anything. note what the failure says: byte 10, invalid literal—the parser saw n, committed to null, and reported exactly where the input betrayed it.

six files, one namespace

the library is a directory, so chapter 07 already told you its shape: one namespace, files split by topic, alphabetical inside. the split reads like a table of contents:

• json.kso—the public api: decode, encode, must, the failure types
• value.kso—parsing values: objects, arrays, literals
• text.kso—strings, escapes, hex digits
• number.kso—ints, floats, and which bytes may start them
• scan.kso—whitespace and low-level cursor work
• json_test.kso—the suite

value.kso calls scan.kso's helpers without ceremony, and the test file reaches private helpers when it wants to. (the underscored names you'll see are house style from before pub arrived—the sweep that retires them rides with import enforcement.) the api file is where every read should start:

type _parsed
  pos:int
  value:json

type parse_failure
  position:int
  reason:string

fn _finish _ none
  err (parse_failure 0 "unexpected end of input")

fn _finish cs (_parsed p v)
  p2 = _skip_ws cs p
  if (p2 > (length cs)) v (_fail p2 "unexpected trailing characters")

fn decode s
  cs = bytes s
  _finish cs (_parse_value cs 1)

decode is three lines because the whole design lives in its two collaborators. every internal step returns a _parsed—a cursor position paired with the value parsed so far—because a parser's real product is not just a value but how far it got. and _finish is dispatch doing the last two checks: an absent result is the empty-input failure, and a present one is accepted only if nothing but whitespace remains. decode "1 x" fails at byte 3, and you will meet that exact assertion in the suite.

the next byte is the dispatch

most hand-written parsers begin with a lexer, a token enum, and a loop with a switch in it. kanso-json has none of those, because it doesn't need a value to branch on—it needs a value to dispatch on, and the next byte already is one. the heart of the parser is the chapter-03 move at industrial scale:

# value starts by byte: 34 quote, 91 bracket, 102 f, 110 n, 116 t, 123 brace
fn _value_for 34 cs p
  _parse_string cs (p + 1)

fn _value_for 91 cs p
  _parse_array cs (p + 1)

fn _value_for 102 cs p
  _word cs p _bytes_false false

fn _value_for 110 cs p
  _word cs p _bytes_null json_null

fn _value_for 116 cs p
  _word cs p _bytes_true true

fn _value_for 123 cs p
  _parse_object cs (p + 1)

fn _value_for none _ p
  _fail p "unexpected end of input"

fn _value_for c cs p
  numeric = _is_number_start c
  if numeric (_parse_number cs p) (_fail p "unexpected character `{utf8 [c]}`")

eight arms, one per way a json value can begin—including the none arm, because at past the end of the byte array returns none, and running off the end of input is just an absent byte, dispatched like everything else. the three keyword literals share one checker: _word slices the next few bytes and compares them against the expected spelling as a plain value equality. here is that whole path, small enough to re-run on its own:

import "std/text"

type parse_failure
  position
  reason

bytes_false = [102 97 108 115 101]

bytes_null = [110 117 108 108]

bytes_true = [116 114 117 101]

fn decode_literal s
  cs = text/bytes s
  value_for cs[1] cs 1

fn fail p reason
  err (parse_failure p reason)

pub play =
  print (show (decode_literal "true"))
  >> print (show (decode_literal "null"))
  >> print (show (decode_literal "nope"))
  >> print (show (decode_literal "?"))

fn show (err (parse_failure p reason))
  "err at {p}: {reason}"

fn show v
  "ok: {v}"

fn value_for 102 cs p
  word cs p bytes_false false

fn value_for 110 cs p
  word cs p bytes_null "null"

fn value_for 116 cs p
  word cs p bytes_true true

fn value_for none _ p
  fail p "unexpected end of input"

fn value_for c _ p
  fail p "unexpected character `{text/utf8 [c]}`"

fn word cs p expected v
  n = length expected
  seg = text/slice cs p (p + n - 1)
  if (seg == expected) v (fail p "invalid literal")

ok: true
ok: null
err at 1: invalid literal
err at 1: unexpected character `?`

the constants are the spellings—[116 114 117 101] is true in utf-8—and comparing byte arrays is just ==, because arrays are values. nope starts with n, so the parser commits to null and _word reports the mismatch; ? starts nothing, so the catch-all arm names the offending byte. and notice what the parser never writes: not one line of "check if the last step failed." a bad literal becomes an err deep inside, and the railway from chapter 04 carries it out through every caller with its position intact.

arrays and objects follow the same discipline one level up. _array_delim dispatches on the byte after an element—comma means more items, 93 (a closing bracket) means done, none means the input ended mid-array—and the object path adds only a key: _obj_key_start insists on a quote, _expect_char demands the colon, and _obj_value folds each pair into a map with put. there is no recursion limit to manage and no parser state to restore, because every step passes its whole world—the bytes and a position—forward as arguments.

strings and the escape path

strings are where json parsers earn or lose their keep, and text.kso is the library's biggest file. the design is a two-speed scanner. the fast path bets that most strings contain no escapes: find2 hunts for the next quote or backslash byte, and dispatch on what it found decides which world you're in:

fn _string_at cs 34 start p
  _parsed (p + 1) (_string_ok p (utf8 (slice cs start (p - 1))))

fn _string_at cs 92 start p
  _str_chars cs p (concat [] (slice cs start (p - 1)))

fn _string_at _ none _ p
  _fail p "unterminated string"

fn _string_at cs _ start p
  _string_scan cs start (p + 1)

fn _string_scan cs start p
  n = find2 cs p 34 92
  _string_at cs (at cs n) start n

read from the bottom. a quote (34) ends the string, and the value is one slice of the original input—no copying, no character loop. a backslash (92) abandons the bet: the bytes so far are moved into an accumulator and the slow path takes over, appending decoded bytes one escape at a time. none means the input ended inside the string. the slow path is another dispatch table—one arm per legal escape character, each pushing the byte it names:

fn _str_escape cs 110 p acc
  _str_chars cs (p + 2) (push acc 10)

fn _str_escape cs 117 p acc
  _str_unicode cs p acc

fn _str_escape _ c p _
  _fail p "invalid escape `\\{utf8 [c]}`"

fn _str_unicode cs p acc
  h1 = _hex_digit (at cs (p + 2))
  h2 = _hex_digit (at cs (p + 3))
  h3 = _hex_digit (at cs (p + 4))
  h4 = _hex_digit (at cs (p + 5))
  code = h1 * 4096 + h2 * 256 + h3 * 16 + h4
  _str_chars cs (p + 6) (concat acc (bytes (from_code code)))

the file has eight arms for the named escapes—110 is \n, and its siblings for \", \\, \/, \b, \f, \r, \t read identically—plus an end-of-input arm and the two shown here. \uXXXX is the interesting one: four hex digits become a code point, from_code turns the code point into a character, and its utf-8 bytes join the accumulator. the digit math is worth running with your own eyes:

import "std/text"

fn decode_u s
  cs = text/bytes s
  h1 = hex_digit cs[1]
  h2 = hex_digit cs[2]
  h3 = hex_digit cs[3]
  h4 = hex_digit cs[4]
  text/from_code (h1 * 4096 + h2 * 256 + h3 * 16 + h4)

fn hex_alpha c
  if (96 < c and c < 103) (c - 87) (hex_upper c)

fn hex_digit c
  if (47 < c and c < 58) (c - 48) (hex_alpha c)

fn hex_upper c
  if (64 < c and c < 71) (c - 55) (err "invalid hex digit")

pub play =
  print (decode_u "00e9")
  >> print (decode_u "305f")
  >> print (decode_u "7b80")

é
た
简

those are the library's own hex functions, verbatim—the three-way relay _hex_digit → _hex_alpha → _hex_upper tries digits, then lowercase, then uppercase, and a byte that is none of them raises the one genuinely unhandleable failure in the file. that last character is 7b80—简, the first half of 簡素. the library can spell its own name.

numbers without a lexer

numbers get the same treatment: find where the number ends, then decide what kind it is. _number_end walks forward while the byte could belong to a number; _mark_from re-scans the span asking one question—is there a ., e, or E anywhere in it?—and that answer picks the conversion:

# float marks: . E e
fn _mark_step _ 46 _ _
  true

fn _mark_step _ 69 _ _
  true

fn _mark_step _ 101 _ _
  true

fn _mark_step cs _ p q
  _mark_from cs (p + 1) q

fn _number_end cs p
  if (_is_number_char (at cs p)) (_number_end cs (p + 1)) p

fn _number_value cs p q
  digits = slice cs p q
  raw = if (_mark_from cs p q) (to_float digits) (to_int digits)
  _number_ok p raw

replay the decision on its own inputs—the sample carries the number file's scanning functions wholesale and prints which converter would fire:

import "std/text"

fn classify s
  cs = text/bytes s
  q = number_end cs 1
  digits = text/slice cs 1 (q - 1)
  floaty = mark_from cs 1 (q - 1)
  if floaty "float64 {text/to_float digits}" "int {text/to_int digits}"

fn is_number_char 43
  true

fn is_number_char 45
  true

fn is_number_char 46
  true

fn is_number_char 69
  true

fn is_number_char 101
  true

fn is_number_char none
  false

fn is_number_char c
  47 < c and c < 58

fn mark_from cs p q
  if (p > q) false (mark_step cs cs[p] p q)

fn mark_step _ 46 _ _
  true

fn mark_step _ 69 _ _
  true

fn mark_step _ 101 _ _
  true

fn mark_step cs _ p q
  mark_from cs (p + 1) q

fn number_end cs p
  if (is_number_char cs[p]) (number_end cs (p + 1)) p

print (classify "42")
>> print (classify "-17")
>> print (classify "3.25")
>> print (classify "6e3")

error[naming]: `is_number_char` answers only true or false: name it `is_number_char?`
  --> numbers.kso:10:4
  10 | fn is_number_char 43
          ^
error[naming]: `mark_from` answers only true or false: name it `mark_from?`
  --> numbers.kso:31:4
  31 | fn mark_from cs p q
          ^
error[naming]: `mark_step` answers only true or false: name it `mark_step?`
  --> numbers.kso:34:4
  34 | fn mark_step _ 46 _ _
          ^

(the panel shows classify and main; the sample file also carries _both, _is_number_char, _mark_from, _mark_step, and _number_end verbatim from number.kso.) an integer stays an integer—42 decodes to int 42, not 42.0—and anything marked with a dot or an exponent becomes a float64, so 6e3 is 6000.0. the greedy scan is deliberately looser than the json grammar: _is_number_char accepts any plausible number byte, and to_int or to_float is the real judge. if the conversion fails, _number_ok converts that refusal into a parse_failure pinned to where the number began. let the builtin own the hard part; the parser only needs to know where the number stops and whether a float mark appeared.

failure carries its position

every failure in the library flows through one two-line function:

fn _fail p reason
  err (parse_failure p reason)

# json whitespace bytes: tab, newline, carriage return, space
fn _is_ws 9
  true

fn _is_ws 10
  true

fn _is_ws 13
  true

fn _is_ws 32
  true

fn _is_ws none
  false

fn _is_ws _
  false

fn _skip_ws cs p
  if (_is_ws (at cs p)) (_skip_ws cs (p + 1)) p

a parse_failure is an err wrapping a record: the byte position and a human reason. because the cursor is always an argument, the position is always at hand when something goes wrong—there is no "current location" to reconstruct from a lexer's state. the mini-decoder below is _finish's trailing-characters check in miniature: skip whitespace, read a value, skip whitespace again, and demand the input be over. the sample keeps the library's _is_ws and _skip_ws byte for byte:

import "std/text"

type parse_failure
  position
  reason

fn decode_int s
  cs = text/bytes s
  p = skip_ws cs 1
  q = digits_end cs p
  v = text/to_int (text/slice cs p (q - 1))
  trailing cs (skip_ws cs q) v

fn digits_end cs p
  if (digit? cs[p]) (digits_end cs (p + 1)) p

fn fail p reason
  err (parse_failure p reason)

fn digit? none
  false

fn digit? c
  47 < c and c < 58

fn is_ws? 9
  true

fn is_ws? 10
  true

fn is_ws? 13
  true

fn is_ws? 32
  true

fn is_ws? none
  false

fn is_ws? _
  false

pub play =
  print (show (decode_int "  42  "))
  >> print (show (decode_int "1 x"))
  >> print (show (decode_int "  7 ;"))

fn show (err (parse_failure p reason))
  "err at {p}: {reason}"

fn show v
  "ok: {v}"

fn skip_ws cs p
  if (is_ws? cs[p]) (skip_ws cs (p + 1)) p

fn trailing cs p v
  if (p > length cs) v (fail p "unexpected trailing characters")

ok: 42
err at 3: unexpected trailing characters
err at 5: unexpected trailing characters

positions point at the first byte the parser could not accept: the x in 1 x sits at byte 3, the ; at byte 5. the api file rounds the failure story out with a policy choice—two of them, actually:

type defect
  reason:string

fn _failure_position (err (parse_failure p _))
  p

fn _failure_position _
  0

fn must (err reason)
  err (defect "{reason}")

fn must v
  v

chapter 04 drew the line: an err a caller might reasonably handle is a value to dispatch on; an err nobody should handle rises to the endpoint. decode hands back the first kind, because malformed input is the caller's business—maybe it retries, maybe it reports byte 10 to a user. must converts to the second: wrap a decode of input you control—a config file baked into the program, a fixture—and a failure stops being data and becomes a defect, a bug with your name on it. same railway, different destination, chosen by the caller in one word. and _failure_position is nothing but a projection—dispatch reaching through err into the record to pull the position out, which is exactly how the suite pins error locations.

the suite

sixteen booleans pin the whole library—decoding every value kind, both escape directions, unicode, whitespace, error positions, must, and a round trip:

test_decode_unicode = (decode "\"\\u00e9\"") == "é"

test_error_position = (_failure_position (decode "[1, nope]")) == 5

test_error_trailing = (_failure_position (decode "1 x")) == 3

test_must_wraps_defect = _is_defect (must (decode "nope"))

test_roundtrip =
  text = "\{\"a\":[1,2.5,\"x\"],\"b\":null}"
  (encode (decode text)) == text

test_decode_array ... ok
test_decode_bool ... ok
test_decode_escapes ... ok
test_decode_float ... ok
test_decode_int ... ok
test_decode_nested ... ok
test_decode_object ... ok
test_decode_string ... ok
test_decode_unicode ... ok
test_decode_whitespace ... ok
test_encode_escapes ... ok
test_encode_nested ... ok
test_error_position ... ok
test_error_trailing ... ok
test_must_wraps_defect ... ok
test_roundtrip ... ok
16 passed, 0 failed

look at what each test costs. test_error_position asserts that decoding [1, nope] fails at byte 5—the whole failure model in one comparison, using the private _failure_position because one namespace means the suite reaches wherever it needs. test_roundtrip is the deepest assertion in the file: decode a document, encode the result, and demand the original text back. it holds because encode is canonical—no spaces, keys in map order—so a canonically-written input survives the loop byte for byte. and every one of them is a plain equality between values, which chapter 07 promised would be enough.

encode and pretty-printing

the decoder is a search; the encoder is a fold. one arm per value kind, and every arm appends onto the same byte accumulator, so a byte is written once no matter how deep in the tree it was born. encode itself is two calls: seed the builder, decode the finished bytes back into a string.

fn encode v
  text/utf8 (_encode_onto (text/bytes "") v)

fn _encode_onto acc true
  text/append acc "true"

fn _encode_onto acc json_null
  text/append acc "null"

fn _encode_onto acc n:int
  text/append acc "{n}"

fn _encode_onto acc s:string
  opened = _escape_onto (text/append acc 34) s
  text/append opened 34

fn _encode_onto acc xs:[]json
  if (length xs == 0) (text/append acc "[]") (_encode_list acc xs)

fn _encode_onto acc m:map[string json]
  es = entries m
  if (length es == 0) (text/append acc "\{}") (_encode_map acc es)

(the false and float64 arms read exactly like their neighbors, and _encode_list and _encode_map walk their elements with a comma between each pair.) where the literal arms dispatch on a value, the last four dispatch on a type—n:int, xs:[]json—which is the other half of chapter 03. text/append is the builder from chapter 06: the accumulator owns its buffer and extends it in place when it can, so the fold of appends costs what one pass over the output costs, while every intermediate stays an ordinary value.

the one arm with real work is strings, and the work is _escape_onto: decode's escape path run in reverse. it dispatches on bytes, the way decode's scanner does, and it borrows decode's other habit too—one scan (find2_below: the first quote, backslash, or control byte) proves most strings need no escaping at all, and a proven-clean string is appended whole, in one copy. the sample carries the library's escape functions verbatim and quotes two awkward strings—one with a quote and a newline, one with a bell character (byte 7) in the middle:

import "std/list"
import "std/text"

fn esc_byte acc 9
  text/append (text/append acc 92) 116

fn esc_byte acc 10
  text/append (text/append acc 92) 110

fn esc_byte acc 13
  text/append (text/append acc 92) 114

fn esc_byte acc 34
  text/append (text/append acc 92) 34

fn esc_byte acc 92
  text/append (text/append acc 92) 92

fn esc_byte acc b
  if (b < 32) (u_bytes acc b) (text/append acc b)

fn escape_able acc bs
  list/fold bs acc (a b -> esc_byte a b)

fn escape_clean acc s bs n
  if (length bs < n) (text/append acc s) (escape_able acc bs)

fn escape_onto acc s
  bs = text/bytes s
  escape_clean acc s bs (text/find2_below bs 1 34 92 32)

fn escape_str s
  text/utf8 (escape_onto (text/bytes "") s)

hex_byte_table = text/bytes "0123456789abcdef"

fn hex_code n
  hex_byte_table[n + 1]

pub play =
  bell = text/from_code 7
  print (quote_str "a\"b\nc")
  >> print (quote_str "ring{bell}ring")

fn quote_str s
  "\"{escape_str s}\""

fn u_bytes acc b
  lo = hex_code (b % 16)
  hi = hex_code (b / 16)
  text/append (text/append (text/append acc "\\u00") hi) lo

"a\"b\nc"
"ring\u0007ring"

encode is deliberately compact—machine-to-machine json wants no whitespace. a pretty-printer is the same fold with an indent depth threaded through, and it is small enough to write here in full. this is the chapter's one extension beyond the library, written string-per-node because that style reads best on a page; threading the byte builder through it instead is a good exercise, and the library's own encode above is the worked answer:

import "std/list"
import "std/text"

type json_null

fn entry_block (entry k v) d
  "{pad d}\"{k}\": {pretty v d}"

fn pad 0
  ""

fn pad n
  "  {pad (n - 1)}"

pub play =
  doc = { "name":"kanso" "port":8080 "tags":["fast" "simple"] }
  print (pretty doc 0)

fn pretty true _
  "true"

fn pretty false _
  "false"

fn pretty json_null _
  "null"

fn pretty n:int _
  "{n}"

fn pretty x:float64 _
  "{x}"

fn pretty s:string _
  "\"{s}\""

fn pretty xs:[]some d
  d2 = d + 1
  lines = list/to_list (list/map xs (v -> "{pad d2}{pretty v d2}"))
  inner = text/join lines ",\n"
  "[\n{inner}\n{pad d}]"

fn pretty m:map[string some] d
  d2 = d + 1
  lines = list/to_list (list/map (entries m) (e -> entry_block e d2))
  inner = text/join lines ",\n"
  "\{\n{inner}\n{pad d}}"

{
  "name": "kanso",
  "port": 8080,
  "tags": [
    "fast",
    "simple"
  ]
}

same eight-arm shape as encode, one extra argument. the scalar arms ignore the depth with _; the compound arms deepen it and lay each element on its own indented line. _pad is recursion instead of a loop—dispatch on 0 is the base case—and the whole printer is under forty lines. drop it into lib/json as pretty.kso (with _escape_str in place of the naive string arm) and it would be the module's seventh file, sharing the namespace like everything else.

what you can do now

you have now read a real library end to end, and nothing in it was new: byte dispatch is chapter 03, the err railway is chapter 04, the module layout is chapter 07, and the rest is arithmetic. that is the point of part ii—the language does not grow a special mode for serious programs. you can navigate kanso-json by name, extend it in its own style, and—the sharper skill—recognize its shapes elsewhere: a two-speed scanner betting on the common case, a cursor threaded as an argument so failures always know where they are, a caller choosing between handleable failure and defect in one word. next chapter, the same language simulates elections.

exercises

1. make pretty the module's seventh file. add pretty.kso to a copy of lib/json, swap its naive string arm for one that uses _escape_str, and add test_pretty_roundtrip: decoding a pretty-printed document must equal decoding the compact original.
2. json forbids leading zeros—007 is not a number—but to_int happily accepts it. add the rejection to _parse_number with a positioned failure, and a test pinning where decode "[1, 007]" fails.
3. _str_unicode reads exactly one \uXXXX. characters outside the basic plane arrive as surrogate pairs—"😀" is one emoji, not two characters. extend the escape path to spot a high surrogate, demand its partner, and combine them; pin it with a test.
4. the suite pins positions for a bad literal and trailing characters, but not for an unterminated string. write test_error_unterminated asserting the position decode "\"abc" fails at, predicting the number before you run it.