a command-line tool

this chapter builds a complete program from an empty directory. wordfreq reads some text and prints the words it uses most, with how many times each appears. it takes a file name or reads standard input, accepts a --top option, prints a usage message to standard error when it is called wrongly, and exits with a status a shell script can test. at the end it is a native binary of about 700 KB that runs on a machine with no kanso installed.

nothing here is new language. the program uses records and dispatch (chapter 03), failure as a value (chapter 04), effects (chapter 05), and an entry with a module (chapter 07). the new material is how those pieces fit into a program you would ship.

the layout

a project is a directory holding an entry and the modules it imports. wordfreq has one module, freq, split into three files by job, each with tests beside it:

wordfreq/
  main.kso              the entry: hands the arguments to freq/main
  freq/
    count.kso           words, counting and ranking, rendering: no i/o
    count_test.kso
    options.kso         reading the command line
    options_test.kso
    cli.kso             the edges: files, standard input, exit codes

two names in that tree are chosen with care. the module is freq rather than wordfreq, because kanso build names the binary after the project directory, and it refuses to write a file named wordfreq beside a directory of the same name. and no file inside the module is called main.kso: that name marks an entry wherever it appears, so a module that contains one is refused for holding statements.

the core

start with the part that has no i/o in it. count.kso turns text into words, counts them, picks the most common, and renders them as lines:

import "std/list"
import "std/regexp"
import "std/text"

pub type ranked
  count
  word

# The words of a text, lowercased, in the order they appear.
pub fn words text
  regexp/split "[^a-z']+" (text/lowercase text)
    .> (parts -> list/to_list (list/reject parts (w -> w == "")))

# The n most common words, the most common first. A map's entries come in
# key order and sort_by keeps ties in the order they arrived, so words with
# the same count stay alphabetical.
pub fn top n ws
  ranks = list/map (entries (list/tally ws)) (e -> ranked e.value e.key)
  list/to_list (list/take (list/sort_by ranks (r -> 0 - r.count)) n)

pub fn render rows
  text/join (list/to_list (list/map rows line)) "\n"

fn line (ranked count word)
  "{text/pad_start "{count}" 4 " "}  {word}"

words lowercases the text and splits it on every run of characters that cannot be part of a word, then drops the empty strings a split leaves at the edges. regexp/split answers an effect, because a pattern that does not compile is a failure, so words opens it with .> and answers an effect too. everything after it is plain values.

top is three standard-library calls. list/tally counts each word into a map, and entries hands back the map's pairs in key order, so the words arrive alphabetical. list/sort_by orders by the negated count and keeps equal keys in the order they arrived, so words with the same count stay alphabetical without a second sort key. list/take keeps the first n. ranked is a record so the rest of the module can say r.count rather than remembering which end of a pair holds what.

render right-aligns each count in four columns with text/pad_start, the way uniq -c does, so a column of numbers lines up.

reading the command line

a command line is a list of strings, and reading one is a small parser. it can fail, and the failure is something the person at the keyboard did, so it is a value: bad_usage carries a reason written for them. this is chapter 04's rule about designing the sad path. a verdict the caller acts on ships as a value, and err is kept for what nobody should handle.

import "std/text"

# What the command line asked for: a path, or none for standard input, and
# how many words to show.
pub type options
  path
  top

# The command line could not be read. The reason is for the person who typed
# it, so it says what was wrong in their terms.
pub type bad_usage
  reason

pub fn options_of args
  flags args 1 (options none 5)

fn flags args at opts
  flag args at opts args[at]

fn flag _ _ opts none
  opts

fn flag args at opts "--top"
  with_top args at opts (number args[at + 1])

fn flag args at opts word
  with_path args at opts word opts.path

fn number none
  none

fn number word
  text/to_int word

fn with_top _ _ _ none
  bad_usage "--top wants a number after it"

fn with_top args at opts n
  flags args (at + 2) (merge opts { "top":n })

fn with_path args at opts word none
  flags args (at + 1) (merge opts { "path":word })

fn with_path _ _ _ word _
  bad_usage "one file at a time, and {word} is a second"

the walk is one function per decision. flags reads the argument at a position, and a plain index answers none past the end, which is how the walk stops: flag's first arm takes the none and answers what was gathered. a literal arm catches "--top", and the last arm takes anything else as the path. the number after --top may be missing or may not be a number, and both read as none, so with_top has one arm for the mistake and one for the number. merge builds the next options with one field changed. there is no mutable state, and no loop variable to get wrong.

the edges

the last file is the only one that touches the world. main reads the options and dispatches on what came back:

import "std/io"
import "std/os"

usage = "usage: wordfreq [--top N] [file]"

# The whole program: read the command line, then do what it asked.
pub fn main args
  start (options_of args)

fn start (bad_usage reason)
  io/write_err "wordfreq: {reason}\n{usage}\n" .> (_ -> os/exit 2)

fn start (options none n)
  io/stdin .> (text -> report n text)

fn start (options path n)
  os/read_file path .> (got -> opened path n got)

fn opened path _ (file_not_found _)
  io/write_err "wordfreq: cannot open {path}\n" .> (_ -> os/exit 1)

fn opened _ n text
  report n text

fn report n text
  words text .> (ws -> print (render (top n ws)))

each arm of start is one way the program can begin. bad_usage writes the reason and the usage line to standard error and exits with status 2, the convention for being called wrongly. an options with no path reads standard input. an options with a path reads the file, and os/read_file answers file_not_found for a missing file, an outcome the program plans for (chapter 05), so opened takes it in an arm and exits with status 1.

os/exit is a failure, and it stops the program by the rule that stops every failure: whatever was bound after it does not run. the bind in io/write_err "…" .> (_ -> os/exit 2) is what makes the message appear first, because the exit is built only once the write has finished.

the entry is a single line. it opens the argument list and hands it to freq/main:

import "./freq"
import "std/os"

os/args .> freq/main

running it

here is a short text to count, in the directory above the project:

The kettle sang, and the tea was poured.
The cups were warm, and the tea was green.
Mugi drank the tea and asked for more tea.

everything after -- is the program's own arguments. with a file name and nothing else, it prints the top five:

   5  the
   4  tea
   3  and
   2  was
   1  asked

asked is fifth of the words that appear once because it is first alphabetically. --top changes how many it shows:

   5  the
   4  tea

with no file name it reads standard input, so it works at the end of a pipe:

   5  the
   4  tea
   3  and
   2  was
   1  asked

and the two mistakes a person can make each get a message on standard error and an exit status. --top without a number exits with 2, and a file that is not there exits with 1:

wordfreq: --top wants a number after it
usage: wordfreq [--top N] [file]

wordfreq: cannot open nope.txt

neither message is an error[endpoint] report. those are for failures nobody planned for, and they are addressed to the programmer. a missing file and a bad option were planned for, so the program words them for the person who ran it.

testing it

the tests sit beside the files they test and see the module as any importer does. count_test.kso checks the core:

import "std/expect"

sample = "The cat and the hat, and THE bat."

test_words_are_lowercased = words "The CAT" .> (ws -> ws == ["the" "cat"])

test_top_two = words sample
  .> (ws -> expect (render (top 2 ws)) . to (equal "   3  the\n   2  and"))

words answers an effect, so each test opens it with .> and answers the boolean from inside. the runner opens a test's effect the way the program's own executor would. test_top_two compares the rendered text, which is what a user sees. a test cannot build a ranked to compare against, because only the module that declares a type may construct one (error[opacity]), and the rendered text is the better thing to pin in any case.

no_number = "--top wants a number after it"

test_defaults = (options_of []).top == 5

test_top_and_path = options_of ["--top" "3" "notes.txt"]
  . (o -> o.top == 3 and o.path == "notes.txt")

test_top_needs_a_number = (options_of ["--top" "many"]).reason == no_number

a test may read a record's fields, so options_test.kso checks what the parser answered field by field. kanso test pointed at the project runs every test file in its modules:

test_words_are_lowercased ... ok
test_top_two ... ok
test_defaults ... ok
test_top_and_path ... ok
test_top_needs_a_number ... ok
5 passed, 0 failed

none of the five tests touches a file, standard input or an exit status. the logic that could be wrong is in count.kso and options.kso, where it is plain functions. cli.kso is a switchboard from outcomes to effects, short enough to check by reading. the runs above cover it, and every one of them is re-run against the program each time this book is built.

building a binary

kanso run compiles to native code and caches the result, which is the right tool while you work. to ship, build from inside the project with --release, which optimizes the whole program at once:

$ cd wordfreq
$ kanso build . --release
built ./wordfreq (llvm ir at wordfreq.ll)
$ ./wordfreq --top 3 ../tea.txt
   5  the
   4  tea
   3  and

the result is one executable. on linux x86-64 it is about 700 KB and links against nothing but the c library and the maths library, so it can be copied to another machine of the same kind and run there with no kanso installed. the .ll file beside it is the llvm ir the compiler wrote, kept for reading; nothing needs it at run time.

what you can do now

you can lay out a project as an entry and a module, and choose their names so the build and the entry rule stay out of each other's way. you can keep the logic in pure functions and the i/o in one file of edges. you can read a command line into a record, report a mistake as a value, and turn it into a message on standard error and an exit status. and you can test the core with kanso test, run the whole program on fixtures, and build a binary that stands on its own.

exercises

1. add a --min N option that leaves out words seen fewer than N times. give it a test in options_test.kso and one in count_test.kso, and make --min without a number a bad_usage with its own reason.
2. wordfreq one.txt two.txt is refused today. change options.kso so it accepts several paths, and cli.kso so it counts the words of all of them together. decide what a missing second file should do, and write the test before the code.
3. contractions such as don't count as one word because the pattern keeps '. find a text where that splits a word you think should be whole, or joins two you think should be apart, and change words. which test would have caught it?
4. build the release binary and run it under time on a large text. then run the same text through tr -cs "a-z'" '\n' | sort | uniq -c | sort -rn | head -5 after lowercasing it, and compare the two answers before you compare the two times.