Epsilon

Table of Contents

Work in progress. Epsilon and this page are incomplete and subject to change.

Epsilon is the DUST expression engine. It evaluates DUST under CPU and memory limits. The epsilon command converts and formats documents, evaluates DUST, opens an interactive REPL, and serves nREPL clients.

See DUST for the language and the Function Index for its functions.

1. Command line ↑

Command Does
epsilon convert INPUT OUTPUT [--from FORMAT] [--to FORMAT] [--no-sort] Converts between formats or formats a document
epsilon convert --from FILES --to FILES [--no-sort] Converts every file a path or glob names
epsilon fmt [PATH...] [--write] [--check] [--from FORMAT \vert FILES] [--include GLOB] [--exclude GLOB] Formats DUST and RON files, keeping their key order
epsilon eval FILE Evaluates FILE form by form and writes each result as canonical DUST
epsilon repl [FILE...] Starts the REPL, loading each FILE as a namespace first
epsilon nrepl-server [options] Serves editor clients over TCP

convert supports .json, .edn, .tron, .ron, and .dust. The input and output extensions select the formats. Using the same extension for both files writes a normalized representation. The output file is written only after the input has been parsed and converted successfully.

Use - for standard input or standard output. Specify standard input's format with --from FORMAT and standard output's format with --to FORMAT. The format names are json, edn, tron, ron, and dust. These options can also override a file extension. Use --comment-roundtrip to preserve DUST and RON comment placements in JSON's $comment strings and interpret those strings when converting back. Without this option JSON $comment strings remain literal.

epsilon convert input.dust output.json
epsilon convert input.dust - --to json
cat input.edn | epsilon convert - output.dust --from edn
cat input.dust | epsilon convert - - --from dust --to json

--from and --to also take files. --from is then a path or a glob, and --to a path, or a directory followed by *.EXT or **/*.EXT. Each matching file is converted to the path it maps to: the --to directory, then, with **, the file's directories below the --from directory, then its name with the --to extension. Formats come from the extensions. Missing directories are created, an output that would not change is not rewritten, and two inputs that map to one output are an error before anything is written. A single --from file with a --to format name goes to standard output.

epsilon convert --from order.ron --to order.dust
epsilon convert --from 'data/**/*.json' --to 'out/**/*.dust'   # data/x/b.json -> out/x/b.dust
epsilon convert --from 'data/*.json' --to 'out/*.ron'          # top level of data only

In a glob, *, ?, and [...] match within one name and a ** name matches any number of names, none included. ** does not enter hidden directories, and hidden files match only a name pattern that starts with ..

Run epsilon convert -h for command help and examples.

DUST and RON output sorts map keys and the tags in a tag group, which gives the canonical form. --no-sort keeps both in source order instead. Either way a map's tag group comes first. A source that does not parse gives a #err diagnostic with its line and column on standard error and exit status 1.

fmt is convert from each file to its own format, dust or ron, with --no-sort, and its output ends with a newline, as editors save files: DUST output already does, and fmt adds one to RON output, which convert writes without one. A PATH is a file, a directory, or - for standard input (with --from dust or --from ron); with no PATH and no --from files, it is the current directory. A directory is searched recursively for files that match an --include glob (default *.dust and *.ron) and no --exclude glob. Hidden files and directories, build, and vendor are always skipped, and an excluded directory is not entered. A glob without / matches a name at any depth; one with / matches the path below the directory. --from also takes files, as for convert.

One file prints its result. --write (-w) rewrites each file whose result differs: it writes a temporary file in the same directory, gives it the file's permissions, and renames it over the file, so the file is never half written. Through a symbolic link it rewrites the target and keeps the link. --check writes nothing; it names each file formatting would change on standard error. Several files need --write or --check. A file that does not parse gives its path and a #err diagnostic on standard error, and the other files are still formatted. The exit status is 1 when any file failed or would change, and 2 for an invalid combination of options.

epsilon fmt order.dust
epsilon fmt --write order.dust
epsilon fmt --check                                 # the current directory
epsilon fmt --write src --exclude fixtures
epsilon fmt --check . --include '*.dust'
epsilon fmt --write --from 'src/**/*.dust'
cat order.dust | epsilon fmt - --from dust

For example, with tags and keys out of order, and with a root list:

Input epsilon fmt epsilon convert ... --to dust
{b 1 a 2 <z x>} {<z x> / = b 1= / = a 2}= {<x z> / = a 2= / = b 1}=
[1 2 7] 1, 2, 7 on three lines the same

EDN lists and sets become arrays, while keywords, symbols, and characters become text. Map keys become strings. Numeric ratios and decimal values become floating point numbers; integers outside the supported range are rejected. EDN #inst values are represented by #utc, and #uuid values become strings. Other EDN tags retain their tag and value as DUST tagged elements. DUST tags are written as EDN tagged elements when their names can be read as EDN tags; otherwise they are represented as maps. These mappings do not preserve all original EDN types. Only the first EDN form is read; an empty input reads as nil.

TRON encodes JSON-compatible values as a binary document. Floating-point values retain their numeric kind, including integral floats such as 1.0. This can differ from encoders that normalize integral floats to integers. A #b64 tagged value with valid base64 is represented as binary data and decoded back to that tag.

task repl builds the engine and starts the REPL from the repository.

Each REPL and each nREPL session may use 1 GiB of runtime memory, because it keeps its top-level locals and namespaces between evaluations; epsilon eval may use 64 MiB. An evaluation that needs more fails, and the garbage it leaves is collected before the next one. !reset returns the REPL to a fresh runtime.

1.1. nREPL server ↑

Start a server for an editor client:

epsilon nrepl-server
epsilon nrepl-server --port 7888
epsilon nrepl-server --bind 127.0.0.1 --port-file /tmp/epsilon.port

The default address is 127.0.0.1 and the default port is 0, which asks the operating system to choose a port. The server writes that port to .nrepl-port in the current directory, or to the path given by --port-file. On shutdown it removes the file only if it still contains this server's port. The protocol has no authentication; choose the bind address accordingly.

Messages use bencode. These examples show their contents as maps:

client -> {op "clone", id "1"}
server <- {id "1", new-session "s-1", status ["done"]}

client -> {op "eval", id "2", session "s-1", code "(+ 1 2)\n(+ 3 4)"}
server <- {id "2", session "s-1", value "3"}
server <- {id "2", session "s-1", value "7"}
server <- {id "2", session "s-1", status ["done"]}

load-file loads the DUST source in file into a session, as !load does for a file in the interactive REPL: the locals its export forms name become constants. The server does not read the disk, so an editor can send an unsaved buffer:

client -> {op "load-file", id "3", session "s-1", file "(local unit 7)\n(export unit)", file-path "lib/shapes.dust", file-name "shapes.dust"}
server <- {id "3", session "s-1", status ["done"]}
client -> {op "eval", id "4", session "s-1", code ":shapes/unit"}
server <- {id "4", session "s-1", value "7"}

The namespace is the file-name without its extension, or else the file-path without its directory and extension. Add ns "s" to the load request to use :s/unit as the name; a request without ns, file-name, or file-path is rejected. file-path also names the source in error responses. Reloading the same namespace replaces its bindings. lookup reports the name and type of a symbol from the session or the core libraries:

client -> {op "lookup", id "5", session "s-1", sym "shapes/unit"}
server <- {id "5", session "s-1", info {name "shapes/unit", type "integer"}, status ["done"]}

If the name is unknown, the response omits info.

describe maps each operation name to a descriptor. The descriptors are empty unless the request has verbose?, which adds each operation's doc and, for eval, its requires, optional, and returns fields:

client -> {op "describe", id "6"}
server <- {id "6", ops {clone {}, close {}, describe {}, eval {}, interrupt {}, load-file {}, lookup {}, stdin {}}, status ["done"]}

clone starts a fresh session; copying a session is unsupported. eval returns each top-level form's result separately, as one DUST form: a list keeps its brackets, so [1 2 (+ 3 4)] returns [1 2 7] and [] returns [], and map keys are in canonical order. A session keeps the top-level locals that its evaluations define. A returned #err is a DUST value: it arrives in value, with file, line, and column for its source. Parse, compile, and runtime failures arrive in ex. interrupt stops an evaluation at a checkpoint and leaves the session usable. Completed mutations are not rolled back. Run epsilon nrepl-server -h for the options and examples.

eval writes the result to standard output and exits with 0. A failure is written to standard error as a #err element and exits with 1:

$ echo '(/ 1 0)' > zero.dust
$ epsilon eval zero.dust
#err {<division integer zero>
      column 1
      line 1
      message "division by zero"}

The tag group classifies the failure; the other members are diagnostics. See error tags for every tag set.

2. Formatting ↑

Converting a DUST document to DUST writes its canonical spelling: the same value always formats to the same bytes. For example, epsilon convert input.dust output.dust writes canonical DUST to output.dust. epsilon fmt keeps map keys in source order, so its output is canonical only when the keys are sorted. Canonical output is what Epsilon prints, what conformance fixtures store, and what hash digests, so two programs that format alike are the same program.

The rules are those of RON and DUST. In brief, the formatter:

  • sorts map keys and collects boolean members into a tag group;
  • uses the short forms .name, :name, (f x), `x, and \x for locals, constants, invocations, quotes, and muted forms;
  • drops the brackets of a root list with two or more forms, writing one form per line;
  • normalizes #utc and #dur payloads, so #dur PT90M becomes #dur PT1H30M.

3. REPL ↑

The REPL reads an entry, evaluates each of its top-level forms on its own, and prints each result as one canonical DUST form. A list keeps its brackets, so [1 2 (+ 3 4)] prints [1 2 7]. It keeps history and highlights input as you type. A form that spans lines continues until its brackets balance.

> (+ 1 2)
   compile 77.9µs · 143.1KiB arena · key 5d902e22 · 1/64 cached
3  eval 57.9µs cpu · 130.5KiB peak · +4.6KiB net · 323.5KiB live
> (+ 1 2)
3  eval 17.3µs cpu · 128.5KiB peak · +2.5KiB net · 326.0KiB live

A local form defines a top-level local that later entries read:

> (local rate 3)
   compile 47.9µs · 129.0KiB arena · key 64c24dfd · 2/64 cached
.rate  eval 10.6µs cpu · 5.7KiB peak · +3.3KiB net · 580.8KiB live
> [.rate (* .rate 2)]
   compile 81.8µs · 130.3KiB arena · key eb0abee0 · 3/64 cached
[3 6]  eval 8.1µs cpu · 8.4KiB peak · +5.5KiB net · 586.3KiB live

3.1. Loading namespaces ↑

!load PATH [ALIAS] evaluates a file and binds each local its export forms name under a namespace as a constant. The file's other locals stay private. PATH is relative to the working directory and the .dust extension is implied. The namespace is ALIAS, or else the last path segment. Loading a namespace again replaces its bindings. A namespace whose names match library entries, such as math/pi, is rejected with #err <conflict name>; load it under an alias instead, as in !load lib/math m. A line that starts with ! is a REPL command; every other line is DUST, so :shapes/square is a constant reference.

Given lib/shapes.dust:

(local double (fn [x] (+ .x .x)))
(local square (fn [x] (* .x .x)))
(export square double)
> !load lib/shapes
shapes: square double
> (shapes/square 7)
49  eval 140.8µs cpu · 116.8KiB peak · -128.6KiB net · 209.7KiB live
> !load lib/shapes s
s: square double
> (s/double 4.5)
9.0  eval 17.7µs cpu · 130.5KiB peak · +5.8KiB net · 228.0KiB live

What a library path names is up to each runtime. The REPL reads files; Stardust resolves the same paths within its database.

!reset drops every top-level local, loaded namespace, and compiled entry and starts a fresh runtime with only the core libraries:

> !load lib/shapes
shapes: square double
> !reset
reset: locals, namespaces, and compiled programs cleared
> (shapes/square 7)
#err {<expected function>
      message "expected a function value"}  eval 28.3µs cpu · 144.4KiB peak · -136.6KiB net · 539.7KiB live

4. Costs ↑

Every evaluation reports its cost, dimmed, to the right of the result. A form that had to be compiled first reports the compile on its own line above.

4.1. Evaluation ↑

eval 57.9µs cpu · 130.5KiB peak · +4.6KiB net · 323.5KiB live

Field Meaning
cpu CPU time spent running the compiled program
peak The peak heap growth during evaluation
net How much larger (+) or smaller (-) the heap is afterwards; a negative value means a collection ran
live The heap size after evaluation, including loaded namespaces

cpu and memory are the quantities the engine's limits bound: a program that exceeds either fails with #err <limit resource>. Costs vary from run to run, so compare them by order of magnitude rather than exactly.

Epsilon's native regex compiler currently limits a pattern to 4,096 UTF-8 bytes, 256 compiled states, and 16 capture groups. Each bound of a counted repetition is limited to 64. Exceeding compiler capacity reports a resource limit. A search also has a work limit, in addition to the runtime's CPU and memory limits. The compiled-pattern cache holds up to 64 entries per runtime.

4.2. Compilation ↑

compile 77.9µs · 143.1KiB arena · key 5d902e22 · 1/64 cached

Field Meaning
first value Time to compile and prepare the program
arena Memory the cached program keeps
key The start of the program's cache key, a hash of its canonical DUST
1/64 cached Programs now cached, of the cache's capacity

The REPL caches compiled programs, keyed by the hash of their canonical DUST, and evicts the least recently used program when the cache is full. Because the key is canonical, (+ 1 2) and (+ 1 2) share one program, and repeating a form prints no compile line. Cached programs live in their own arenas, separate from the memory each evaluation uses.