Epsilon
Table of Contents
Work in progress. Epsilon and this page are incomplete and subject to change.
Epsilon is the DUST expression engine. It evaluates DUST under CPU and memory
limits. The epsilon command converts and formats documents, evaluates DUST,
opens an interactive REPL, and serves nREPL clients.
See DUST for the language and the Function Index for its functions.
1. Command line ↑
| Command | Does |
|---|---|
epsilon convert INPUT OUTPUT [--from FORMAT] [--to FORMAT] [--no-sort] |
Converts between formats or formats a document |
epsilon convert --from FILES --to FILES [--no-sort] |
Converts every file a path or glob names |
epsilon fmt [PATH...] [--write] [--check] [--from FORMAT \vert FILES] [--include GLOB] [--exclude GLOB] |
Formats DUST and RON files, keeping their key order |
epsilon eval FILE |
Evaluates FILE form by form and writes each result as canonical DUST |
epsilon repl [FILE...] |
Starts the REPL, loading each FILE as a namespace first |
epsilon nrepl-server [options] |
Serves editor clients over TCP |
convert supports .json, .edn, .tron, .ron, and .dust. The
input and output extensions select the formats. Using the same extension for
both files writes a normalized representation. The output file is written only
after the input has been parsed and converted successfully.
Use - for standard input or standard output. Specify standard input's format
with --from FORMAT and standard output's format with --to FORMAT. The
format names are json, edn, tron, ron, and dust. These options can
also override a file extension. Use --comment-roundtrip to preserve DUST and
RON comment placements in JSON's $comment strings and interpret those strings
when converting back. Without this option JSON $comment strings remain literal.
epsilon convert input.dust output.json
epsilon convert input.dust - --to json
cat input.edn | epsilon convert - output.dust --from edn
cat input.dust | epsilon convert - - --from dust --to json
--from and --to also take files. --from is then a path or a glob, and
--to a path, or a directory followed by *.EXT or **/*.EXT. Each matching
file is converted to the path it maps to: the --to directory, then, with
**, the file's directories below the --from directory, then its name with
the --to extension. Formats come from the extensions. Missing directories are
created, an output that would not change is not rewritten, and two inputs that
map to one output are an error before anything is written. A single --from
file with a --to format name goes to standard output.
epsilon convert --from order.ron --to order.dust
epsilon convert --from 'data/**/*.json' --to 'out/**/*.dust' # data/x/b.json -> out/x/b.dust
epsilon convert --from 'data/*.json' --to 'out/*.ron' # top level of data only
In a glob, *, ?, and [...] match within one name and a ** name matches
any number of names, none included. ** does not enter hidden directories,
and hidden files match only a name pattern that starts with ..
Run epsilon convert -h for command help and examples.
DUST and RON output sorts map keys and the tags in a tag group, which gives
the canonical form. --no-sort keeps both in source order instead. Either way
a map's tag group comes first. A source that does not parse gives a
#err diagnostic with its line and column on standard error and exit status 1.
fmt is convert from each file to its own format, dust or ron, with
--no-sort, and its output ends with a newline, as editors save files: DUST
output already does, and fmt adds one to RON output, which convert writes
without one. A PATH is a file, a directory, or - for standard input (with
--from dust or --from ron); with no PATH and no --from files, it is the
current directory. A directory is searched recursively for files that match an
--include glob (default *.dust and *.ron) and no --exclude glob.
Hidden files and directories, build, and vendor are always skipped, and an
excluded directory is not entered. A glob without / matches a name at any
depth; one with / matches the path below the directory. --from also takes
files, as for convert.
One file prints its result. --write (-w) rewrites each file whose result
differs: it writes a temporary file in the same directory, gives it the file's
permissions, and renames it over the file, so the file is never half written.
Through a symbolic link it rewrites the target and keeps the link. --check
writes nothing; it names each file formatting would change on standard error.
Several files need --write or --check. A file that does not parse gives
its path and a #err diagnostic on standard error, and the other files are
still formatted. The exit status is 1 when any file failed or would change, and
2 for an invalid combination of options.
epsilon fmt order.dust
epsilon fmt --write order.dust
epsilon fmt --check # the current directory
epsilon fmt --write src --exclude fixtures
epsilon fmt --check . --include '*.dust'
epsilon fmt --write --from 'src/**/*.dust'
cat order.dust | epsilon fmt - --from dust
For example, with tags and keys out of order, and with a root list:
| Input | epsilon fmt |
epsilon convert ... --to dust |
|---|---|---|
{b 1 a 2 <z x>} |
{<z x> / = b 1= / = a 2}= |
{<x z> / = a 2= / = b 1}= |
[1 2 7] |
1, 2, 7 on three lines |
the same |
EDN lists and sets become arrays, while keywords, symbols, and characters become
text. Map keys become strings. Numeric ratios and decimal values become floating
point numbers; integers outside the supported range are rejected. EDN #inst
values are represented by #utc, and #uuid values become strings. Other EDN
tags retain their tag and value as DUST tagged elements. DUST tags are written
as EDN tagged elements when their names can be read as EDN tags; otherwise they
are represented as maps. These mappings do not preserve all original EDN types.
Only the first EDN form is read; an empty input reads as nil.
TRON encodes JSON-compatible values as a binary document. Floating-point
values retain their numeric kind, including integral floats such as 1.0.
This can differ from encoders that normalize integral floats to integers. A
#b64 tagged value with valid base64 is represented as binary data and decoded
back to that tag.
task repl builds the engine and starts the REPL from the repository.
Each REPL and each nREPL session may use 1 GiB of runtime memory, because
it keeps its top-level locals and namespaces between evaluations; epsilon eval may use 64 MiB. An
evaluation that needs more fails, and the garbage it leaves is collected
before the next one. !reset returns the REPL to a fresh runtime.
1.1. nREPL server ↑
Start a server for an editor client:
epsilon nrepl-server
epsilon nrepl-server --port 7888
epsilon nrepl-server --bind 127.0.0.1 --port-file /tmp/epsilon.port
The default address is 127.0.0.1 and the default port is 0, which asks
the operating system to choose a port. The server writes that port to
.nrepl-port in the current directory, or to the path given by --port-file.
On shutdown it removes the file only if it still contains this server's port.
The protocol has no authentication; choose the bind address accordingly.
Messages use bencode. These examples show their contents as maps:
client -> {op "clone", id "1"}
server <- {id "1", new-session "s-1", status ["done"]}
client -> {op "eval", id "2", session "s-1", code "(+ 1 2)\n(+ 3 4)"}
server <- {id "2", session "s-1", value "3"}
server <- {id "2", session "s-1", value "7"}
server <- {id "2", session "s-1", status ["done"]}
load-file loads the DUST source in file into a session, as !load does
for a file in the interactive REPL: the locals its export forms name
become constants. The server does not read the disk, so an editor can send
an unsaved buffer:
client -> {op "load-file", id "3", session "s-1", file "(local unit 7)\n(export unit)", file-path "lib/shapes.dust", file-name "shapes.dust"}
server <- {id "3", session "s-1", status ["done"]}
client -> {op "eval", id "4", session "s-1", code ":shapes/unit"}
server <- {id "4", session "s-1", value "7"}
The namespace is the file-name without its extension, or else the
file-path without its directory and extension. Add ns "s" to the load
request to use :s/unit as the name; a request without ns, file-name, or
file-path is rejected. file-path also names the source in error responses.
Reloading the same namespace replaces its bindings. lookup reports the name
and type of a symbol from the session or the core libraries:
client -> {op "lookup", id "5", session "s-1", sym "shapes/unit"}
server <- {id "5", session "s-1", info {name "shapes/unit", type "integer"}, status ["done"]}
If the name is unknown, the response omits info.
describe maps each operation name to a descriptor. The descriptors are
empty unless the request has verbose?, which adds each operation's doc
and, for eval, its requires, optional, and returns fields:
client -> {op "describe", id "6"}
server <- {id "6", ops {clone {}, close {}, describe {}, eval {}, interrupt {}, load-file {}, lookup {}, stdin {}}, status ["done"]}
clone starts a fresh session; copying a session is unsupported. eval
returns each top-level form's result separately, as one DUST form: a list
keeps its brackets, so [1 2 (+ 3 4)] returns [1 2 7] and [] returns
[], and map keys are in canonical order. A session keeps the
top-level locals that its evaluations define. A returned #err is a DUST value: it
arrives in value, with file, line, and column for its source. Parse,
compile, and runtime failures arrive in ex. interrupt stops an evaluation
at a checkpoint and leaves the session usable. Completed mutations are not
rolled back. Run epsilon nrepl-server -h for the options and examples.
eval writes the result to standard output and exits with 0. A failure is
written to standard error as a #err element and exits with 1:
$ echo '(/ 1 0)' > zero.dust
$ epsilon eval zero.dust
#err {<division integer zero>
column 1
line 1
message "division by zero"}
The tag group classifies the failure; the other members are diagnostics. See error tags for every tag set.
2. Formatting ↑
Converting a DUST document to DUST writes its canonical spelling: the same value
always formats to the same bytes. For example, epsilon convert input.dust
output.dust writes canonical DUST to output.dust. epsilon fmt keeps map
keys in source order, so its output is canonical only when the keys are sorted. Canonical output is what Epsilon prints,
what conformance fixtures store, and what hash digests, so two
programs that format alike are the same program.
The rules are those of RON and DUST. In brief, the formatter:
- sorts map keys and collects boolean members into a tag group;
- uses the short forms
.name,:name,(f x),`x, and\xfor locals, constants, invocations, quotes, and muted forms; - drops the brackets of a root list with two or more forms, writing one form per line;
- normalizes
#utcand#durpayloads, so#dur PT90Mbecomes#dur PT1H30M.
3. REPL ↑
The REPL reads an entry, evaluates each of its top-level forms on its own,
and prints each result as one canonical DUST form. A list keeps its brackets,
so [1 2 (+ 3 4)] prints [1 2 7]. It keeps history and highlights input as
you type. A form that spans lines continues until its brackets balance.
> (+ 1 2) compile 77.9µs · 143.1KiB arena · key 5d902e22 · 1/64 cached 3 eval 57.9µs cpu · 130.5KiB peak · +4.6KiB net · 323.5KiB live > (+ 1 2) 3 eval 17.3µs cpu · 128.5KiB peak · +2.5KiB net · 326.0KiB live
A local form defines a top-level local that later entries read:
> (local rate 3) compile 47.9µs · 129.0KiB arena · key 64c24dfd · 2/64 cached .rate eval 10.6µs cpu · 5.7KiB peak · +3.3KiB net · 580.8KiB live > [.rate (* .rate 2)] compile 81.8µs · 130.3KiB arena · key eb0abee0 · 3/64 cached [3 6] eval 8.1µs cpu · 8.4KiB peak · +5.5KiB net · 586.3KiB live
3.1. Loading namespaces ↑
!load PATH [ALIAS] evaluates a file and binds each local its
export forms name under a namespace as a constant. The file's
other locals stay private. PATH is relative to the working directory and the
.dust extension is implied. The namespace is ALIAS, or else the last path
segment. Loading a namespace again replaces its bindings. A namespace whose
names match library entries, such as math/pi, is rejected with
#err <conflict name>; load it under an alias instead, as in
!load lib/math m. A line that starts
with ! is a REPL command; every other line is DUST, so :shapes/square is
a constant reference.
Given lib/shapes.dust:
(local double (fn [x] (+ .x .x)))
(local square (fn [x] (* .x .x)))
(export square double)
> !load lib/shapes shapes: square double > (shapes/square 7) 49 eval 140.8µs cpu · 116.8KiB peak · -128.6KiB net · 209.7KiB live > !load lib/shapes s s: square double > (s/double 4.5) 9.0 eval 17.7µs cpu · 130.5KiB peak · +5.8KiB net · 228.0KiB live
What a library path names is up to each runtime. The REPL reads files; Stardust resolves the same paths within its database.
!reset drops every top-level local, loaded namespace, and compiled entry
and starts a fresh runtime with only the core libraries:
> !load lib/shapes
shapes: square double
> !reset
reset: locals, namespaces, and compiled programs cleared
> (shapes/square 7)
#err {<expected function>
message "expected a function value"} eval 28.3µs cpu · 144.4KiB peak · -136.6KiB net · 539.7KiB live
4. Costs ↑
Every evaluation reports its cost, dimmed, to the right of the result. A form that had to be compiled first reports the compile on its own line above.
4.1. Evaluation ↑
eval 57.9µs cpu · 130.5KiB peak · +4.6KiB net · 323.5KiB live
| Field | Meaning |
|---|---|
cpu |
CPU time spent running the compiled program |
peak |
The peak heap growth during evaluation |
net |
How much larger (+) or smaller (-) the heap is afterwards; a negative value means a collection ran |
live |
The heap size after evaluation, including loaded namespaces |
cpu and memory are the quantities the engine's limits bound: a program that
exceeds either fails with #err <limit resource>. Costs vary from run to run,
so compare them by order of magnitude rather than exactly.
Epsilon's native regex compiler currently limits a pattern to 4,096 UTF-8 bytes, 256 compiled states, and 16 capture groups. Each bound of a counted repetition is limited to 64. Exceeding compiler capacity reports a resource limit. A search also has a work limit, in addition to the runtime's CPU and memory limits. The compiled-pattern cache holds up to 64 entries per runtime.
4.2. Compilation ↑
compile 77.9µs · 143.1KiB arena · key 5d902e22 · 1/64 cached
| Field | Meaning |
|---|---|
| first value | Time to compile and prepare the program |
arena |
Memory the cached program keeps |
key |
The start of the program's cache key, a hash of its canonical DUST |
1/64 cached |
Programs now cached, of the cache's capacity |
The REPL caches compiled programs, keyed by the hash of their canonical DUST,
and evicts the least recently used program when the cache is full. Because the
key is canonical, (+ 1 2) and (+ 1 2) share one program, and repeating a
form prints no compile line. Cached programs live in their own arenas,
separate from the memory each evaluation uses.