Files, CSV and JSON

Read and write files, and convert CSV and JSON

Read text with •nget, parse it with •csv or •json, and calculate. •csv⁻¹ and •json⁻¹ produce text, which •nput writes to a file.

sales←•csv •nget "sales.csv"
totals←+/¨sales
"totals.json" •nput •json⁻¹ totals

Options go on the left as a keyed vector, and the data goes on the right. Option names are case-insensitive.

Files

]help •nget

•nget — Read a file

•nget path reads a UTF-8 text file. •nget "-" reads the rest of standard input. A path that starts with ./ or ../ is relative to the file holding the code, as for •load.

X •nget path takes options on the left: binary (1 reads a vector of byte values) and encoding ("UTF-8"). Files covers bytes and options.

Errors: IO for missing files, invalid UTF-8, other file errors, and standard input that fails or that the frontend doesn’t provide; DOMAIN for invalid options.

]help •nput

•nput — Write a file

path •nput data writes data to a new file. It returns the number of bytes written, which isn’t displayed, as with an assignment. Text is written as UTF-8. An array of numbers is written as bytes, each from 0 to 255, in ravel order. A path that starts with ./ or ../ is relative to the file holding the code, as for •load.

"-" •nput data writes data to standard output, with no line ending added. Text written this way just before a read of ⎕ is that read’s prompt. See ⎕.

X •nput data takes options on the left: path, overwrite (1 replaces an existing file) and encoding ("UTF-8", for text). With the option unique, •nput writes a new file with a unique name inside the directory path. It returns the new file’s path, which is displayed. The option prefix sets the start of the name, as for •mkdir. Plain text on the left is the path. Files covers bytes and options.

Errors: IO for an existing file without overwrite, and for other file errors; DOMAIN for invalid options or byte values, and for bytes written to standard output that aren’t UTF-8.

text←•nget "sales.csv"
"copy.csv" •nput text
["path":"copy.csv" "overwrite":1] •nput text

Directories must already exist. Binary reads return exact integers in 0…255. •nput writes an array of numbers as bytes, each a whole number in that range. It checks the data before opening the file.

bytes←["binary":1] •nget "image.bin"
"copy.bin" •nput bytes
"UTF-8" •ucs bytes           ⍝ decode UTF-8 when appropriate
bytes-256ₓ×bytes≥128ₓ         ⍝ signed-byte interpretation

Numeric input

]help •vfi

•vfi — Numeric input

•vfi text returns two vectors, [valid numbers], for the whitespace-separated fields of text. An invalid field has the flag 0ₓ and the value 0. Fields are parsed as numbers, never executed.

Fields take BPL’s written forms of numbers, with plain letters or subscripts. •vfi reads back the numbers that ⍕ displays:

•vfi ⍕1r3 1j2 1e20   ⍝ [[1 1 1]ₓ [1ᵣ3 1ⱼ2 1ₑ20]]

separators •vfi text splits on each character in separators instead. It trims whitespace around fields. An empty field is a valid 0. Numeric input covers it.

Errors: DOMAIN when either argument is not text.

Invalid fields have flag 0ₓ and value 0. Valid fields have flag 1ₓ:

[valid nums]←•vfi "12 nope -3 1.5"
valid
nums
valid#nums
[1 0 1 1]ₓ
12 0 ¯3 1.5
12 ¯3 1.5

A number written plainly, such as 2 or 1.5, is approximate. A number written with an x suffix, such as 3x, or as a rational, such as 1r4, is exact. •vfi also accepts the subscript forms that ⍕ displays, such as 1ᵣ4 and 1ₑ20, complex numbers written with j or J, ∞, inf, infinity, ASCII signs and signed exponents. nan gives $n. An invalid field gives the number 0 with the flag 0ₓ.

The numbers all become approximate when one of them is approximate and a float can represent every one exactly. Otherwise each number keeps its kind. The 0 from an invalid field doesn’t count as approximate.

1⊃•vfi "2 3x 1r4 1j-2 -1e-3 ∞"
2 3 0.25 1ⱼ¯2 ¯0.001 ∞

separators •vfi text splits on any supplied character and trims surrounding whitespace. Empty fields are valid zero. Internal whitespace remains part of the field.

',' •vfi "3.9,2.4,,76,"
'⋄' •vfi "1 ⋄ 2 3 ⋄ 4"
(1ₓ 1ₓ 1ₓ 1ₓ 1ₓ⋄3.9 2.4 0 76 0)
(1ₓ 0ₓ 1ₓ⋄1 0 4)

Empty input returns two empty vectors. With no separator characters ("" on the left), nonempty input is one field. DOMAIN: either argument is not character text.

JSON

]help •json

•json — JSON

•json text reads JSON5, which includes all JSON. JSON5 adds comments, trailing commas, single-quoted strings, unquoted keys, hexadecimal numbers, Infinity and NaN. Objects become keyed vectors. Arrays become vectors. Strings become character vectors. Integers of up to 128 bits stay exact. true and false become $t and $f.

•json⁻¹ Y writes JSON text. Keyed axes become objects. Unkeyed axes become arrays. Character vectors become strings. Keyed entries that hold functions are left out. NaN writes as null.

["fill":v] •json text reads null as v. The default is NaN. An integer array with a null stays exact. ["fill":v] •json⁻¹ Y writes v as null. JSON covers the conversions.

Errors: DOMAIN for malformed JSON, with its line and column, and for values that JSON can’t hold: an infinity that isn’t fill, out-of-range floats, nonintegral rationals, complex numbers and other functions.

Nested structure is kept:

person←•json "{""name"":""Ann"",""scores"":[10,20]}"
person.name
person.scores
Ann
[10 20]ₓ

•json also reads JSON5, with comments, trailing commas, single-quoted strings and unquoted keys.

•json "{name: 'Ann', scores: [10, 20,]}  // JSON5"
["name":"Ann" "scores":[10 20]ₓ]

"f.json" •nput •json⁻¹ Y writes JSON to a file. Scalar arrays export their contents. Axis names and empty-array prototypes are left out:

•json⁻¹ ["name":"Ann" "scores":[10 20]ₓ]
•json⁻¹ [1 2⋄3 4]ₓ
•json⁻¹ •json "{}"
{"name":"Ann","scores":[10,20]}
[[1,2],[3,4]]
{}

JSON integers are exact. Numbers with a decimal point or an exponent are approximate. An array’s numbers combine by the rule in Numeric input. true and false read as $t and $f. •json⁻¹ writes Booleans back as true and false:

•json "[9223372036854775809,1.5,true,false]"
•json⁻¹ •json "[true,false]"
9223372036854775809ₓ 1.5 $t $f
[true,false]

fill replaces null on import. A fill takes the array’s type wherever it keeps its value in that type, as ¯1 does here. With the default fill, NaN, an integer array with a null reads as floats. On export, NaN and the fill value become null:

•json "[1,null,3]"
["fill":¯1] •json "[1,null,3]"
["fill":∞] •json⁻¹ 1ₓ ∞ 3ₓ
1 $n 3
[1 ¯1 3]ₓ
[1,null,3]

Duplicate object members keep the last value.

CSV

]help •csv

•csv — CSV

•csv text parses CSV into a vector of columns. Headers become keys. Numeric columns become numbers. Missing numeric cells become NaN. An integer column with missing cells stays exact. Missing text cells become "".

•csv⁻¹ T writes CSV text for a vector of columns. Keys supply the header. Column lengths must agree. NaN writes as an empty cell, and so does the exact value given as fill.

Both directions take options on the left: header, separator, quotechar, doublequote, escapechar, decimal, thousands, trim and fill. Reading also takes text_columns, numeric_columns and missing, and writing takes forcequotes and lineending. "forcequotes":2 quotes every field. CSV describes the options.

Errors: DOMAIN for invalid options, duplicate headers, and cells that CSV can’t hold: nonintegral rationals, complex numbers, functions or nested cells; LENGTH for unequal record widths or columns.

Each numeric column uses compact integer or float storage where possible:

text←"price,qty
10.5,2
20.0,4"
T←•csv text
T
T.qty
+/¨T
["price":[10.5 20] "qty":[2 4]ₓ]
[2 4]ₓ
["price":30.5 "qty":6ₓ]

"f.csv" •nput •csv⁻¹ T writes CSV to a file. Unkeyed input writes the data without a header. Parsing the text restores the columns:

T←["price":10.5 20;"qty":2ₓ 4ₓ]
•csv •csv⁻¹ T
["price":[10.5 20] "qty":[2 4]ₓ]

CSV options

text←"price;qty
10,5;2
20,0;4"
opts←["separator":';' "decimal":',']
T←opts •csv text
T
csv←opts •csv⁻¹ T
opts •csv csv
["price":[10.5 20] "qty":[2 4]ₓ]
["price":[10.5 20] "qty":[2 4]ₓ]
Option Default Meaning
header Import: 1; export: has keys Read/write column names
separator ',' Field separator; use •ucs 9 for TSV
quotechar '"' Quote character; "" disables quoting
doublequote 1 Represent a quote inside a quoted field by doubling it
escapechar "" Escape character inside quoted fields
decimal '.' Decimal mark: '.' or ','
thousands "" Group separator; groups after the first contain three digits
trim 0 Trim surrounding whitespace from fields
fill Import: NaN; export: none Numeric missing-cell replacement. On export, NaN and this exact value write as an empty cell
missing ⍬ Additional missing-cell strings; empty cells are always missing
types none Import: column types, each given by a value of that type: a number for a numeric column, or text such as "" for a text column. A record names columns, as in ["id":0ₓ₈ "zip":""]. A vector gives every column’s type in order. Under static kinds, integer columns otherwise take the int width
forcequotes 0 Export: 0 as needed, 2 all fields
lineending •ucs 10 Export: LF or CRLF (•ucs 13 10)
text←"id,qty
00123,5"
T←["types":["id":""]] •csv text
T.id
["00123"]

Separator, quote and escape must be distinct ASCII characters other than CR, LF or NUL. Fields support Unicode, doubled quotes and embedded newlines. Import accepts LF, CRLF and CR record endings. Export with escapechar quotes every field and doubles literal escape characters. With quoting disabled, export errors if a field requires quoting.

Numbers and missing cells

Inference examines a whole column, ignoring missing cells. Integer fields produce exact numbers. Decimal/scientific notation produces floats. A column containing other text stays text. Quoted numbers participate in inference too.

T←•csv "count,label
2,001
3,yes"
T.count
T.label
[2 3]ₓ
"001" "yes"

A column of integers uses integer storage when every value fits in 64 bits. Larger integers stay exact. A value with a decimal point or an exponent is approximate. A column’s numbers combine by the rule in Numeric input.

Missing numeric cells become NaN, written $n. An integer column with a gap becomes floats. A column that also holds an integer too large for a float to hold exactly keeps each item’s kind, in mixed storage. Missing text cells become "". Entirely missing columns are text unless forced numeric.

T←•csv "price,qty
10.5,2
,4"
T.price
T.qty
10.5 $n
[2 4]ₓ

Configure extra missing markers and numeric fill explicitly.

text←"qty
NA
4"
["missing":"NA" "fill":¯1] •csv text
["qty":[¯1 4]ₓ]

Export uses - for negative numbers, decimal/scientific float spelling and plain exact integers. Infinity writes as inf or -inf. Set fill to export a chosen numeric sentinel as an empty field. Rationals that aren’t whole, complex numbers, functions and nested nonstring cells give DOMAIN.

Empty input returns an empty unkeyed vector. Header-only input returns keyed empty columns. Empty header names are allowed. Duplicate header names give DOMAIN. Unequal record widths or column lengths give LENGTH. Invalid forced numeric fields report their row and column.

Repeated text

When •csv, •json or Python data repeat a text, every cell that repeats it holds the same array. Each •csv column, •json document and Python conversion shares its first 65,536 distinct texts this way. A long column of repeated text then takes less memory, and classifying it, as =, ∪, ⍷ and Key do, compares only the distinct arrays.

Text made in BPL, such as the words that splitting a string gives, holds a separate array in each cell. Picking each word from the distinct words makes the repeats share one array:

words←' '(≠⊆⊢)"to be or not to be"
u←∪words
words←[u⍳words]⌷u
words