TOON Format for Rust

Token-Oriented Object Notation (TOON) is a compact, human-readable format designed for passing structured data to Large Language Models with significantly reduced token usage.

This crate provides the official, spec-compliant Rust implementation of TOON v2.0 with v1.5 optional features, offering both a library (toon-format) and a full-featured command-line tool (toon).

Quick Example

JSON (16 tokens, 40 bytes):

{
  "users": [
    { "id": 1, "name": "Alice" },
    { "id": 2, "name": "Bob" }
  ]
}

TOON (13 tokens, 28 bytes) - 18.75% token savings:

users[2]{id,name}:
  1,Alice
  2,Bob

Features

Generic API: Works with any Serialize/Deserialize type - custom structs, enums, JSON values, and more
Spec-Compliant: Fully compliant with TOON Specification v2.0
v1.5 Optional Features: Key folding and path expansion
Safe & Performant: Built with safe, fast Rust
Powerful CLI: Full-featured command-line tool
Strict Validation: Enforces all spec rules (configurable)
Well-Tested: Comprehensive test suite with unit tests, spec fixtures, and real-world scenarios

Installation

As a Library

cargo add toon-format

As a CLI Tool

cargo install toon-format

Library Usage

Basic Encode & Decode

The encode and decode functions work with any type implementing Serialize/Deserialize:

With custom structs:

use serde::{Serialize, Deserialize};
use toon_format::{encode_default, decode_default};

#[derive(Serialize, Deserialize, Debug, PartialEq)]
struct User {
    name: String,
    age: u32,
    email: String,
}

fn main() -> Result<(), toon_format::ToonError> {
    let user = User {
        name: "Alice".to_string(),
        age: 30,
        email: "alice@example.com".to_string(),
    };

    // Encode to TOON
    let toon = encode_default(&user)?;
    println!("{}", toon);
    // Output:
    // name: Alice
    // age: 30
    // email: alice@example.com

    // Decode back to struct
    let decoded: User = decode_default(&toon)?;
    assert_eq!(user, decoded);

    Ok(())
}

With JSON values:

use serde_json::{json, Value};
use toon_format::{encode_default, decode_default};

fn main() -> Result<(), toon_format::ToonError> {
    let data = json!({
        "users": [
            {"id": 1, "name": "Alice"},
            {"id": 2, "name": "Bob"}
        ]
    });

    // Encode to TOON
    let toon_str = encode_default(&data)?;
    println!("{}", toon_str);
    // Output:
    // users[2]{id,name}:
    //   1,Alice
    //   2,Bob

    // Decode back to JSON
    let decoded: Value = decode_default(&toon_str)?;
    assert_eq!(decoded, data);
    
    Ok(())
}

API Reference

Encoding

`encode<T: Serialize>(&value, &options) -> Result<String, ToonError>`

Encode any serializable type to TOON format. Works with custom structs, enums, collections, and serde_json::Value.

use toon_format::{encode, EncodeOptions, Delimiter, Indent};
use serde_json::json;

let data = json!({"items": ["a", "b", "c"]});

// Default encoding
let toon = encode(&data, &EncodeOptions::default())?;
// items[3]: a,b,c

// Custom delimiter
let opts = EncodeOptions::new()
    .with_delimiter(Delimiter::Pipe);
let toon = encode(&data, &opts)?;
// items[3|]: a|b|c

// Custom indentation
let opts = EncodeOptions::new()
    .with_indent(Indent::Spaces(4));
let toon = encode(&data, &opts)?;

`EncodeOptions`

Method	Description	Default
`with_delimiter(d)`	Set delimiter: `Comma`, `Tab`, or `Pipe`	`Comma`
`with_indent(i)`	Set indentation (spaces only)	`Spaces(2)`
`with_spaces(n)`	Shorthand for `Indent::Spaces(n)`	`2`
`with_key_folding(mode)`	Enable key folding (v1.5)	`Off`
`with_flatten_depth(n)`	Set max folding depth	`usize::MAX`

Decoding

`decode<T: Deserialize>(&input, &options) -> Result<T, ToonError>`

Decode TOON format into any deserializable type. Works with custom structs, enums, collections, and serde_json::Value.

With custom structs:

use serde::Deserialize;
use toon_format::{decode, DecodeOptions};

#[derive(Deserialize)]
struct Config {
    host: String,
    port: u16,
}

let toon = "host: localhost\nport: 8080";
let config: Config = decode(toon, &DecodeOptions::default())?;

With JSON values:

use serde_json::Value;
use toon_format::{decode, DecodeOptions};

let toon = "name: Alice\nage: 30";

// Default (strict) decode
let json: Value = decode(toon, &DecodeOptions::default())?;

// Non-strict mode (relaxed validation)
let opts = DecodeOptions::new().with_strict(false);
let json: Value = decode(toon, &opts)?;

// Disable type coercion
let opts = DecodeOptions::new().with_coerce_types(false);
let json: Value = decode("active: true", &opts)?;
// With coercion: {"active": true}
// Without: {"active": "true"}

Helper functions:

encode_default<T>(&value) - Encode with default options
decode_default<T>(&input) - Decode with default options

`DecodeOptions`

Method	Description	Default
`with_strict(b)`	Enable strict validation	`true`
`with_coerce_types(b)`	Auto-convert strings to types	`true`
`with_expand_paths(mode)`	Enable path expansion (v1.5)	`Off`

v1.5 Features

Key Folding (Encoder)

New in v1.5: Collapse single-key object chains into dotted paths to reduce tokens.

Standard nesting:

data:
  metadata:
    items[2]: a,b

With key folding:

data.metadata.items[2]: a,b

Example:

use serde_json::json;
use toon_format::{encode, EncodeOptions, KeyFoldingMode};

let data = json!({
    "data": {
        "metadata": {
            "items": ["a", "b"]
        }
    }
});

// Enable key folding
let opts = EncodeOptions::new()
    .with_key_folding(KeyFoldingMode::Safe);

let toon = encode(&data, &opts)?;
// Output: data.metadata.items[2]: a,b

With Depth Control

let data = json!({"a": {"b": {"c": {"d": 1}}}});

// Fold only 2 levels
let opts = EncodeOptions::new()
    .with_key_folding(KeyFoldingMode::Safe)
    .with_flatten_depth(2);

let toon = encode(&data, &opts)?;
// Output:
// a.b:
//   c:
//     d: 1

Safety Features

Key folding only applies when:

All segments are valid identifiers (a-z, A-Z, 0-9, _)
Each level contains exactly one key
No collision with sibling literal keys
Within the specified flatten_depth

Keys like full-name, user.email (if quoted), or numeric keys won't be folded.

Path Expansion (Decoder)

New in v1.5: Automatically expand dotted keys into nested objects.

Compact input:

a.b.c: 1
a.b.d: 2
a.e: 3

Expanded output:

{
  "a": {
    "b": {
      "c": 1,
      "d": 2
    },
    "e": 3
  }
}

Example:

use serde_json::Value;
use toon_format::{decode, DecodeOptions, PathExpansionMode};

let toon = "a.b.c: 1\na.b.d: 2";

// Enable path expansion
let opts = DecodeOptions::new()
    .with_expand_paths(PathExpansionMode::Safe);

let json: Value = decode(toon, &opts)?;
// {"a": {"b": {"c": 1, "d": 2}}}

Round-Trip Example:

use serde_json::{json, Value};
use toon_format::{encode, decode, EncodeOptions, DecodeOptions, KeyFoldingMode, PathExpansionMode};

let original = json!({
    "user": {
        "profile": {
            "name": "Alice"
        }
    }
});

// Encode with folding
let encode_opts = EncodeOptions::new()
    .with_key_folding(KeyFoldingMode::Safe);
let toon = encode(&original, &encode_opts)?;
// Output: "user.profile.name: Alice"

// Decode with expansion
let decode_opts = DecodeOptions::new()
    .with_expand_paths(PathExpansionMode::Safe);
let restored: Value = decode(&toon, &decode_opts)?;

assert_eq!(restored, original); // Perfect round-trip!

Quoted Keys Remain Literal:

use serde_json::Value;
use toon_format::{decode, DecodeOptions, PathExpansionMode};

let toon = r#"a.b: 1
"c.d": 2"#;

let opts = DecodeOptions::new()
    .with_expand_paths(PathExpansionMode::Safe);
let json: Value = decode(toon, &opts)?;
// {
//   "a": {"b": 1},
//   "c.d": 2        <- quoted key preserved
// }

Interactive TUI

TOON includes a full-featured Terminal User Interface for interactive conversions!

# Launch interactive mode
toon --interactive
# or
toon -i

Features:

Real-time conversion as you type
Live statistics (tokens, bytes, savings)
Interactive settings - adjust all options on-the-fly
File browser with visual navigation
Side-by-side diff viewer
Conversion history tracking
File operations (open, save, new)
Clipboard integration (copy/paste)
REPL mode for command-line interaction
Round-trip testing
Theme support (Dark/Light)
Built-in help with keyboard shortcuts

Perfect for:

Learning TOON format interactively
Testing conversions in real-time
Experimenting with different settings
Visual before/after comparisons
Quick data transformations

See docs/TUI.md for complete documentation and keyboard shortcuts!

CLI Usage

Basic Commands

# Auto-detect from extension
toon data.json        # Encode
toon data.toon        # Decode

# Force mode
toon -e data.txt      # Force encode
toon -d output.txt    # Force decode

# Pipe from stdin
cat data.json | toon
echo '{"name": "Alice"}' | toon -e

Encode Options

# Custom delimiter
toon data.json --delimiter pipe
toon data.json --delimiter tab

# Custom indentation
toon data.json --indent 4

# Key folding (v1.5)
toon data.json --fold-keys
toon data.json --fold-keys --flatten-depth 2

# Show statistics
toon data.json --stats

Decode Options

# Pretty-print JSON
toon data.toon --json-indent 2

# Relaxed validation
toon data.toon --no-strict

# Disable type coercion
toon data.toon --no-coerce

# Path expansion (v1.5)
toon data.toon --expand-paths

Full Example

$ echo '{"data":{"meta":{"items":["x","y"]}}}' | toon --fold-keys --stats

data.meta.items[2]: x,y

Stats:
+--------------+------+------+---------+
| Metric       | JSON | TOON | Savings |
+======================================+
| Tokens       | 13   | 8    | 38.46%  |
|--------------+------+------+---------|
| Size (bytes) | 38   | 23   | 39.47%  |
+--------------+------+------+---------+

Testing

The library includes a comprehensive test suite covering core functionality, edge cases, spec compliance, and real-world scenarios.

# Run all tests
cargo test

# Run specific test suites
cargo test --test spec_fixtures
cargo test --lib

# With output
cargo test -- --nocapture

Error Handling

All operations return Result<T, ToonError> with descriptive error messages:

use serde_json::Value;
use toon_format::{decode_strict, ToonError};

match decode_strict::<Value>("items[3]: a,b") {
    Ok(value) => println!("Success: {:?}", value),
    Err(ToonError::LengthMismatch { expected, found, .. }) => {
        eprintln!("Array length mismatch: expected {}, found {}", expected, found);
    }
    Err(e) => eprintln!("Error: {}", e),
}

Error Types

ParseError - Syntax errors with line/column info
LengthMismatch - Array length doesn't match header
TypeMismatch - Unexpected value type
InvalidStructure - Malformed TOON structure
SerializationError / DeserializationError - Conversion failures

Examples

Run with cargo run --example examples to see all examples:

structs.rs - Custom struct serialization
tabular.rs - Tabular array formatting
arrays.rs - Various array formats
arrays_of_arrays.rs - Nested arrays
objects.rs - Object encoding
mixed_arrays.rs - Mixed-type arrays
delimiters.rs - Custom delimiters
round_trip.rs - Encode/decode round-trips
decode_strict.rs - Strict validation
empty_and_root.rs - Edge cases

Resources

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

Development

# Clone the repository
git clone https://github.com/your-org/toon-rust.git
cd toon-rust

# Run tests
cargo test --all

# Run lints
cargo clippy -- -D warnings

# Format code
cargo fmt

# Build docs
cargo doc --open

Name		Name	Last commit message	Last commit date
Latest commit History 53 Commits
.github		.github
docs		docs
examples		examples
spec @ c09f73b		spec @ c09f73b
src		src
tests		tests
.gitignore		.gitignore
.gitmodules		.gitmodules
CHANGELOG.md		CHANGELOG.md
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
Cargo.toml		Cargo.toml
LICENSE		LICENSE
README.md		README.md
rustfmt.toml		rustfmt.toml

License

toon-format/toon-rust

Folders and files

Latest commit

History

Repository files navigation

TOON Format for Rust

Quick Example

Features

Installation

As a Library

As a CLI Tool

Library Usage

Basic Encode & Decode

API Reference

Encoding

encode<T: Serialize>(&value, &options) -> Result<String, ToonError>

EncodeOptions

Decoding

decode<T: Deserialize>(&input, &options) -> Result<T, ToonError>

DecodeOptions

v1.5 Features

Key Folding (Encoder)

With Depth Control

Safety Features

Path Expansion (Decoder)

Interactive TUI

Features:

CLI Usage

Basic Commands

Encode Options

Decode Options

Full Example

Testing

Error Handling

Error Types

Examples

Resources

Contributing

Development

License

About

Resources

License

Code of conduct

Contributing

Uh oh!

Stars

Watchers

Forks

Releases 14

Packages 0

Contributors 4

Languages

`encode<T: Serialize>(&value, &options) -> Result<String, ToonError>`

`EncodeOptions`

`decode<T: Deserialize>(&input, &options) -> Result<T, ToonError>`

`DecodeOptions`

Packages