User Manual

Introduction

Basic Concept

Installation

Tutorial (jsvine link or inline?)

The Design of VisiData: Foundations

Design

Culture this has grown out of

- textpunk connection
- vim
- nethack
- VisiCalc

Design goals

- lightweight, portable
- universal interface
- for hackers and data engineers
- easy workflow customization, experimentation, and prototyping

Bold choices

- terminal interface
- horizontal style (commands are one-liners; greppable)
- minimal code for maximal power
- Python

Architecture

The Core Principle: Columns are Functions of Rows

- Rows are opaque objects, uniquely referencing their row, containing the data in the row,  at least an index into ome other data structure.
- Columns are functions, which know how to extract (or set) a value, given the sheet and the row..
- Therefore, rows can be loaded in whatever format is most convenient or efficient.
    and Columns are extremely lightweight and composable.

[dev] Extensible

• VisiData, Sheet, Column inherit from Extensible to spread out over functional modules
• Extensible API (@Sheet.api, etc)

Sources

• [dev] Path wrapper
  • UrlPath
  • PathFd

Sheets

Rows

- references to any Python object
- dup-sheet
- dive-row

Columns

- Compute Engine

Cursor

- both row and column
- getters/setters
- types
- getting values for savers
- aggregators
- nulls

Commands

metasheets

- natural consequence
- brings it all together
  - universal interface
  - [useful to know even if not dev] ColumnsSheet: sheet.rows = sheet.source.columns
- sheets
- columns
- options

Details

VisiData singleton

Display and Computation Engine

Screen Layout

[dev] BaseSheet.draw()

[dev] Sheet.draw()

• computation, caching

  • [dev] core Sheet value computation path in Sheet.draw
    • Column.getDisplayValue ->
    • Column.getCell ->
    • Column.getTypedValue ->
    • Column.getValue (possibly memoized) ->
    • Column.calcValue ->
    • Column.getter

• formatting

• DisplayWrapper

[dev] Colorizers

• adding your own Colorizers

Types

• [dev] properties of Types

Settings

• settings context
  • hierarchy: manual override, command-line override, Sheet instance, Sheet class, Sheet parents, global

Options

[dev] OptionsObject

OptionsSheet

.visidatarc and .visidata/

Commands

• command concepts
  • add
  • delete
  • copy
  • dive
  • open
  • save
  • reload
  • visibility
  • cursor movement

longnames and keybindings

• sheet, stack, jump, selected, go
  • active sheet determines options context
• prefix: g z ESC ALT

[dev] Command(keystroke, longname, execstr)

helpstr and commands.tsv

CommandHelpSheet

execstr

rules

• extensions use cursor* only in execstr

the visidata.globals() sandbox

undo

CommandLog and Replay

• conversion is null replay
• parameterization

Status

• left status
• right status
• status history
• status(), warning()

Errors and Exceptions

• ^E previous [global] error traceback (g^E all prev error tracebacks; z^E for this cell caught during calcValue/typeConversion/formatValue)

[dev] EscapeException

- all try blocks with a catch-all except: clauses must also have a separate except
- EscapeException: clauses that re-raises, so that user can early-out slow operations

[dev] error()

- fail vs error: fail is for user 'mistakes'. not a problem, just didn't work. error is for internal or system errors?

[dev] ExpectedException

[dev] exceptionCaught()

Loaders

• [dev] reload()

Savers

openSource

piping through stdin/stdout

Interesting Loaders

pandas adapter

@asyncthread

• [dev] Progress contextmgr, .addProgress()
• [dev] sync(expectedThreads)
• ThreadsSheet
• (bisee VisiData for async exceptions, ^C, ^T)

Input

Builtin readline-like editor

[dev] confirm()

commandlog Input column

Features of all Sheets

search in column(s)

sort by column(s)

select rows

• [dev] Sheet.selectedRows and Sheet.isSelected(row)

• select by regex

• select by python expr

• [dev] rowid()

  • rows are not hashable and can't be looked up easily by content without linear (and expensive) search.
  • But id(obj) is a hashable integer which is guaranteed to be unique and constant for this object during its lifetime.
  • We store id(row) as the keys in a dict pointing to the row itself (is this convenience used?)
  • this makes selection/unselection and checking for selection, have the same cost as set add/remove/check
  • Sheet.selectedRows has to be computed bc _selectedRows.values()are not sorted in sheet order. This is O(n).
  • select/unselet/toggle all are now O(n log n), whereas they could have been O(n) if selection were in e.g. a parallel array, or an attribute on the row.

fill-nulls

Derived Columns

split/parse

Modules

clipboard

motd

[dev] urlcache, to get data and cache it in .visidata, only refetching it every n days, and returning a Path

Canvas

Derived Sheets

Grouping

Frequency Table

Pivot Table

Aggregators

Join

Melt

Describe

Graph (Scatterplot)

Extending and Customizing

How to make a VisiData theme/skin

[dev] How to make a sheet

Plugins

Examples

Loader

Saver

Command

API Reference

VisiData (vd)

- vd.push, vd.replace, vd.remove sheets
- vd.windowWidth, vd.windowHeight
- vd.searchRegex(sheet, moveCursor=, reverse=)
  - .searchContext is global
- vd.getkeystroke()
- vd.refresh()
- vd.editText(y, x, w, **editargs)
- vd.addHook(hookname, hookfunc), .hooks
- vd.threads
- vd.status(), .statuses, .statusHistory
- vd.input(), .lastInputs, .inInput
- .keystrokes, .prefixWaiting
- exceptionCaught, .lastErrors
- .sheets

Sheet

- .rows, .columns, .commands, .colorizers
- __init__(name, *sources, **attrs)
- nKeys
- topRowIndex
- cursorRowIndex
- leftVisibleColIndex
- rightVisibleColIndex

- cursorVisibleColIndex, into visibleCols
- cursorColIndex, into columns (computed, O(n))
- cursorCol

- keyCols
- keyColNames
- nonKeyVisibleCols
- toggleKeyColumn(colidx)
- reload
   - .loader so don't have to subclass everything
- addRow, in case a subclass wants to track them all
- addColumn, to set up Column.sheet and ._id (columns belong to sheets)
- newRow: [blankRow] should be overridden for different Sheets; returns a new blank row
- leftStatus (reconcile with right status interface)
- recalc
- searchColumnNameRegex
- searchRegex
- moveRegex
- addColorizer

- semantics of copy
- (deepcopy same as copy)
- visibleCols (computed in O(ncols); cached because was too expensive for datasets with many columns; cache must be cleared every frame)
- nVisibleCols
- visibleColNames (?)
- nCols
- nRows
- cursorValue -> any
- cursorTypedValue -> cursorCol.type
- cursorDisplay -> str
- cursorCell -> DisplayWrapper
- nVisibleRows
- visibleRows
- cursorRow
- progressPct
- source/sources
- name
- exec_command
- evalexpr

- deleteSelected
- isSelected
- selectedRows
- select
- toggle
- unselect
- selectRow
- unselectRow
- selectByIdx (internal, used by |)
- unselectByIdx

- gatherBy(func(r)): yield rows for which func(r) is not false
- moveVisibleCol
- cursorDown
- cursorRight
- pageLeft
- moveToNextRow
- draw()
    - checkCursor (internal)
    - calcColLayout (internal)
    - drawColHeader (internal)
- editCell

TextSheet

Column

- __init__
- copy/deepcopy
- name vs id
- format() and fmtstr
- hidden
- getValueRows
- getValues
- calcValue
- getTypedValue
- getValue
- getCell
- getDisplayValue
- putValue
- setValue
- setValues
- setValuesFromExpr
- getMaxWidth
- toggleWidth

ColumnAttr

ColumnItem

SubrowColumn

ColumnEnum

ColumnExpr

LazyMap

LazyMapRow

other globals

• status()
• run()

[dev] curses helpers

• [dev] clipstr
• [dev] EnableCursor
• [dev] SuspendCurses

Man page

Keyboard layouts