dataframe

Tabular data structure for data analysis in CHICKEN Scheme, inspired by the data frame implementations in R, Python, and Racket, with an API modeled on dplyr's verbs (filter, group-by/summarize, joins, arrange, distinct, slice, sample, head/tail).

Documentation

The dataframe library provides an interface for representing tabular data in rows and columns. It is inspired by the various dataframe implementations found in R, Python and Racket, and its row-filtering, grouping/summarizing, join, and row-selection operations are modeled on dplyr's verbs.

The dataframe library also provides functions for loading and saving data from data frames as well as routines for descriptive statistics and linear regression.

A data frame is a persistent (immutable) structure: every operation that "changes" one returns a new data frame and leaves the original untouched.

Installation

chicken-install dataframe

This builds and installs four components: fmt-table, dataframe, dataframe-statistics, and dataframe-regression.

Columns

Each dataframe consists of a collection of columns, which in turn is an object consisting of a unique key, data collection, and an associative list of properties. The following operations are defined on columns.

column? objprocedure: Returns true if the given object is a column.

column-key columnprocedure: Returns the key of the column.

column-properties columnprocedure: Returns an associative list with column properties.

column-collection columnprocedure: Returns the data collection of the column.

column-deserialize column portprocedure: Loads the data collection of a column from the given port.

column-serialize column portprocedure: Stores the data collection of a column to the given port in an s-expression format.

Creating data frames

(make-data-frame [column-key-compare: compare-symbol])procedure: Creates a new dataframe, with optional argument a procedure that specifies how to compare column keys. Default is comparison on symbols (compare-int is also provided, for integer keys). Returns the new dataframe.

df-insert-column df key collection propertiesprocedure: Inserts a new column with the given key, data collection, and properties (or replaces the column, if key already exists). Returns a new dataframe with the inserted column.

df-insert-derived df parent-key key proc propertiesprocedure: Inserts a derived column, that is a column whose data elements are obtained by mapping a procedure onto the elements of an existing (parent) column. The derived column tracks its parent lazily, i.e. it is recomputed from the parent's current values rather than copied. Returns a new dataframe with the inserted column.

df-insert-columns df lseqprocedure: Inserts the columns contained in the given lseq of column objects.

df-update-column df key collection propertiesprocedure: Returns a new data frame with the existing column key replaced.

df-delete-column df keyprocedure: Returns a new data frame with column key removed.

(df-from-rows column-keys source [column-key-compare: compare-symbol])procedure: Creates a data frame with the given column keys and populates it with data from source, a list of rows (or a generator of rows), each row being a list of values in column-keys order.

Accessing data frames

show df portprocedure: Displays a table of the rows and columns contained in the dataframe to port (#f means the current output port). show is exported by the yasos egg, not by dataframe itself -- (import yasos dataframe) to use it. How many rows/columns are shown is controlled by the parameters below.

display.max-elementsprocedure
display.max-columnsprocedure: Parameters controlling how many rows (default 20) and columns (default 10) show prints before truncating; set to 0 to disable truncation.

df-row-count dfprocedure: Returns the number of rows in the dataframe.

df-column df key failure-objectprocedure: Returns the (key . column) pair indicated by the given key, or failure-object if key isn't present.

df-column-properties df key failure-objectprocedure: Returns the properties of column key, or failure-object if absent.

df-collection df key failure-objectprocedure: Returns the data collection of column key, or failure-object if absent.

df-columns dfprocedure: Returns a lazy sequence containing the columns of the dataframe.

df-filter-columns df procprocedure: Returns a filtered lseq of the columns of the dataframe according to the given filter predicate procedure.

df-select-columns df keysprocedure: Returns an lseq of the columns of the dataframe that have the keys enumerated in the given list of keys.

df-keys dfprocedure: Returns the keys of all columns in the dataframe.

df-items dfprocedure: Returns an lseq of the key-column pairs contained in the dataframe.

(map-columns proc df [keys: #f])procedure: Applies the given procedure to the named columns of the dataframe (default: every column) and returns the result as a dataframe. proc receives the column object itself.

(map-collections proc df [keys: #f])procedure: Like map-columns, but proc receives each column's data collection rather than the column object.

apply-collections proc df key ...procedure: Applies the given procedure to the data collections of the named columns of the dataframe and returns the result directly (i.e. not wrapped in a dataframe).

(reduce-collections proc df seed [keys: #f])procedure: Fold over the data collections of the named columns, starting from seed.

Iterators

df-for-each-column df procprocedure: Applies proc to each (key . column) pair.

df-for-each-collection df procprocedure: Applies proc to the data collection of each column.

df-gen-rows dfprocedure: Returns a generator procedure that returns the dataframe rows in succession, each row a list of values in df-keys order.

df-gen-columns dfprocedure: Returns a generator procedure that returns (key . column) pairs in succession.

Row filtering

df-filter-rows df predicateprocedure

Returns a new data frame with only the rows for which predicate is true. predicate is called once per row with a row accessor: a one-argument function that, given a column key, returns that column's value in the current row.

(df-filter-rows df (lambda (get) (> (get 'age) 30)))

Automatically picks between two strategies depending on df's row count (rebuilding row by row for small data, or marking matches with a bit vector for large data -- see *filter-strategy-threshold* below); both give identical results, so this is a performance detail.

df-filter-rows-multi df predicate ...procedure: Like df-filter-rows, but keeps rows for which every given predicate is true, evaluated together in a single pass.

*filter-strategy-threshold*procedure

A parameter (default 10000): df-filter-rows uses the bitmap strategy at or above this many rows, and the simpler row-rebuilding strategy below it.

If df is grouped (see Grouping and summarizing below), the result stays grouped by the same columns. A group left with no rows after filtering simply doesn't appear, the same way dplyr's filter() treats a grouped tibble.

Grouping and summarizing

df-group-by df group-keysprocedure: Groups df by one column (a single key) or several (a list of keys). Returns a grouped data frame: it responds to every ordinary data-frame operation above (df-row-count, df-keys, show, and so on) by delegating to the underlying, ungrouped data, the same way a grouped tibble in dplyr is still a data frame.

grouped-dataframe? objprocedure: Returns true if obj was returned by df-group-by (or any operation that preserves grouping).

df-summarize grouped-df summary-specsprocedure

Reduces a grouped data frame to one row per group. Each spec in summary-specs is (result-name source-col-key . function): function is called with that group's values from source-col-key, as a plain list; if source-col-key is #f, function is called with the group's list of row indices instead, so a column-agnostic function like length gives the group's row count.

(df-summarize (df-group-by df 'category)
  (list (cons* 'n #f length)
        (cons* 'total 'amount (lambda (vs) (apply + vs)))))

The result is an ordinary (ungrouped) data frame, matching dplyr's summarise(), which likewise drops grouping once there's only one grouping variable left.

df-group-by-apply df group-keys procprocedure: Shorthand for (proc (df-group-by df group-keys)).

df-ungroup grouped-dfprocedure: Returns the data frame grouped-df was built from, discarding the grouping.

Joins

Four dplyr-style joins, each (join left-df right-df by [suffix: '(".x" . ".y")]):

(df-inner-join left-df right-df by [suffix: '(".x" . ".y")])procedure: Keeps rows whose join key is present in both data frames.

(df-left-join left-df right-df by [suffix: '(".x" . ".y")])procedure: Keeps every row of left-df; right-df's columns are filled with the symbol na where there's no match.

(df-right-join left-df right-df by [suffix: '(".x" . ".y")])procedure: Keeps every row of right-df; left-df's columns are filled with na where there's no match.

(df-full-join left-df right-df by [suffix: '(".x" . ".y")])procedure

Keeps every row of both data frames, filling the other side with na where there's no match.

by is one of:

a symbol: join on that column, same name on both sides
a list of symbols: join on all of them, same names on both sides
a list of (left-key . right-key) pairs: for differently-named join columns, e.g. '((dept-id . id))

The output's join-key column is always named after the left side's key and takes the left row's value when there is one, falling back to the right row's value (under its own name) for a right-only row in a right/full join. Duplicate keys fan out into the cross product of matches, as in a real relational join. Non-key columns that collide between the two sides are renamed with suffix (a (left . right) pair of strings, default (".x" . ".y")).

(df-inner-join users orders 'user-id)
(df-left-join employees departments '((dept-id . id)))
(df-inner-join sales targets '(date store-id) suffix: (cons ".a" ".b"))

Row selection and ordering

(df-head df [n 6])procedure: Returns the first n rows (default 6, as in R/dplyr); n is clamped to df's row count.

(df-tail df [n 6])procedure: Returns the last n rows.

df-slice df indicesprocedure: Returns the rows at the given 0-based positions, in the order given. An index may repeat, duplicating that row. Row-index convention is 0-based, not R's 1-based.

df-arrange df order-spec ...procedure

Sorts df by one or more columns. Each spec is a column key (ascending) or (key . 'desc) (descending); ties are broken by the next spec, and rows tied on every spec keep their original relative order.

(df-arrange df 'year (cons 'revenue 'desc))

(df-distinct df [columns] [keep-all: #f])procedure: Returns the distinct rows of df (by default, distinct whole rows; columns restricts what counts as distinct), keeping the first occurrence of each and preserving their original relative order. Unless keep-all is true, only columns is kept in the result.

(df-sample df n [with-replacement: #f])procedure

Returns n rows chosen at random: independently with replacement (duplicates possible) when with-replacement is true, or without replacement otherwise (errors if n exceeds df's row count, since that many distinct rows don't exist).

All six of the above preserve grouping: applied to a grouped data frame, the result is regrouped by the same columns.

Descriptive statistics

From the dataframe-statistics component ((import dataframe-statistics)). Each of the following (other than describe and the grouped-* procedures) returns a data frame with the same columns as its input, each holding the single computed value for that column.

describe df portprocedure: Displays a table with the min/max/mean/sdev of each column in the dataframe.

cmin dfprocedure: Computes the minimum value of each column.

cmax dfprocedure: Computes the maximum value of each column.

mean dfprocedure: Computes the mean value of each column.

median dfprocedure: Computes the median value of each column.

mode dfprocedure: Computes the mode value of each column.

range dfprocedure: Computes the difference between maximum and minimum value of each column.

percentile dfprocedure: Computes the percentile values of each column.

variance dfprocedure: Computes the (sample) variance of each column.

standard-deviation dfprocedure: Computes the (sample) standard deviation of each column.

coefficient-of-variation dfprocedure: Computes the coefficient of variation of each column.

grouped-mean grouped-df col-keyprocedure: Computes the mean of col-key within each group, as a data frame with one row per group (built on df-summarize).

grouped-summary grouped-df col-keyprocedure: Computes the mean, min, max, standard deviation, and row count (n) of col-key within each group.

Regression and correlation

From the dataframe-regression component ((import dataframe-regression)).

linear-regression df x yprocedure: Least-squares linear regression of column y on column x. Returns five values: intercept, slope, correlation coefficient r, R^2, and the significance of the slope (use let-values to capture all five).

correlation-coefficient df x yprocedure: Correlation coefficient between columns x and y (Pearson).

spearman-rank-correlation df x yprocedure: Spearman rank correlation coefficient between columns x and y.

I/O

df-serialize df portprocedure: Stores the dataframe in an s-expression format to the given port.

df-deserialize df portprocedure: Loads the data collections of the dataframe columns from the given port (typically called on a fresh (make-data-frame)) and returns the resulting data frame.

dplyr equivalents

filter(): df-filter-rows, df-filter-rows-multi
group_by(): df-group-by
summarise(): df-summarize
ungroup(): df-ungroup
inner_join(): df-inner-join
left_join(): df-left-join
right_join(): df-right-join
full_join(): df-full-join
head(): df-head
tail(): df-tail
slice(): df-slice (0-based)
arrange(): df-arrange
distinct(): df-distinct
slice_sample(): df-sample

filter()/arrange()/slice()/distinct()/head()/tail() all preserve a grouped data frame's grouping, as in dplyr; summarise() drops it, also as in dplyr. Two real dplyr behaviors aren't implemented: filter()'s predicate can reference per-group aggregates in real dplyr (e.g. filter(x > mean(x)) computed within each group); here, a predicate always sees a single row. And dplyr's slice() and grouped distinct() operate per group, whereas here they always operate on the whole table (then re-group the result) -- the same simplification applied uniformly across all six row-selection functions rather than singled out for just those two.

Examples

(import scheme srfi-1 yasos dataframe dataframe-statistics)

(define df (make-data-frame))

(define df1
  (df-insert-column
   df
   'base
   (list-tabulate 100 (lambda (x) (- x 10)))
   '()))


;;  exponential series
(define df2
  (df-insert-derived
   df1 'base 'exp
   (lambda (x) (* 2.0 (exp (* 0.1 x))))
   '()
   ))

(show df2 #f)
(describe df2 #f)

(linear-regression df2 'base 'exp)

Testing

csi -s tests/run.scm

1.0: Added dplyr-style operators for joins, filters selection; ported to CHICKEN 6.
0.1: Initial release

License

Copyright 2019-2026 Ivan Raikov.

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or (at
your option) any later version.

This program is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
General Public License for more details.

A full copy of the GPL license can be found at
<http://www.gnu.org/licenses/>.

chickadee » dataframe

Identifier search

TOC »