ORiGAMi

Most mixed-type Machine Learning models, tools and algorithms (e.g. Pandas, scikit-learn) operate on flat tables. Real-world application layer data is often nested JSON with optional fields and variable-length arrays. This mismatch forces practitioners to flatten the data into tabular form, which is lossy, laborious, and doesn’t scale as it creates very wide and sparse tables.

The project began at MongoDB Research as an attempt to make predictions from JSON data without lossy flattening into tabular form. It grew into a general architecture for semi-structured density estimation, with applications in cardinality estimation, learned indexes, and privacy-preserving synthetic data.

The key insights are:

1. Density estimation is a general primitive on which many downstream tasks can be built: sampling, conditional generation, outlier detection, imputation, and more. By focusing on density estimation rather than a specific task, we could design a more general architecture.
2. Transformers are general-purpose sequence models which can operate on any tokenised inputs. By designing a tokenisation scheme for nested JSON, we could leverage the power of transformers without flattening the data.