Introducing dpkit

Introduction

dpkit is a fast data management framework built on top of the Data Package standard and Polars DataFrames. It supports various formats like CSV, JSON, and Parquet and integrates with platforms such as CKAN, Zenodo, and GitHub.

Where to get it

dpkit is available

as a CLI tool for Linux, macOS, and Windows - https://terminal.dpkit.app
as a TypeScript library for Node/Deno/Bun/browser - https://typescript.dpkit.app

How fast it is

dpkit is 10x-100x faster than frictionless-py for tabular validation, showing performance similar to qsv. For example, dpkit takes around 5 seconds to validate a 1GB CSV file. In addition to a spectacular speed, dpkit is based on the rich Data Package validation system compared to JSON Schema based validation used in qsv.

Exploring data from the terminal

dpkit provides an interactive CLI tool for data wrangling and validation. For example, you can explore a table from a Zenodo dataset with a simple command providing you an interactive interface to the dataset and the table (including sorting and pagination):

dp table explore -p https://zenodo.org/records/7559361

dpkit is in its early days and we are looking for feedback and contributions. Please star it on Github to help the project grow - https://github.com/datisthq/dpkit

Special thanks to NLnet for supporting this development!

What we build

Web application development

Data layer through to interface: search, integrations, background jobs, and the infrastructure to keep it running.

Full-stack web applications · Typed APIs · Durable workflows · Container orchestration

Internal tools & back-office systems

The internal tool your industry actually needs — leads, deals, inventory, invoicing, permissions, an audit trail. Built around how you work rather than around a template.

Transactional databases · Identity & access · Typed APIs · Full-stack web applications

Mobile app development

iOS and Android from one codebase, sharing types with your backend so the two cannot drift apart. Through the stores, or over the air.

Mobile & desktop · Typed APIs · Identity & access

Landing pages & marketing sites

Fast, prerendered sites that load instantly and cost almost nothing to run. Content you can edit yourself, in as many languages as you need.

Full-stack web applications · Design systems · Internationalization · Edge platform

Project examples

capturemycarro.app

The Mycarro car search engine as it runs today, searching across its live Portuguese listings.

Mycarro — car marketplace and dealer CRM

A used-car marketplace live in Portugal. Consumer search over listings gathered from across the market, a dealer back office for leads, inventory and invoicing, and a mobile app — built end to end.

we built

under NDA

Work we cannot show

NDA — Swiss fintech platform MVP

A multiplatform MVP for a regulated financial services platform, designed and built end to end: a contract-first backend, an operations dashboard, and iOS and Android apps generated from the same contract. Delivered.

client · NDA