archives – Wes McKinney

Joining Posit’s Polyglot Data Science Mission

work

open source

posit

TL;DR I am joining Posit today as a Principal Architect where I will advocate for the needs of the PyData ecosystem in Posit’s work as well as continue advancing critical…

Voltron Data Update: Transitions

startups

work

TL;DR I am transitioning out of my full-time CTO role at Voltron Data so that I can expand my portfolio of entrepreneurial and open source data projects. While no longer…

The Road to Composable Data Systems: Thoughts on the Last 15 Years and the Future

retrospective

thoughts

A new joint VLDB paper on Composable Data Management Systems with Meta, Databricks, Sundeck, and others at is out! This post is a reflection on how I arrived at thinking…

Joining Forces for an Arrow-Native Future

work

apache arrow

Too often people say “let’s do something together” in passing, and don’t. There’s the occasional inter-project collaboration, but rarely will people take that next step.…

Ursa Labs March 2019 Report

ursa labs

work

The first quarter of 2019 has now wrapped up. In March we spent a good amount of time focused on getting the 0.13.0 Apache Arrow release out of the door. I will mention a…

Ursa Labs February 2019 Report

ursa labs

work

The team had a busy 28 days this February. The Apache Arrow community is discussing a 0.13 release toward the end of March, so we spent February helping the project toward…

Ursa Labs January 2019 Report

ursa labs

work

Ursa Labs had a busy January that went by too quickly. After a high-intensity 3 months of development, we helped release Apache Arrow 0.12 on January 20th. A good chunk of…

Leaving NYC for Nashville

For ten out of the last eleven years, I’ve lived in two places: New York City and San Francisco. The last two years have been in NYC. After founding Ursa Labs, a…

Announcing Ursa Labs’s partnership with NVIDIA

ursa labs

work

I’m excited to announce that NVIDIA AI Labs has signed on as a supporter of Ursa Labs. NVIDIA’s new open source RAPIDS data science platform uses Apache Arrow for an…

Announcing Ursa Labs: an innovation lab for open source data science

work

ursa labs

apache arrow

Funding open source software development is a complicated subject. I’m excited to announce that I’ve founded Ursa Labs (https://ursalabs.org), an independent development lab…

Some comments to Daniel Abadi’s blog about Apache Arrow

apache arrow

databases

Well-known database systems researcher Daniel Abadi published a blog post yesterday asking Apache Arrow vs. Parquet and ORC: Do we really need a third Apache project for…

Feather format update: Whence and Whither?

apache arrow

Earlier this year, development for the Feather file format moved to the Apache Arrow codebase. I will explain how this has already affected Feather and what to expect from…

Apache Arrow and the “10 Things I Hate About pandas”

pandas

apache arrow

This post is the first of many to come on Apache Arrow, pandas, pandas2, and the general trajectory of my work in recent times and into the foreseeable future. This is a bit…

Extreme IO performance with parallel Apache Parquet in Python

parquet

apache arrow

In this post, I show how Parquet can encode very large datasets in a small file footprint, and how we can achieve data throughput significantly exceeding disk IO bandwidth…

Streaming Columnar Data with Apache Arrow

apache arrow

Over the past couple weeks, Nong Li and I added a streaming binary format to Apache Arrow, accompanying the existing random access / IPC file format. We have implementations…

2017 Outlook: pandas, Arrow, Feather, Parquet, Spark, Ibis

apache arrow

pandas

ibis

parquet

work

2017 is shaping up to be an exciting year in Python data development. In this post I’ll give you a flavor of what to expect from my end. In follow up blog posts, I plan to…

From Arrow to pandas at 10 Gigabytes Per Second

apache arrow

pandas

In this post I discuss some recent work in Apache Arrow to accelerate converting to pandas objects from general Arrow columnar memory.

Kinesis Advantage2: Impressions

work

gear

I discuss my impressions of the newest version of the classic Kinesis Advantage contoured mechnical keyboard

GitHub’s one-dimensional view of open source contributions

rants

open source

TL;DR One of the most harmful parts of the GitHub platform is the code contribution calendar. This “hacker score card” overemphasizes the value of commits over the other…

Feather: it’s about metadata

apache arrow

Summary: Feather’s good performance is a side effect of its design, but the primary goal of the project is to have a common memory layout (Apache Arrow) and metadata (type…

Why pandas users should be excited about Apache Arrow

apache arrow

pandas

I’m super excited to be involved in the new open source Apache Arrow community initiative. For Python (and R, too!), it will help enable

The problem with the data science language wars

rants

data science

python

r

I really enjoyed the cheeky blog post by my pal Rob Story.

What’s changed

personal

Some reflections from turning 30.

Thoughts on joining Cloudera

work

After some unanticipated media leaks (here and here), I was very excited to finally share that my team and I are joining Cloudera. You can find out all the concrete details…

Introducing vbench, new code performance analysis and monitoring tool

python

benchmarks

Do you know how fast your code is? Is it faster than it was last week? Or a month ago? How do you know if you accidentally made a function slower by changes elsewhere?…

A Roadmap for Rich Scientific Data Structures in Python

rants

python

pandas

Discussion thread on Hacker News