papotch

Motivation

This is an idea that has been hitching my brain for some times now. I grow frustrated seeing people running toward FME for very simple data transformations. Nothing against that in itself, but having alternatives is always good.

There are obviously the amazing geopandas and polars-st. But these libraries take sometimes to learn and I feel there is too much friction for someone using FME regularly to even consider using them.

So I experimented a sort of ETL API that would be approached the same way you would construct a FME workspace: start with a reader, chain transformations and write the result.

Example

This would look like :

from papotch import Reader

r = Reader("path/to/data.shp")

d = r.extract()

d.head()
┌────┬─────────┬─────────┬──────────┬─────────────────────────────┐
│id  ┆value_int┆value_txt┆value_bool┆geometry                     │
│--- ┆---      ┆---      ┆---       ┆---                          │
│i64 ┆i64      ┆str      ┆bool      ┆str                          │
╞════╪═════════╪═════════╪══════════╪═════════════════════════════╡
│0   ┆0        ┆test     ┆true      ┆POLYGON ((2532293.4834 11600…│
│1   ┆1        ┆éjwefw   ┆false     ┆POLYGON ((2548329.787 115443…│
│2   ┆2        ┆kjprg    ┆false     ┆POLYGON ((2538461.2386 11469…│
│null┆null     ┆null     ┆false     ┆POLYGON ((2532026.1 1159262.…│
│null┆null     ┆null     ┆false     ┆POLYGON ((2530319.7138 11547…│
└────┴─────────┴─────────┴──────────┴─────────────────────────────┘

# Buffer, calculate the resulting area, then the bounding coordinates.
_ = d.buffer(distance=2).area(column_name="calc_area").bounds()

d.head()
┌────┬─────────┬─────────┬──────────┬───────────────┬─────────┬────────────────────────┐
│id  ┆value_int┆value_txt┆value_bool┆geometry       ┆calc_area┆_bounds                 │
│--- ┆---      ┆---      ┆---       ┆---            ┆---      ┆---                     │
│i64 ┆i64      ┆str      ┆bool      ┆binary         ┆f64      ┆array[f64, 4]           │
╞════╪═════════╪═════════╪══════════╪═══════════════╪═════════╪════════════════════════╡
│0   ┆0        ┆test     ┆true      ┆b"\x01\x03\x08…┆112.79491┆[2.5323e6,1.16e6,…1.16e…│
│1   ┆1        ┆éjwefw   ┆false     ┆b"\x01\x03\x08…┆112.79491┆[2.5483e6,1.15e6,…1.15e…│
│2   ┆2        ┆kjprg    ┆false     ┆b"\x01\x03\x08…┆112.79491┆[2.5385e6,1.14e6,…1.14e…│
│null┆null     ┆null     ┆false     ┆b"\x01\x03\x08…┆112.79491┆[2.5320e6,1.15e6,…1.15e…│
│null┆null     ┆null     ┆false     ┆b"\x01\x03\x08…┆112.79491┆[2.5303e6,1.15e6,…1.15e…│
└────┴─────────┴─────────┴──────────┴───────────────┴─────────┴────────────────────────┘

d.write("PG:service=my_service", layer="my_layer")

All the heavy lifting is done by polars-st, papotch would just be an opiniated abstraction simplifying operations for someone who wants a quick pipeline. At the cost of flexibility of course.

Installation

You can install papotch from the pypi index.

pip install papotch

Repository

The code is hosted on Codeberg.

Documentation

The API documentation is accessible at papotch.jujube.ch.