Blog

Concise names for cached tables.

By Georg Heiler

Have you ever wondered where the cryptic names of cached dataframes and RDD in Spark’s web UI belong to? Usually no specific name is set. When you apply a df.cache spark will auto generate the name as a snippet from the query plan.…

By Georg Heiler

More and more people start to work with large quantities of geospatial data and think about using spark and one of its geospatial additions like geomesa, geospark or geotrellis. And soon they come to realize that one of the tools does not provide a function which would be commonly required.…

Use Dask to paralellize python

By Georg Heiler

I have a some binary cohorts and need to sample for each cohort where the target label 1 is matching a observation from a dataframe where the target is 0.

sample data

First, I will generate some sample data.…

By Georg Heiler

R users traditionally use the sf package to process geospatial operations. This is limited to a single compute node, often a single thread and to data which fits in memory.

Traditionally scaling this type of data analysis involved heay lifting on the coding side.…