Georg Heiler | Vienna Data Science Group

blazing-fast data science on GPUs

Mon, 06 Apr 2020 00:00:00 +0000

graph analytics with cuGraph – ego network

by Georg Heiler

In an ever more connected world the size of datasets for various use cases is increasing more and more. For traditional graph tools using the CPU this fact can make your analyses painfully slow. Calculations can easily run for hours - if not days. This makes the analytical worflow anything but interactive.

Accellerators like GPUs can help to speed things up - a lot! Even complex graph algorithms can now execute at interactive speed in seconds and development is a breeze again. In fact I myself experienced this for a graph of about 100 million edges. The traditional tools would not be able to handle load it well - whereas cuGraph operates on it in a matter of seconds for various graph algorithms.

In the past, learning to write low level CUDA as well as being able to write code which deals well with the intricacies of GPUs regarding memory limitations and parallelism was very complex. NVIDIA developed RAPIDS AI to ease this pain by offering python bindings. The API of these adheres mostly to well known ones of popular python packages like pandas or networkx.

However, cuGraph is still rather early. Various well established graph algorithms are not ported to this framework yet. This is also the case for calculating an ego network. In the following lines I will demonstrate how to add this functionality.

ego network for cuGraph

using the management tool

!nvidia-smi

you can check the gpus available for you and their utilization. If you have an issue like a resource allocation error or an out of memory error, consider to restart the jupyter notebook or manually kill the offending process like outlined on Stackoverflow

nvidia-smi | grep 'python' | awk '{ print $3 }' | xargs -n1 kill -9

# show the user
ps -u -p <<pid>>

In case you have multiple gpus available, it is good practice to start out with a specific GPU:

import os
os.environ["CUDA_VISIBLE_DEVICES"] = "1"

This especially can help in a multi-user scenario to share resources fairly.

Import the various packages

import pandas as pd
import cudf
import cugraph

Load the data and create a graph. This particular karate data set is made available directly by the developers of cuGraph on gitHub. Download the CSV and read with pandas:

karate = pd.read_csv('karate.csv', header=None, sep=' ')
karate.columns = ['src', 'dst', 'weights']

Now move the CPU data to the gpu.

Note

Indeed, cuDf offers bult in tooling to load the data directly. Obviously you can speed up the analyses even more if you use them.

However there are two points why I choose not to use them and rely on traditional pandas to get the job done:

memory: I use spark to pre-aggregate the data (on my real dataset). Even after aggregation with about 100 million edges it is still fairly large. I need to add a preprocessing step to convert identifiers to cuGraphs src/dst ids. Usually cugraph offers a function for renumbering. But it fails: as I cannot load all the data into the gpu in the first place and secondly it even runs out of memory even when reducing the data a bit. This is why I choose to use pandas for preprocessing to trim the data down even further until it fits into the GPU as I only have a P100 with 16GB of RAM.
parquet cuDf can read a single parquet file. But any big-data tool will usually not write a single file, but instead a whole folder with many randomly named parquet files. In my opinon it is simply easier to rely on the battle-tested functions offered by pandas which happily read the whole directory.

karate = cudf.from_pandas(karate)

Let’s create the cuGraph graph object


G_karate = cugraph.DiGraph()
G_karate.from_cudf_edgelist(karate, source='src', destination='dst', edge_attr='weights', renumber=False)

Now finally, lets derive the ego network. This functionality is currently not a built in part of cuGraph, but it offers the necessary building blocks to create such a function quickly.

CuGraph offers various graph traversal functions like breadth first search. Starting from a given vertex it will traverse the whole graph and derive the distance. The candidate list is filtered to only contain vertices of the desired maximum distance from the ego node. In a second step, the edge list is filtered to only contain entries where the src or dst column is contained in the list of candidates.

Warning

cuGraph offers a bult in function to create a subgraph (cugraph.subgraph(G_carate, ego_candidates)). However, for me this always deleted the main ego node (on my real data set). As for an ego graph this is the main vertex I want to retain I had to hand-roll the extraction of the desired entries from the edge list.

ego_id = 1

def make_ego_csv(level):
 cudf_res = cugraph.bfs(G_karate, ego_id)
 ego_candidates = cudf_res[cudf_res.distance <= level].vertex
 print(ego_candidates.shape)
 sub_edges_g = karate[(karate.src.isin(ego_candidates) | karate.dst.isin(ego_candidates))]
 print(sub_edges_g.shape)

 # rename to gephi supported names
 sub_edges_g.columns = ['Source', 'Target', 'Weight']

 if level > 1:
 # to save memory in gephi, reduce the details if even the sub-graph gets large
 sub_edges_g.drop(['Weight']).to_csv(f"ego_{level}_graph.csv", index=False)
 else:
 sub_edges_g.to_csv(f"ego_{level}_graph.csv", index=False)

 print('**********')

sub_edges_g = make_ego_csv(0)
make_ego_csv(1)
make_ego_csv(2)

(1,)
(18, 3)
**********
(10,)
(80, 3)
**********
(23,)
(148, 3)
**********

And finally, for validation: indeed the ego node is still in the data.

sub_edges_g[(sub_edges_g.Source == ego_id) | (sub_edges_g.Destination == ego_id)]

	Source	Destination
0	0	1
16	1	0
17	1	2
18	1	3
19	1	7
20	1	13
21	1	14
22	1	15
23	1	16
24	1	19
26	2	1
36	3	1
51	7	1
68	13	1
73	14	1
75	15	1
78	16	1
84	19	1

summary

cuGraph can run various graph analyses in a matter of seconds. But it is still a little bit immature and some functions like calculating an ego network are missing. You will run into edge cases especially when the datasets grow larger - as mentioned above in the notice box: some preprocessing might become necessary.

But overall adding the function for the ego network is simple!

This was a repost. The original post is found here: https://georgheiler.com/2020/04/03/blazing-fast-data-science-on-gpus/

reproducible geospatial visualization in kepler.gl

Tue, 26 Nov 2019 00:00:00 +0000

Jinja templates in jupyter for kepler.gl

Effortless and great looking visualizations can be achieved using https://kepler.gl/. However, kepler by itself is tedious to use as updating the data files like you might have done with QGIS or ArcGIS does not work on the website.

This is easily fixed with the kepler.gl python package. Using tools like pandas and jinja templates these visualizations can be created once via drag and drop and then loaded programmatically by storing the configuration in git.

An example visualization will be created below to contain hexagons within the boundaries of Austria.

MAPBOX_ACCESS_TOKEN = 'your_API_token'

%pylab inline

import pandas as pd
import seaborn as sns; sns.set()
import geopandas as gp
from h3 import h3

from shapely.ops import unary_union
from shapely.geometry.polygon import Polygon

from keplergl import KeplerGl
import json
import jinja2

def python_dict_to_json_file(dict_object, file_path):
 try:
 # Get a file object with write permission.
 file_object = open(file_path, 'w')

 # Save dict data into the JSON file.
 json.dump(dict_object, file_object, indent=4)

 print(file_path + " created. ")
 except FileNotFoundError:
 print(file_path + " not found. ")


def jinja_json_to_dict(env, template_name, **kwargs):
 """Convert JSON Jinja2 template to dictionary and apply variables from kwargs."""
 template = env.get_template(template_name)
 return json.loads(template.render(kwargs))

The dataset from GADM: https://gadm.org/data.html is great for a quick visualization. Let’s download it first:

!wget https://biogeo.ucdavis.edu/data/gadm3.6/gpkg/gadm36_AUT_gpkg.zip

and unzip it as the next step.

!unzip gadm36_AUT_gpkg.zip

you receive a geopackage file containing multiple layers.

%ls

gadm36_AUT.gpkg license.txt
gadm36_AUT_gpkg.zip reproducible_kepler.ipynb

data generation

Using geopandas we can load a single layer with the national borders of Austria. As they are represented as two POLYGONS within the MULTIPOLYGON and tools later in the process expect a single geometry and not a geometry collection this needs to be fixed up quickly:

df = gp.read_file('gadm36_AUT.gpkg', driver='GPKG')

# fixup geometries to be a single polygon
polygons = df.geometry.apply(lambda x: list(x))[0]
polygons.append(polygons[0].intersection(polygons[1]).buffer(0.0001))
combined = [unary_union(polygons)]
df.geometry = combined


display(df.head())
df.plot()

	GID_0	NAME_0	geometry
0	AUT	Austria	POLYGON ((10.45455919 47.55573654, 10.45455870...

To fill the shape with hexagons it needs to be converted to geojson first:

gj = gp.GeoSeries([df.geometry[0]]).__geo_interface__
geoJson = gj['features'][0]['geometry']

Then the polyfill function of h3-py can be used. Be aware of the geo_json_conformant parameter. You will most likely need to set it to True to find your hexagons on the right place on the globe, i.e. not flipped.

In case you are in an enterprise setting where the installation of h3 or h3-py fails as the build scripts assume to be run on the open internet: Don’t worry, both packages are available on conda-forge https://github.com/conda-forge/h3-py-feedstock and install nicely now. By the way I am one of the maintainers of these packages.

Let’s generate the hexagons:

res = 8
hexagons = pd.DataFrame(h3.polyfill(geoJson, res, geo_json_conformant=True), columns=['hexagons'])
hexagons['value'] = 1 # some dummy data we want to plot at the map
hexagons.head()

	hexagons	value
0	881f892551fffff	1
1	881e3221b7fffff	1
2	881e336729fffff	1
3	881e150427fffff	1
4	881e105937fffff	1

visualization

The default dark theme of kepler already is quite nice. To demonstrate how to create even more custom visualizations try to use a different basemap one good looking example is the streets default from mapbox available at mapbox://styles/mapbox/streets-v11.

To instanciate a fresh map and load the data run:

prototype_map = KeplerGl(height=700)
prototype_map.add_data(data=hexagons, name=f'res-{res}')
prototype_map

You will receive an empty map. But when clicking on the arrow in the upper left hand corner you should notice that one data file has already been loaded.

A caption

Apply some configuration changes like the suggested change of the background color. Most importantly do not forget to add a new Hexagonal layer to visualize the data. The result should be similar to:

A caption

As the configuration of the visualization has been defined now the time has come to make it reproducible and store it. The python dictionary is stored as a JSON file in the templates directory. You need to create it, if it does not exist yet:

!mkdir templates

Now you can store it:

# uncomment to overwrite existing template
python_dict_to_json_file(prototype_map.config, 'templates/introduction.j2')

reproducibly apply existing configuration

When applying an existing visualization, stored like outlined above, you need to:

Instanciate a new map
Load the data files and add them to a new map
Load the existing jinja template to visualize all the layers

This is achieved with:

from jinja2 import Environment, PackageLoader, select_autoescape, FileSystemLoader
env = Environment(
 loader=FileSystemLoader('templates/'),
 autoescape=select_autoescape(['html', 'xml'])
)

eval_map = KeplerGl(height=700)
eval_map.add_data(data=hexagons, name=f'res-{res}')

config_dict = jinja_json_to_dict(env, 'introduction.j2')
eval_map.config = config_dict
eval_map

Now you should see the same map as clicked together using drag and drop before. As a last step we can store the map in a HTML file to hand it to other people ore share it on a website.

eval_map.save_to_html(file_name='eval_map.html')

In case you do prefer drag and drop tools without code - be aware that kepler.gl recently started to be available as a Tableau plugin: https://extensiongallery.tableau.com/products/108.

summary

It is easy to use kepler.gl, but by itself it is tedious to inclide it inside a workflow where data is updated for an existing visualizatio. Such shortcomings are fixed by the approach outlined above where a minimal amount of python code is used to reproducibly load the configuration of the visualization.

You can extend this and create a true Jinja template with actual variables. Using this you could loop over additional dynamically added data files and generate new visualization layers according to a template.

This was a repost. The original post is found here: https://georgheiler.com/2019/11/26/reproducible-geospatial-visualization-in-kepler.gl/

Geospatial binning with hexagons on spark

Wed, 20 Nov 2019 00:00:00 +0000

Discrete global grid systems recently got quite some attention in the GIS community when Uber released H3 .

It is a hexagonal spatial index. Neighbours can be accessed easily.

Several tasks are supported:

spatial index
spatial smoothing
efficient spatial join

And a variety of libraries around the core c based code has been created. nicely demonstrates how spatial anomaly detection can be quickly built using the python API.

But wouldn’t it also be great to use this functionality in spark? They do provide a java library:

Adding this to spark is straight forward. Using your build tool of choice first add a dependency to the jar.

// https://mvnrepository.com/artifact/com.uber/h3
compile group: 'com.uber', name: 'h3', version: '3.6.0'

or just start an interactive shell such as:

spark-shell --packages 'com.uber:h3:3.6.0'

and run:

import com.uber.h3core.H3Core
import org.apache.spark.sql.expressions.UserDefinedFunction
import org.apache.spark.sql.DataFrame

@transient lazy val h3 = new ThreadLocal[H3Core] {
 override def initialValue() = H3Core.newInstance()
 }


def convertToH3Address(xLong: Double, yLat: Double, precision: Int): String = {
 h3.get.geoToH3Address(yLat, xLong, precision)
 }

val geoToH3Address: UserDefinedFunction = udf(convertToH3Address _)

def createSpatialIndexAddress(result: String,
 XLongColumn: String,
 YLatColumn: String,
 precision: Int)(df: DataFrame): DataFrame = {
 df.withColumn(
 result,
 geoToH3Address(col(XLongColumn), col(YLatColumn), lit(precision)))
 }

which can now be used as:

val df = Seq((1,2), (3,4)).toDF("x", "y")
df.transform(createSpatialIndexAddress("h3","x", "y", 7)).show(false)

+---+---+---------------+
|x |y |h3 |
+---+---+---------------+
|1 |2 |877545796ffffff|
|3 |4 |8775642cdffffff|
+---+---+---------------+

Other functions of the H3 library can be supported similarly. Using the hexagons greatly simplifies efficient spatial smoothing and aggregation (hexbinning).

This was a repost. The original post is found here:

	Source	Destination
0	0	1
16	1	0
17	1	2
18	1	3
19	1	7
20	1	13
21	1	14
22	1	15
23	1	16
24	1	19
26	2	1
36	3	1
51	7	1
68	13	1
73	14	1
75	15	1
78	16	1
84	19	1

	Source	Destination
0	0	1
16	1	0
17	1	2
18	1	3
19	1	7
20	1	13
21	1	14
22	1	15
23	1	16
24	1	19
26	2	1
36	3	1
51	7	1
68	13	1
73	14	1
75	15	1
78	16	1
84	19	1

	Source	Destination
0	0	1
16	1	0
17	1	2
18	1	3
19	1	7
20	1	13
21	1	14
22	1	15
23	1	16
24	1	19
26	2	1
36	3	1
51	7	1
68	13	1
73	14	1
75	15	1
78	16	1
84	19	1