How fast are they?

Added full size version, just click on the above

I posted the above on linkedin earlier this month. I hinted at the code in the header picture, but no code. Ok, so let's get into some code here.

WARNING: Once you discover this API, you are guaranteed to ~~waste~~spend a lot of time with it. It's not too late to turn around, you've been warned!

REST API

The main thing I want to point out tonight, is how easy it is to interact with web services in python. In this case, the Star Wars API (swapi). First, import the usual suspects for visualization and analytics, and add some json handling:

In [1]:

%matplotlib inline
import matplotlib.patches as mpatches
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns

In [2]:

import requests
import json
from pandas.io.json import json_normalize

Ok, what next? Let's first check the swapi api:

In [4]:

r = requests.get('http://swapi.co/api/')
urls = r.json()
urls

Out[4]:

{'films': 'http://swapi.co/api/films/',
 'people': 'http://swapi.co/api/people/',
 'planets': 'http://swapi.co/api/planets/',
 'species': 'http://swapi.co/api/species/',
 'starships': 'http://swapi.co/api/starships/',
 'vehicles': 'http://swapi.co/api/vehicles/'}

Each service will spit back out only part of the data, so we need to follow the link to the next page. Let's write up a function, something quick as a helper. It is not the most efficient, but it is the most readable way of doing this, and for a blog that's important.

In [5]:

def get_swapi(url):
    r = requests.get(url)
    data = r.json()
    df = json_normalize(data['results'])
    while len(df.index) < data['count']:
        r = requests.get(data['next'])
        data = r.json()
        df = pd.concat([df,json_normalize(data['results'])])
    return df

Chewie, we're home

And now to use it:

In [6]:

df = get_swapi(urls['starships'])

Finally, let's clean up the data to correct the 1000km into 1000, remove the unknowns etc, then we can display the full table.

In [7]:

df['max_atmosphering_speed'][df['max_atmosphering_speed']=='1000km'] = 1000
df = df[~(df['hyperdrive_rating']=='unknown')]
df = df[~(df['max_atmosphering_speed'].isin(['unknown','n/a']))]

In [8]:

df['max_atmosphering_speed'] = df['max_atmosphering_speed'].astype(int)

In [9]:

df['hyperdrive_rating'] = df['hyperdrive_rating'].astype(float)
df.sort_values(by='hyperdrive_rating', inplace=True)
df

Out[9]:

	MGLT	cargo_capacity	consumables	cost_in_credits	created	crew	edited	films	hyperdrive_rating	length	manufacturer	max_atmosphering_speed	model	name	passengers	pilots	starship_class	url
2	75	100000	2 months	100000	2014-12-10T16:59:45.094000Z	4	2014-12-22T17:35:44.464156Z	[http://swapi.co/api/films/7/, http://swapi.co...	0.5	34.37	Corellian Engineering Corporation	1050	YT-1300 light freighter	Millennium Falcon	6	[http://swapi.co/api/people/13/, http://swapi....	Light freighter	http://swapi.co/api/starships/10/
0	unknown	unknown	unknown	unknown	2014-12-20T19:55:15.396000Z	3	2014-12-22T17:35:45.258859Z	[http://swapi.co/api/films/6/]	0.5	29.2	Theed Palace Space Vessel Engineering Corps/Nu...	1050	J-type star skiff	Naboo star skiff	3	[http://swapi.co/api/people/10/, http://swapi....	yacht	http://swapi.co/api/starships/64/
7	unknown	unknown	1 year	2000000	2014-12-20T11:05:51.237000Z	5	2014-12-22T17:35:45.124386Z	[http://swapi.co/api/films/5/]	0.7	39	Theed Palace Space Vessel Engineering Corps, N...	2000	J-type diplomatic barge	J-type diplomatic barge	10	[]	Diplomatic barge	http://swapi.co/api/starships/43/
0	unknown	unknown	unknown	unknown	2014-12-20T17:46:46.847000Z	4	2014-12-22T17:35:45.158969Z	[http://swapi.co/api/films/5/]	0.9	47.9	Theed Palace Space Vessel Engineering Corps	8000	H-type Nubian yacht	H-type Nubian yacht	unknown	[http://swapi.co/api/people/35/]	yacht	http://swapi.co/api/starships/49/
2	100	110	5 days	155000	2014-12-20T20:03:48.603000Z	3	2014-12-22T17:35:45.287214Z	[http://swapi.co/api/films/6/]	1.0	14.5	Incom Corporation, Subpro Corporation	1000	Aggressive Reconnaissance-170 starfighte	arc-170	0	[]	starfighter	http://swapi.co/api/starships/66/
1	unknown	60	2 days	320000	2014-12-20T19:56:57.468000Z	1	2014-12-22T17:35:45.272349Z	[http://swapi.co/api/films/6/]	1.0	5.47	Kuat Systems Engineering	1500	Eta-2 Actis-class light interceptor	Jedi Interceptor	0	[http://swapi.co/api/people/10/, http://swapi....	starfighter	http://swapi.co/api/starships/65/
9	unknown	20000000	2 years	59000000	2014-12-20T19:52:56.232000Z	7400	2014-12-22T17:35:45.224540Z	[http://swapi.co/api/films/6/]	1.0	1137	Kuat Drive Yards, Allanteen Six shipyards	975	Senator-class Star Destroyer	Republic attack cruiser	2000	[]	star destroyer	http://swapi.co/api/starships/63/
3	unknown	50000	56 days	1000000	2014-12-20T19:48:40.409000Z	5	2014-12-22T17:35:45.208584Z	[http://swapi.co/api/films/6/]	1.0	18.5	Cygnus Spaceworks	2000	Theta-class T-2c shuttle	Theta-class T-2c shuttle	16	[]	transport	http://swapi.co/api/starships/61/
9	unknown	60	7 days	180000	2014-12-20T17:35:23.906000Z	1	2014-12-22T17:35:45.147746Z	[http://swapi.co/api/films/5/, http://swapi.co...	1.0	8	Kuat Systems Engineering	1150	Delta-7 Aethersprite-class interceptor	Jedi starfighter	0	[http://swapi.co/api/people/10/, http://swapi....	Starfighter	http://swapi.co/api/starships/48/
5	unknown	60	15 hours	102500	2014-12-20T20:43:04.349000Z	1	2014-12-22T17:35:45.396711Z	[http://swapi.co/api/films/6/]	1.0	7.9	Kuat Systems Engineering	1050	Alpha-3 Nimbus-class V-wing starfighter	V-wing	0	[]	starfighter	http://swapi.co/api/starships/75/
4	unknown	65	7 days	200000	2014-12-19T17:39:17.582000Z	1	2014-12-22T17:35:45.079452Z	[http://swapi.co/api/films/5/, http://swapi.co...	1.0	11	Theed Palace Space Vessel Engineering Corps	1100	N-1 starfighter	Naboo fighter	0	[http://swapi.co/api/people/11/, http://swapi....	Starfighter	http://swapi.co/api/starships/39/
0	70	180000	1 month	240000	2014-12-10T15:48:00.586000Z	5	2014-12-22T17:35:44.431407Z	[http://swapi.co/api/films/1/]	1.0	38	Sienar Fleet Systems, Cyngus Spaceworks	1000	Sentinel-class landing craft	Sentinel-class landing craft	75	[]	landing craft	http://swapi.co/api/starships/5/
3	80	110	1 week	134999	2014-12-12T11:00:39.817000Z	2	2014-12-22T17:35:44.479706Z	[http://swapi.co/api/films/3/, http://swapi.co...	1.0	14	Koensayr Manufacturing	1000	BTL Y-wing	Y-wing	0	[]	assault starfighter	http://swapi.co/api/starships/11/
1	120	40	1 week	175000	2014-12-18T11:16:34.542000Z	1	2014-12-22T17:35:44.978754Z	[http://swapi.co/api/films/3/]	1.0	9.6	Alliance Underground Engineering, Incom Corpor...	1300	RZ-1 A-wing Interceptor	A-wing	0	[http://swapi.co/api/people/29/]	Starfighter	http://swapi.co/api/starships/28/
4	100	110	1 week	149999	2014-12-12T11:19:05.340000Z	1	2014-12-22T17:35:44.491233Z	[http://swapi.co/api/films/3/, http://swapi.co...	1.0	12.5	Incom Corporation	1050	T-65 X-wing	X-wing	0	[http://swapi.co/api/people/1/, http://swapi.c...	Starfighter	http://swapi.co/api/starships/12/
5	105	150	5 days	unknown	2014-12-12T11:21:32.991000Z	1	2014-12-22T17:35:44.549047Z	[http://swapi.co/api/films/1/]	1.0	9.2	Sienar Fleet Systems	1200	Twin Ion Engine Advanced x1	TIE Advanced x1	0	[http://swapi.co/api/people/4/]	Starfighter	http://swapi.co/api/starships/13/
8	50	80000	2 months	240000	2014-12-15T13:04:47.235000Z	6	2014-12-22T17:35:44.795405Z	[http://swapi.co/api/films/3/, http://swapi.co...	1.0	20	Sienar Fleet Systems	850	Lambda-class T-4a shuttle	Imperial shuttle	20	[http://swapi.co/api/people/1/, http://swapi.c...	Armed government transport	http://swapi.co/api/starships/22/
6	unknown	2500000	30 days	55000000	2014-12-20T09:39:56.116000Z	1	2014-12-22T17:35:45.105522Z	[http://swapi.co/api/films/4/]	1.5	26.5	Republic Sienar Systems	1180	Star Courier	Scimitar	6	[http://swapi.co/api/people/44/]	Space Transport	http://swapi.co/api/starships/41/
2	unknown	50000000	4 years	125000000	2014-12-20T19:40:21.902000Z	600	2014-12-22T17:35:45.195165Z	[http://swapi.co/api/films/6/]	1.5	1088	Rendili StarDrive, Free Dac Volunteers Enginee...	1050	Providence-class carrier/destroyer	Trade Federation cruiser	48247	[http://swapi.co/api/people/10/, http://swapi....	capital ship	http://swapi.co/api/starships/59/
8	unknown	240	7 days	35700	2014-12-20T18:37:56.969000Z	3	2014-12-22T17:35:45.183075Z	[http://swapi.co/api/films/5/]	1.5	15.2	Huppla Pasa Tisc Shipwrights Collective	1600	Punworcca 116-class interstellar sloop	Solar Sailer	11	[]	yacht	http://swapi.co/api/starships/58/
5	unknown	unknown	unknown	unknown	2014-12-19T17:45:03.506000Z	8	2014-12-22T17:35:45.091925Z	[http://swapi.co/api/films/4/]	1.8	76	Theed Palace Space Vessel Engineering Corps, N...	920	J-type 327 Nubian royal starship	Naboo Royal Starship	unknown	[http://swapi.co/api/people/39/]	yacht	http://swapi.co/api/starships/40/
6	60	3000000	1 year	3500000	2014-12-10T14:20:33.369000Z	165	2014-12-22T17:35:45.408368Z	[http://swapi.co/api/films/6/, http://swapi.co...	2.0	150	Corellian Engineering Corporation	950	CR90 corvette	CR90 corvette	600	[]	corvette	http://swapi.co/api/starships/2/
1	60	36000000	2 years	150000000	2014-12-10T15:08:19.848000Z	47060	2014-12-22T17:35:44.410941Z	[http://swapi.co/api/films/3/, http://swapi.co...	2.0	1,600	Kuat Drive Yards	975	Imperial I-class Star Destroyer	Star Destroyer	0	[]	Star Destroyer	http://swapi.co/api/starships/3/
9	40	6000000	2 years	8500000	2014-12-15T13:06:30.813000Z	854	2014-12-22T17:35:44.848329Z	[http://swapi.co/api/films/3/, http://swapi.co...	2.0	300	Kuat Drive Yards	800	EF76 Nebulon-B escort frigate	EF76 Nebulon-B escort frigate	75	[]	Escort ship	http://swapi.co/api/starships/23/
2	91	45	1 week	220000	2014-12-18T11:18:04.763000Z	1	2014-12-22T17:35:45.011193Z	[http://swapi.co/api/films/3/]	2.0	16.9	Slayn & Korpil	950	A/SF-01 B-wing starfighter	B-wing	0	[]	Assault Starfighter	http://swapi.co/api/starships/29/
3	unknown	unknown	unknown	unknown	2014-12-19T17:01:31.488000Z	9	2014-12-22T17:35:45.027308Z	[http://swapi.co/api/films/4/]	2.0	115	Corellian Engineering Corporation	900	Consular-class cruiser	Republic Cruiser	16	[]	Space cruiser	http://swapi.co/api/starships/31/
7	70	70000	1 month	unknown	2014-12-15T13:00:56.332000Z	1	2014-12-22T17:35:44.716273Z	[http://swapi.co/api/films/5/, http://swapi.co...	3.0	21.5	Kuat Systems Engineering	1000	Firespray-31-class patrol and attack	Slave 1	6	[http://swapi.co/api/people/22/]	Patrol craft	http://swapi.co/api/starships/21/
5	20	19000000	6 months	unknown	2014-12-15T12:34:52.264000Z	6	2014-12-22T17:35:44.680838Z	[http://swapi.co/api/films/3/, http://swapi.co...	4.0	90	Gallofree Yards, Inc.	650	GR-75 medium transport	Rebel transport	90	[]	Medium transport	http://swapi.co/api/starships/17/
4	unknown	140	7 days	168000	2014-12-20T20:38:05.031000Z	1	2014-12-22T17:35:45.381900Z	[http://swapi.co/api/films/6/]	6.0	6.71	Feethan Ottraw Scalable Assemblies	1100	Belbullab-22 starfighter	Belbullab-22 starfighter	0	[http://swapi.co/api/people/10/, http://swapi....	starfighter	http://swapi.co/api/starships/74/

That'll be it for tonight, enjoy playing around with swapi, but as the intro mentions, be warned (and it is the disclaimer I've used each time I've introduced the api into a data science class), you can spend a lot of time with this...

Francois Dion
@f_dion

In an orbit near you

At the last PYPTUG meeting, I demoed Google's Tensorflow deep learning toolkit. I did that using the Jupyter notebook. If you are not familiar with this, check out try.jupyter.org and you'll be able to play with Python, R, Ruby, Scala, Bash etc.

To install jupyter on your computer, pip3 is your friend (more detail at http://jupyter.readthedocs.org/en/latest/install.html):

pip3 install jupyter

By installing jupyter, you'll also get the ipython kernel, so you'll be able to create new jupyter notebooks for python. There are over 50 programming languages supported by jupyter. But that is not all. You can also create specific environments and associate notebooks with them. It works great on pretty much any platform, including the Raspberry Pi, Windows, the Mac, Linux etc. Each kernel has a varying degree of availability, and the same can be said of python modules. Tensorflow will not run on the Pi at this time...

Tensorflow notebook

New notebook dropdown in Jupyter 4

What we are going to do here is to install Tensorflow in a virtual environment, and create a notebook configuration so we can have the choice in the new-> dropdown, as pictured above.

If you've tried to install Tensorflow, particularly on a Mac, you have probably found it challenging, as Tensorflow requires Python 2.7.

Install

On a Mac, since Apple's Python 2.7 is kind of messed up...

brew install python

Create a virtualenv and activate it, then install requirements:

virtualenv -p /usr/local/bin/python2.7 tensor
source tensor/bin/activate

pip install numpy scipy sklearn pandas matplotlib
pip install https://storage.googleapis.com/tensorflow/mac/tensorflow-0.5.0-py2-none-any.whl

Configure Jupyter

I have a globally available Jupyter notebook, on Python 3. This allows me to run various kernels from one notebook. Even virtualenvs, as I'll show here. The below is for Mac. On a Linux or Windows machine it'll be similar: use jupyter --paths to find the data directory (the kernels folder will be in there).

(tensor)LTOXFDION:~ francois$ pip install ipython ipykernel
LTOXFDION:kernels francois$ pwd
/Users/francois/Library/Jupyter/kernels
LTOXFDION:kernels francois$ ls
ir python2
LTOXFDION:kernels francois$ cp -r python2/ tensor
LTOXFDION:kernels francois$ cd tensor/
LTOXFDION:tensor francois$ vi kernel.json

In the editor you'll have to modify the path to python to point to your directory. If you dont have a python2 directory to copy, just create a tensor directory and create the kernel.json file. In the end, your kernel.json file should look something like:

{
 "display_name": "TensorFlow (Py2)",
 "language": "python",
 "argv": [
  "/Users/francois/tensor/bin/python2.7",
  "-c", "from ipykernel.kernelapp import main; main()",
  "-f", "{connection_file}"
 ],
 "codemirror_mode": {
  "version": 2,
  "name": "python"
 }
}

You should be good to go now. Start jupyter:

  jupyter notebook

You'll be able to create a new notebook for Tensorflow. From there, all you have to do is import it:

import tensorflow as tf

We'll continue on this next time.

Francois Dion
@f_dion

Raspberry Pi Python Adventures

Pages

Sunday, September 25, 2016

Something for your mind: Polymath Podcast Episode 001

Two topics will be covered:

Chipmusic, limitations and creativity

Numfocus (Open code = better science)

Wednesday, December 30, 2015

The Star Wars star ships