nathants.com

scaling python data processing vertically

full source code is available here.

processing inconveniently large data is a common task these days, and there are many tools and techniques available to help. here we are going to explore how far we can take python on a single machine.

we'll be working with the nyc taxi dataset in the aws region where it lives, us-east-1. bandwidth between ec2 and s3 is only free within the same region, so make sure you are in us-east-1 if you are following along.

we'll be using some bash functions, aws tooling, and the official aws cli. one could also use other tools without much trouble.

how is the dataset organized?

>> aws s3 ls 's3://nyc-tlc/trip data/' | head

2016-08-11 07:16:22          0
2016-08-11 07:32:21   85733063 fhv_tripdata_2015-01.csv
2016-08-11 07:33:04   97863482 fhv_tripdata_2015-02.csv
2016-08-11 07:33:40  102220197 fhv_tripdata_2015-03.csv
2016-08-11 07:34:24  121250461 fhv_tripdata_2015-04.csv
2016-08-11 07:35:14  133469666 fhv_tripdata_2015-05.csv
2016-08-11 07:35:48  132209226 fhv_tripdata_2015-06.csv
2016-08-11 07:36:09  137153004 fhv_tripdata_2015-07.csv
2016-08-11 07:36:45  164291700 fhv_tripdata_2015-08.csv
2016-08-11 07:37:37  205607912 fhv_tripdata_2015-09.csv

looks like a bunch of csv in a folder. are the prefixes constant?

>> aws s3 ls 's3://nyc-tlc/trip data/' \
    | awk '{print $NF}' \
    | cut -d_ -f1 \
    | sort \
    | uniq -c

      1 0
     64 fhv
     17 fhvhv
     83 green
    138 yellow

nope. we probably want the yellow data. let's check on the sizes first.

>> aws s3 ls 's3://nyc-tlc/trip data/' \
    | grep yellow \
    | awk '{print $3}' \
    | py '"{:,}".format(sum(int(x) for x in i.splitlines()))'

251,267,607,652

looks like about 250GB. what about the others?

>> aws s3 ls 's3://nyc-tlc/trip data/' \
    | awk '{print $NF}' \
    | cut -d_ -f1 \
    | sort -u \
    | tail -n+2 \
    | while read prefix; do
          echo $prefix $(aws s3 ls "s3://nyc-tlc/trip data/${prefix}_" \
                          | awk '{print $3}' \
                          | py '"{:,}".format(sum(int(x) for x in i.splitlines())).rjust(20, ".")')
      done \
    | column -t

fhv     ......37,567,264,171
fhvhv   ......19,542,027,956
green   ......10,381,632,797
yellow  .....251,267,607,652

definitely the yellow dataset then. let's setup some convenience variables.

>> prefix='s3://nyc-tlc/trip data'
>> keys=$(aws s3 ls "$prefix/" \
    | grep yellow \
    | awk '{print $NF}')

let's take a peek at the headers of the first file for each year, selecting the first 10 columns.

>> (for key in $(echo "$keys" | awk 'NR % 12 == 1'); do
       aws s3 cp "$prefix/$key" - 2>/dev/null \
        | head -n1 \
        | cut -d, -f1-8 &
    done; wait) | column -s, -t

VendorID     tpep_pickup_datetime  tpep_dropoff_datetime  passenger_count   trip_distance   pickup_longitude   pickup_latitude     RateCodeID
VendorID     tpep_pickup_datetime  tpep_dropoff_datetime  passenger_count   trip_distance   RatecodeID         store_and_fwd_flag  PULocationID
vendor_id    pickup_datetime       dropoff_datetime       passenger_count   trip_distance   pickup_longitude   pickup_latitude     rate_code
vendor_id    pickup_datetime       dropoff_datetime       passenger_count   trip_distance   pickup_longitude   pickup_latitude     rate_code
VendorID     tpep_pickup_datetime  tpep_dropoff_datetime  passenger_count   trip_distance   RatecodeID         store_and_fwd_flag  PULocationID
vendor_id    pickup_datetime       dropoff_datetime       passenger_count   trip_distance   pickup_longitude   pickup_latitude     rate_code
VendorID     tpep_pickup_datetime  tpep_dropoff_datetime  passenger_count   trip_distance   pickup_longitude   pickup_latitude     RatecodeID
vendor_id    pickup_datetime       dropoff_datetime       passenger_count   trip_distance   pickup_longitude   pickup_latitude     rate_code
VendorID     tpep_pickup_datetime  tpep_dropoff_datetime  passenger_count   trip_distance   RatecodeID         store_and_fwd_flag  PULocationID
vendor_name  Trip_Pickup_DateTime  Trip_Dropoff_DateTime  Passenger_Count   Trip_Distance   Start_Lon          Start_Lat           Rate_Code
VendorID     tpep_pickup_datetime  tpep_dropoff_datetime  passenger_count   trip_distance   RatecodeID         store_and_fwd_flag  PULocationID
vendor_id    pickup_datetime       dropoff_datetime       passenger_count   trip_distance   pickup_longitude   pickup_latitude     rate_code

looks like the first 5 columns are consistent, and then it gets messy. we can punt on data cleanup by just working with those first 5, which contain interesting data like distance, passengers, and date.

before we jump on ec2, let's grab the first million rows of the first file to our local environment and prototype our data scripts.

>> aws s3 cp "$prefix/$(echo $keys | awk '{print $1}')" - 2>/dev/null \
    | head -n1000000 \
    > /tmp/taxi.csv

>> ls -lh /tmp/taxi.csv | awk '{print $5}'

172M

>> head /tmp/taxi.csv \
    | cut -d, -f1-5 \
    | column -s, -t

vendor_name  Trip_Pickup_DateTime  Trip_Dropoff_DateTime  Passenger_Count  Trip_Distance
VTS          2009-01-04 02:52:00   2009-01-04 03:02:00    1                2.6299999999999999
VTS          2009-01-04 03:31:00   2009-01-04 03:38:00    3                4.5499999999999998
VTS          2009-01-03 15:43:00   2009-01-03 15:57:00    5                10.35
DDS          2009-01-01 20:52:58   2009-01-01 21:14:00    1                5
DDS          2009-01-24 16:18:23   2009-01-24 16:24:56    1                0.40000000000000002
DDS          2009-01-16 22:35:59   2009-01-16 22:43:35    2                1.2
DDS          2009-01-21 08:55:57   2009-01-21 09:05:42    1                0.40000000000000002
VTS          2009-01-04 04:31:00   2009-01-04 04:36:00    1                1.72

now that we have data, it's time to ask questions. let's group by passengers and count.

first let's try python's csv module.

# passenger_counts_stdlib.py
import csv
import sys
import collections

sys.stdin.readline() # skip the header

result = collections.defaultdict(int)

for cols in csv.reader(sys.stdin):
    try:
        passengers = cols[3]
    except IndexError:
        continue
    else:
        result[passengers] += 1

for passengers, count in result.items():
    print(f'{passengers},{count}')

>> time cat /tmp/taxi.csv \
    | python3 passenger_counts_stdlib.py \
    | sort -nr -k2 -t, \
    | column -s, -t

1  669627
2  166658
5  93718
3  44360
4  20904
6  4685
0  46

real    0m2.316s
user    0m2.259s
sys     0m0.162s

let's see how pandas compares.

# passenger_counts_pandas.py
import pandas
import sys

df = pandas.read_csv(sys.stdin)

print(df.iloc[:,3].value_counts())

>> time cat /tmp/taxi.csv | python3 passenger_counts_pandas.py

1    669627
2    166658
5     93718
3     44360
4     20904
6      4685
0        46
Name: Passenger_Count   dtype: int64

real    0m2.164s
user    0m2.085s
sys     0m0.499s

about the same.

if we know that our input is well formed, without quotes or escaped delimiters, we can just split on comma. let's try that.

# passenger_counts.py
import sys
import collections

sys.stdin.readline() # skip the header

result = collections.defaultdict(int)

for line in sys.stdin:
    cols = line.split(',')
    try:
        passengers = cols[3]
    except IndexError:
        continue
    else:
        result[passengers] += 1

for passengers, count in result.items():
    print(f'{passengers},{count}')

>> time cat /tmp/taxi.csv \
    | python3 passenger_counts.py \
    | sort -nr -k2 -t, \
    | column -s, -t

1  669627
2  166658
5  93718
3  44360
4  20904
6  4685
0  46

real    0m0.668s
user    0m0.633s
sys     0m0.099s

that is a lot faster, about x4. if we can safely assume that the data is well formed, simple split looks like a good idea. after peeking at this dataset for the fields we care about, this is likely ok.

let's run it again with x25 more data by repeating the input over and over. using tail we can skip the header in all but the first input.

>> time (cat /tmp/taxi.csv; for i in {1..24}; do tail -n+2 /tmp/taxi.csv; done) \
    | python3 passenger_counts.py &>/dev/null

real    0m16.295s
user    0m16.101s
sys     0m2.771s

what if we try pypy?

>> time (cat /tmp/taxi.csv; for i in {1..24}; do tail -n+2 /tmp/taxi.csv; done) \
    | pypy3 passenger_counts.py &>/dev/null

real    0m17.260s
user    0m16.386s
sys     0m4.011s

well that's not ideal. let's see if we can apply performance lessons from compiled languages, which can be summarized as avoid allocations and do as little work as possible. the following file has some boiler plate elided, refer to the full source for the details.

# passenger_counts_inlined.py
...

result = collections.defaultdict(int)

... # FOR ROW IN STDIN
    ...

    if max >= 3:
        passengers = read_buffer[starts[3]:ends[3]]
        result[passengers] += 1

...

for passengers, count in result.items():
    print(f'{passengers.decode()},{count}')

>> time (cat /tmp/taxi.csv; for i in {1..24}; do tail -n+2 /tmp/taxi.csv; done) \
    | pypy3 passenger_counts_inlined.py &>/dev/null

real    0m10.245s
user    0m8.876s
sys     0m3.108s

a x2 improvement on user time, and nearly as much on wall clock. we'll take it. if interested, see further optimizations in go, rust, and c.

a final optimization we can make is to work with less data. since we know we only care about the first 5 columns, we can drop unused data upstream.

>> time cat /tmp/taxi.csv | cut -d, -f1-5 > /tmp/taxi.csv.slim

real    0m0.409s
user    0m0.359s
sys     0m0.140s

>> time (cat /tmp/taxi.csv.slim; for i in {1..24}; do tail -n+2 /tmp/taxi.csv.slim; done) \
    | pypy3 passenger_counts_inlined.py &>/dev/null

real    0m3.764s
user    0m3.196s
sys     0m1.155s

another x2 improvement, we'll take it.

our first significant improvement we got by avoiding allocations, and here we get another one by dropping unused data upstream.

let's take another look at our improvements.

>> time (cat /tmp/taxi.csv; for i in {1..24}; do tail -n+2 /tmp/taxi.csv; done) \
    | python3 passenger_counts_stdlib.py &>/dev/null

real    0m57.986s
user    0m57.854s
sys     0m3.610s

>> time (cat /tmp/taxi.csv.slim; for i in {1..24}; do tail -n+2 /tmp/taxi.csv.slim; done) \
    | pypy3 passenger_counts_inlined.py &>/dev/null

real    0m3.726s
user    0m3.401s
sys     0m0.907s

by doing less work, manually inlining code, avoiding allocations, and reducing the data set, we can get sizeable performance improvements.

just for fun, let's take a look at going even faster. we'll explore this in a later post

>> cat /tmp/taxi.csv \
    | tail -n+2 \
    | bsv \
    | bschema *,*,*,*,*,... --filter \
    > /tmp/taxi.bsv.slim

>> time (for i in {1..25}; do cat /tmp/taxi.bsv.slim; done) \
    | bcut 4 \
    | bcounteach-hash >/dev/null

real    0m0.742s
user    0m0.801s
sys     0m0.950s

system time as the bottleneck is a really good problem to have.

back to python, it's time to deploy and scale vertically. first we're going to need an ec2 instance. let's use a i3en.24xlarge with archlinux.

>> export region=us-east-1

>> aws-ec2-max-spot-price i3en.24xlarge

on demand: 10.848, spot offers 70% savings
us-east-1a 3.254400
us-east-1b 3.254400
us-east-1c 3.254400
us-east-1d 3.254400
us-east-1f 3.254400

looks like cost will be $3/hour.

our machine is going to need s3 access to get the dataset, so let's make an instance profile.

>> aws-iam-ensure-instance-profile \
    --policy AmazonS3ReadOnlyAccess \
    s3-readonly

we are also going to need a vpc, keypair, and security group access for port 22. if you already have aws setup you're probably fine, otherwise do something like this.

>> aws-vpc-new adhoc-vpc

>> aws-ec2-authorize-ip $(curl checkip.amazonaws.com) adhoc-vpc --yes

>> aws-ec2-keypair-new $(whoami) ~/.ssh/id_rsa.pub

before we start, let's note the time.

>> start=$(date +%s)

now it's time to spin up our machine.

>> time id=$(aws-ec2-new --type i3en.24xlarge \
                         --ami arch \
                         --profile s3-readonly \
                         test-machine)

real    1m10.673s
user    0m2.510s
sys     0m0.434s

it takes a moment to format the instance store ssd, so we wait.

>> aws-ec2-ssh $id --yes --cmd '
       while true; do
           sleep 1
           df -h | grep /mnt && break
       done
   '

now we need to install some things.

>> aws-ec2-ssh $id --yes --cmd '
       sudo pacman -Sy --noconfirm python-pip pypy3 git
       sudo pip install awscli git+https://github.com/nathants/py-{util,shell,pool}
   '

then we bump linux limits, reboot, and wait for the machine to come back up.

>> aws-ec2-ssh $id --yes --cmd '
       curl -s https://raw.githubusercontent.com/nathants/bootstraps/master/scripts/limits.sh | bash
       sudo reboot
   '

>> aws-ec2-wait-for-ssh $id --yes

baking an ami instead of starting from vanilla linux can save some bootstrap time.

let's deploy our code.

>> aws-ec2-scp passenger_counts_inlined.py :/mnt $id --yes

our data pipeline is going to look like:

fetch the dataset
select the columns we need
group by and count
merge results

step 1 will fetch and select passengers. this pipeline will run once per input key, and will run in parallel on all cpus.

# download_and_select.py
import os
import shell
import pool.thread

shell.run('mkdir -p /mnt/data')

prefix = "s3://nyc-tlc/trip data"

keys = [x.split()[-1] for x in shell.run(f'aws s3 ls "{prefix}/"').splitlines() if 'yellow' in x]

def download(key):
    shell.run(f'aws s3 cp "{prefix}/{key}" - | cut -d, -f1-5 > /mnt/data/{key}', echo=True)

pool.thread.size = os.cpu_count()

list(pool.thread.map(download, keys))

>> aws-ec2-scp download_and_select.py :/mnt $id --yes

>> time aws-ec2-ssh $id --yes --cmd 'python /mnt/download_and_select.py'

real    1m43.209s
user    0m0.371s
sys     0m0.214s

step 2 will group by passengers and count. this pipeline will run once per input file, and will run in parallel on all cpus.

we'll use shell redirection instead of cat for the input since it's more efficient.

# group_and_count.py
import os
import shell
import pool.thread

shell.run('mkdir -p /mnt/results')

paths = shell.files('/mnt/data', abspath=True)

def process(path):
    shell.run(f'< {path} pypy3 /mnt/passenger_counts_inlined.py > /mnt/results/{os.path.basename(path)}', echo=True)

pool.thread.size = os.cpu_count()

list(pool.thread.map(process, paths))

>> aws-ec2-scp group_and_count.py :/mnt $id --yes

>> time aws-ec2-ssh $id --yes --cmd 'python /mnt/group_and_count.py'

real    0m11.062s
user    0m0.262s
sys     0m0.018s

step 3 will merge the results from step 2. we haven't actually written this code yet, so let's do that now. this pipeline runs on a single core and takes all results as input.

# merge_results.py
import sys
import collections

result = collections.defaultdict(int)

for line in sys.stdin:
    passengers, count = line.split(',')
    result[passengers] += int(count)

for passengers, count in result.items():
    print(f'{passengers},{count}')

>> aws-ec2-scp merge_results.py :/mnt $id --yes

>> time aws-ec2-ssh $id --yes --cmd '
       cat /mnt/results/* \
         | python /mnt/merge_results.py \
         | tr , " " \
         | sort -nrk 2 \
         | head -n9 \
         | column -t
   '

real    0m1.580s
user    0m0.189s
sys     0m0.038s

a final optimization we can apply here is to combine steps 1 and 2, which will avoid iowait as a bottleneck since we never touch local disk.

# combined.py
import os
import shell
import pool.thread

shell.run('mkdir -p /mnt/results')

prefix = "s3://nyc-tlc/trip data"

keys = [x.split()[-1] for x in shell.run(f'aws s3 ls "{prefix}/"').splitlines() if 'yellow' in x]

def process(key):
    shell.run(f'aws s3 cp "{prefix}/{key}" - '
              f'| cut -d, -f1-5'
              f'| pypy3 /mnt/passenger_counts_inlined.py'
              f'> /mnt/results/{key}',
              echo=True)

pool.thread.size = os.cpu_count()

list(pool.thread.map(process, keys))

>> aws-ec2-scp combined.py :/mnt $id --yes

>> time aws-ec2-ssh $id --yes --cmd 'python /mnt/combined.py'

real    0m53.036s
user    0m0.334s
sys     0m0.069s

interesting. reading from the network is faster than writing to disk, and in this case get's us a x2 wall clock improvement.

since we are paying $3/hour for this instance, let's shut it down.

>> aws-ec2-rm $id --yes

let's see how much money we spent getting this result.

>> echo job took $(( ($(date +%s) - $start) / 60 )) minutes

job took 6 minutes

for less than $1, we analyzed a 250GB dataset with python. an individual query took as little as 10 seconds reading from local disk, or 60 seconds reading from s3. vertical scaling with python is a decent technique. now that we've maxed out our instance size, the only way to scale further is to go horizontal.

when analyzing data, it's always good to check the results with an alternate implementation. if they disagree, at least one of them is wrong. you can find alternate implementations of this analysis here.

scaling python data processing horizontally

full source code is available here.

we scaled an analysis of the nyc taxi dataset vertically on a single machine, now let's scale horizontally on multiple machines. instead of a single i3en.24xlarge we'll use twelve i3en.2xlarge.

we'll be using some aws tooling and the official aws cli. one could also use other tools without much trouble.

we'll be using the same code and aws setup from vertical scaling.

first we're going to need some ec2 instances.

>> export region=us-east-1

>> aws-ec2-max-spot-price i3en.2xlarge

on demand: 0.904, spot offers 70% savings
us-east-1b 0.271200
us-east-1c 0.271200
us-east-1d 0.271200
us-east-1f 0.272200
us-east-1a 0.288600

at about $0.25/hour/instance cost will be $3/hour.

before we start, let's note the time.

>> start=$(date +%s)

now it's time to spin up our machines. the following may look familiar. it is almost identical to how we instantiated our machine for vertical scaling, except that we capture and use multiple instance $ids instead of just one $id.

>> time ids=$(aws-ec2-new --type i3en.2xlarge \
                          --num 12 \
                          --ami arch \
                          --profile s3-readonly \
                          test-machines)

real    1m57.050s
user    0m3.154s
sys     0m0.744s

it takes a moment to format the instance store ssd, so we wait.

>> aws-ec2-ssh $ids --yes --cmd '
       while true; do
           sleep 1
           df -h | grep /mnt && break
       done
   '

now we need to install some things.

>> aws-ec2-ssh $ids --yes --cmd '
       sudo pacman -Sy --noconfirm python-pip pypy3 git
       sudo pip install awscli git+https://github.com/nathants/py-{util,shell,pool}
   '

then we bump linux limits, reboot, and wait for the machines to come back up.

>> aws-ec2-ssh $ids --yes --cmd '
       curl -s https://raw.githubusercontent.com/nathants/bootstraps/master/scripts/limits.sh | bash
       sudo reboot
   '

>> aws-ec2-wait-for-ssh $ids --yes

baking an ami instead of starting from vanilla linux can save some bootstrap time.

our data pipeline is going to look like:

fetch the dataset
select the columns we need
group by and count
merge results

step 1 will fetch and select passengers. this pipeline will run once per input key, and will run in parallel on all cpus of every machine.

# download_and_select.py
import os
import shell
import pool.thread
import sys

shell.run('mkdir -p /mnt/data')

prefix = "s3://nyc-tlc/trip data"

keys = sys.stdin.read().splitlines()

def download(key):
    shell.run(f'aws s3 cp "{prefix}/{key}" - | cut -d, -f1-5 > /mnt/data/{key}', echo=True)

pool.thread.size = os.cpu_count()

list(pool.thread.map(download, keys))

since we are running on multiple machines, we'll need to orchestrate the activity. we'll be using a local process and ssh. the local process will divide the keys to process across the machines and monitor their execution.

# orchestrate_download_and_select.py
import shell
import pool.thread
import util.iter
import sys

ids = sys.argv[1:]

prefix = "s3://nyc-tlc/trip data"

keys = [x.split()[-1] for x in shell.run(f'aws s3 ls "{prefix}/"').splitlines() if 'yellow' in x]

def download(arg):
    id, keys = arg
    keys = '\n'.join(keys)
    shell.run(f'aws-ec2-ssh {id} --yes --cmd "python /mnt/download_and_select.py" --stdin "{keys}" >/dev/null', stream=True)

pool.thread.size = len(ids)

args = zip(ids, util.iter.chunks(keys, num_chunks=len(ids)))

list(pool.thread.map(download, args))

>> aws-ec2-scp passenger_counts_inlined.py :/mnt $ids --yes

>> aws-ec2-scp download_and_select.py :/mnt $ids --yes

>> time python orchestrate_download_and_select.py $ids

real    1m23.778s
user    0m4.588s
sys     0m1.950s

step 2 will group by passengers and count. this pipeline will run once per input file, and will run in parallel on all cpus of every machine.

# group_and_count.py
import os
import shell
import pool.thread

shell.run('mkdir -p /mnt/results')

paths = shell.files('/mnt/data', abspath=True)

def process(path):
    shell.run(f'< {path} pypy3 /mnt/passenger_counts_inlined.py > /mnt/results/{os.path.basename(path)}', echo=True)

pool.thread.size = os.cpu_count()

list(pool.thread.map(process, paths))

the local machine will orchestrate.

# orchestrate_group_and_count.py
import shell
import pool.thread
import sys

ids = sys.argv[1:]

def process(id):
    shell.run(f'aws-ec2-ssh {id} --yes --cmd "python /mnt/group_and_count.py" >/dev/null', stream=True)

pool.thread.size = len(ids)

list(pool.thread.map(process, ids))

>> aws-ec2-scp group_and_count.py :/mnt $ids --yes

>> time python orchestrate_group_and_count.py $ids

real    0m17.984s
user    0m2.980s
sys     0m0.933s

step 3 will merge the results. this pipeline runs locally on a single core after fetching results from all machines with rsync.

# merge_results.py
import sys
import collections
import shell

ids = ' '.join(sys.argv[1:])

with shell.tempdir():
    shell.run(f'aws-ec2-rsync :/mnt/results/ ./results/ --yes {ids} 1>&2', stream=True)

    result = collections.defaultdict(int)

    for path in shell.files('results', abspath=True):
        with open(path) as f:
            for line in f:
                passengers, count = line.split(',')
                result[passengers] += int(count)

for passengers, count in result.items():
    print(f'{passengers},{count}')

>> time python merge_results.py $ids \
    | tr , " " \
    | sort -nrk 2 \
    | head -n9 \
    | column -t

real    0m2.638s
user    0m0.465s
sys     0m0.095s

an optimization we can apply here is to combine steps 1 and 2, which will avoid iowait as a bottleneck since we never touch local disk.

# combined.py
import os
import shell
import pool.thread
import sys

shell.run('mkdir -p /mnt/results')

prefix = "s3://nyc-tlc/trip data"

keys = sys.stdin.read().splitlines()

def process(key):
    shell.run(f'aws s3 cp "{prefix}/{key}" - '
              f'| cut -d, -f1-5'
              f'| pypy3 /mnt/passenger_counts_inlined.py'
              f'> /mnt/results/{key}',
              echo=True)

pool.thread.size = os.cpu_count()

list(pool.thread.map(process, keys))

# orchestrate_combined.py
import shell
import pool.thread
import util.iter
import sys

ids = sys.argv[1:]

prefix = "s3://nyc-tlc/trip data"

keys = [x.split()[-1] for x in shell.run(f'aws s3 ls "{prefix}/"').splitlines() if 'yellow' in x]

def process(arg):
    id, keys = arg
    keys = '\n'.join(keys)
    shell.run(f'aws-ec2-ssh {id} --yes --cmd "python /mnt/combined.py" --stdin "{keys}" >/dev/null', stream=True)

pool.thread.size = len(ids)

args = zip(ids, util.iter.chunks(keys, num_chunks=len(ids)))

list(pool.thread.map(process, args))

>> aws-ec2-scp combined.py :/mnt $ids --yes

>> time python orchestrate_combined.py $ids

real    1m19.735s
user    0m4.867s
sys     0m1.949s

since we are paying $3/hour for this instance, let's shut it down.

>> aws-ec2-rm $ids --yes

lets see how much money we spent getting this result.

>> echo job took $(( ($(date +%s) - $start) / 60 )) minutes

job took 8 minutes

for less than $1, we analyzed a 250GB dataset with python on a cluster of twelve machines. an individual query took as little as 18 seconds reading from local disk, or 80 seconds reading from s3.

interestingly, this is up from 10 seconds and 60 seconds respectively in the vertical scaling post, suggesting that both network and disk performance varies with instance size.

we've iterated rapidly on local code with a sample of data, and in production with all of the data. we've experimented with several options for a simple data pipeline on a large single machine and on multiple small machines. we've answered some questions, and discovered more. we did all of this simply, quickly, and for less than the cost of a cup of coffee. most importantly, it was fun.

just for fun, let's try vertical and horizontal scaling together with four i3en.24xlarge. we'll be using the basic ami instead of live bootstrapping.

>> aws-ec2-max-spot-price i3en.24xlarge

on demand: 10.848, spot offers 70% savings
us-east-1a 3.254400
us-east-1b 3.254400
us-east-1c 3.254400
us-east-1d 3.254400
us-east-1f 3.254400

>> start=$(date +%s)

>> time ids=$(aws-ec2-new --type i3en.24xlarge \
                          --num 4 \
                          --ami basic \
                          --profile s3-readonly \
                          test-machines)

real    4m11.740s
user    0m5.453s
sys     0m1.354s

>> aws-ec2-ssh $ids --yes --cmd '
       while true; do
           sleep 1
           df -h | grep /mnt && break
       done
   '

>> aws-ec2-scp passenger_counts_inlined.py :/mnt $ids --yes

>> aws-ec2-scp combined.py :/mnt $ids --yes

>> time python orchestrate_combined.py $ids

real    0m32.145s
user    0m1.478s
sys     0m0.572s

>> time python merge_results.py $ids \
    | tr , " " \
    | sort -nrk 2 \
    | head -n9 \
    | column -t

real    0m2.527s
user    0m0.336s
sys     0m0.057s

>> aws-ec2-ssh $ids --yes --cmd 'sudo poweroff'

>> echo job took $(( ($(date +%s) - $start) / 60 )) minutes

job took 6 minutes

30 seconds, interesting. it's only x2 faster, not the x4 we might expect, than the single machine used in vertical scaling. i wonder why?

refactoring common distributed data patterns into s4

full source code is available here.

in horizontal scaling we manually managed distributed compute across a cluster of machines. we used ssh to execute commands. we created directories and files to hold results. we used rsync to fetch results from multiple machines and merged them locally. we manually managed parallelism in our data scripts.

this wasn't particularly difficult, but neither was it important. let's refactor and build some tooling so next time we can focus more on the data and less on low level details of distributed compute.

our data pipeline looked like:

fetch the dataset
select the columns we need
group by and count
merge results

let's break that down a bit.

input	command	output
files	fetch	files
files	select columns	files
files	group and count	files
files	merge results	file

it looks like we have two types of things going on.

first we have a 1:1 map of input files to output files through a command. we can imagine it as:

for file in inputs/*; do
    cat $file | $command > outputs/$(basename $file)
done

second we have a n:1 map of input files to output file though a command. we can imagine it as:

cat inputs/* | $command > output

we don't have it in this pipeline, but we can imagine a third type as a 1:n map of input file to output files through a command:

cat input | $command --outdir=outputs/

let's code by wishful thinking. what would our pipeline look like if we had something that helped us do these three types of things? let's imagine something like s3.

first we fetch the dataset. our inputs will be keys, the outputs will be the key data, and the command will be copy. first we need to put the inputs.

>> prefix='s3://nyc-tlc/trip data'

>> keys=$(aws s3 ls "$prefix/" \
           | grep yellow \
           | awk '{print $NF}')

>> for key in $key; do
      echo "$prefix/$key" | aws s3 cp - s3://inputs/$key
   done

>> aws s3 ls s3://inputs/ | head -n3

yellow_tripdata_2009-01.csv
yellow_tripdata_2009-02.csv
yellow_tripdata_2009-03.csv

now that we have our inputs, we can do a 1:1 map.

>> aws s3 map \
    --in  s3://inputs/ \
    --out s3://step1/ \
    --cmd 'cat > key && aws s3 cp $(cat key) -'

next we select the columns with a 1:1 map.

>> aws s3 map \
    --in  s3://step1/ \
    --out s3://step2/ \
    --cmd "cut -d, -f1-5"

next we group and count with a 1:1 map.

>> aws s3 map \
    --in  s3://step2/ \
    --out s3://step3/ \
    --cmd "pypy3 passenger_counts_inlined.py"

finally we merge the results with a n:1 map, and fetch the result.

>> aws s3 map-from-n \
    --in  s3://step3/ \
    --out s3://result \
    --cmd "python merge_results.py"

>> aws s3 cp s3://result -

let's put that all together and see what we've got.

>> aws s3 map        --in s3://inputs/ --out s3://step1/ --cmd 'cat > key && aws s3 cp $(cat key) -'
>> aws s3 map        --in s3://step1/  --out s3://step2/ --cmd 'cut -d, -f1-5'
>> aws s3 map        --in s3://step2/  --out s3://step3/ --cmd 'pypy3 passenger_counts_inlined.py'
>> aws s3 map-from-n --in s3://step3/  --out s3://result --cmd 'python merge_results.py'
>> aws s3 cp s3://result -

now we have a series of steps, mapping immutable inputs to immutable outputs. we have no details of infrastructure, data location, or data transfer. we can imagine taking any of these commands and running them locally to debug or optimize. this feels better than threadpools, rsync, and ssh. it's too bad none of this works.

s3 is a pinnacle of modern engineering. it scales automatically, is comically durable, quite available, and significantly cheaper than ebs. in it's standard storage class it replicates across availability zones without bandwidth charges. within the same region, bandwidth between ec2 and s3 is free.

we want to use s3 for durability and scalability. we also want simple distributed compute like we imagined above. let's spin up a system to compliment s3. we'll call it s4.

for a moment, let's think about scope reduction and what we don't want.

we don't want it to be highly durable or available, because we already have s3 for that.
we don't want it to use complicated failure handling, because we can retry idempotent commands on immutable data.
we don't want it to handle security or authentication, because those can be network level concerns.
we don't want it to allow updates to data unless explicitly deleted, because immutability is a simplifying constraint.

this narrower scope means the system is easier to use, has simpler implementation, and is more likely to be correct.

let's give it a try. first we install s4 and then spin up a cluster. we'll size the cluster the same as we did in horizontal scaling.

>> git clone https://github.com/nathants/s4

>> cd s4

>> python3 -m pip install -r requirements.txt .

>> export region=us-east-1

>> name=s4-cluster

>> time type=i3en.xlarge num=12 bash scripts/new_cluster.sh $name

5m17.052s

next we'll proxy traffic through a machine in the cluster. assuming the security group only allows port 22, the machines are only accessible on their internal addresses. since we already have ssh setup, we'll use sshuttle. run this in a second terminal, and don't forget to set region to us-east-1.

>> export region=us-east-1

>> name=s4-cluster

>> bash scripts/connect_to_cluster.sh $name

let's check the cluster health.

>> s4 health

healthy:   10.0.30.103:8080
healthy:   10.0.18.21:8080
healthy:   10.0.29.44:8080
healthy:   10.0.22.60:8080
healthy:   10.0.28.41:8080
healthy:   10.0.29.17:8080
healthy:   10.0.18.163:8080
healthy:   10.0.24.118:8080
healthy:   10.0.22.203:8080
healthy:   10.0.19.10:8080
healthy:   10.0.26.213:8080
healthy:   10.0.28.124:8080

we want to be able to place keys on machines. we'll use consistent hashing to automatically place or [numeric prefixes](https://github.com/nathants/s4/search?q=%22func KeyPrefix%22&type=Code) to explicitly place keys around the cluster.

we want to be able to put, get, and list keys across a cluster of machines. let's try putting some data which is explicitly placed with numeric prefixes.

>> echo input_a | s4 cp - s4://inputs/000_machine0

>> echo input_b | s4 cp - s4://inputs/001_machine1

we want to be able to map 1:1. let's try replacing some text.

>> s4 map s4://inputs/ s4://mapped/ "sed s/input/output/"

>> for key in $(s4 ls -r s4://mapped/ | awk '{print $NF}'); do
       echo $key '=>' $(s4 eval s4://mapped/$key cat)
   done

000_machine0 => output_a
001_machine1 => output_b

we want to be able to map 1:n to shuffle data around the cluster. each input key becomes an output directory filled with keys that will be placed around the cluster according to their name. let's try duplicating some content from the first two machines in the cluster to the next two.

>> s4 map-to-n s4://inputs/ s4://shuffled/ '
       cat > content
       for i in {2..3}; do
           file=$(printf "%03d" $i)_machine$i
           echo -n "$(cat content)$i" > $file
           echo $file
       done
   '

>> for key in $(s4 ls -r s4://shuffled/ | awk '{print $NF}'); do
       echo $key '=>' $(s4 eval s4://shuffled/$key cat)
   done

000_machine0/002_machine2 => input_a2
000_machine0/003_machine3 => input_a3
001_machine1/002_machine2 => input_b2
001_machine1/003_machine3 => input_b3

we want to be able to merge shuffled data with n:1 maps. let's merge the content we just duplicated.

>> s4 map-from-n s4://shuffled/ s4://merged/ "xargs cat"

>> for key in $(s4 ls -r s4://merged/ | awk '{print $NF}'); do
       echo $key '=>' $(s4 eval s4://merged/$key cat)
   done

002_machine2 => input_a2 input_b2
003_machine3 => input_a3 input_b3

all files with the same name have been merged into a single file with that name.

now that we've seen all of the maps in action, let's summarize their semantics.

maps take directories as inputs and outputs and operate on the keys in those directories.
1:1 map commands take data on stdin and emit data on stdout.
1:n map commands take data on stdin, write files to disk, and emit file names on stdout.
n:1 map commands take file names on stdin, and emit data on stdout.

key names are important, since they define data placement around the cluster.

1:1 map operates on keys on a single machine.
1:n map will shuffle output keys around the cluster.
n:1 map operates on keys on a single machine.

now let's try redoing the analysis from horizontal scaling with s4.

we'll be using some aws tooling and the official aws cli. one could also use other tools without much trouble.

we've already spun up an s4 cluster in us-east-1, but let's delete it and make a new one. clusters spin up fast and should only contain ephemeral data. they spin up even faster when using a prebuilt ami instead of live bootstrapping.

>> export region=us-east-1

>> name=s4-cluster

>> aws-ec2-rm $name --yes

>> time ami=s4 type=i3en.xlarge num=12 bash scripts/new_cluster.sh $name

3m43.205s

first we deploy our code to every machine. note that we'll be referring to ec2 instances by name instead of id.

>> aws-ec2-scp passenger_counts_inlined.py :/mnt $name --yes

now we add the s3 keys of the input data to s4 so that we can map over them.

>> prefix='s3://nyc-tlc/trip data'

>> keys=$(aws s3 ls "$prefix/" \
           | grep yellow \
           | awk '{print $NF}' \
           | while read key; do
                 echo "$prefix/$key"
             done)

>> echo "$keys" | while read key; do
       echo "$key"
       echo "$key" | s4 cp - s4://inputs/$(basename "$key")
   done

let's take a peek at the data.

>> s4 ls s4://inputs/ | head -n3 | awk '{print $NF}'

yellow_tripdata_2009-01.csv
yellow_tripdata_2009-02.csv
yellow_tripdata_2009-03.csv

>> s4 eval s4://inputs/yellow_tripdata_2009-01.csv cat

s3://nyc-tlc/trip data/yellow_tripdata_2009-01.csv

now let's run our data pipeline.

>> time s4 map s4://inputs/ s4://step1/ 'cat > url && aws s3 cp "$(cat url)" -'

1m4.920s

>> time s4 map s4://step1/ s4://step2/ 'cut -d, -f1-5'

0m46.054s

>> time s4 map s4://step2/ s4://step3/ 'pypy3 /mnt/passenger_counts_inlined.py'

0m20.310s

we can't merge our results until they are all on one machine, so we need to map 1:n, where n=1, sending all results to the same machine. to do this we are putting all data into keys with the same name, which places them on the same machine.

>> time s4 map-to-n s4://step3/ s4://step4/ 'cat > results && echo results'

0m1.729s

now that all results are on the same machine, we can merge the results with a n:1 map.

# merge_results.py
import sys
import collections
import shell

result = collections.defaultdict(int)

for line in sys.stdin:
    passengers, count = line.split(',')
    result[passengers] += int(count)

for passengers, count in result.items():
    print(f'{passengers},{count}')

>> aws-ec2-scp merge_results.py :/mnt $name --yes

>> time s4 map-from-n s4://step4/ s4://step5/ 'xargs cat | python /mnt/merge_results.py'

0m0.464s

finally we fetch the result.

>> s4 eval s4://step5/results "
       tr , ' ' \
       | sort -nrk2 \
       | head -n9 \
       | column -t
   "

1  1135227331
2  239684017
5  103036920
3  70434390
6  38585794
4  34074806
0  6881330
7  2040
8  1609

let's run at the pipeline again. note that keys cannot be updated, so before we can rerun the pipeline we have to delete intermediate results. we'll delete everything except the inputs.

>> s4 rm -r s4://step

>> time s4 map        s4://inputs/ s4://step1/ 'cat > url && aws s3 cp "$(cat url)" -'

1m5.620s

>> time s4 map        s4://step1/  s4://step2/ 'cut -d, -f1-5'

0m38.109s

>> time s4 map        s4://step2/  s4://step3/ 'pypy3 /mnt/passenger_counts_inlined.py'

0m19.917s

>> time s4 map-to-n   s4://step3/  s4://step4/ 'cat > results && echo results'

0m1.641s

>> time s4 map-from-n s4://step4/  s4://step5/ 'xargs cat | python /mnt/merge_results.py'

0m0.430s

we can optimize by merging some of these steps.

>> s4 rm -r s4://step

>> time s4 map s4://inputs/ s4://step1/ 'cat > url
                                         aws s3 cp "$(cat url)" - \
                                          | cut -d, -f1-5 \
                                          | pypy3 /mnt/passenger_counts_inlined.py'

1m56.197s

performance improves, but we can no longer measure steps independently. sometimes we should combine steps, others we should pull them apart.

while we've got the cluster up, let's do one more thing. we haven't really flexed 1:n and n:1 maps properly yet, so let's do that. the taxi dataset is organized into files by date. let's reorganize it by passenger count. this will make it easier to answer questions about the trips for a given passenger count by without scanning the entire dataset.

we're going to need a new data script for our 1:n map. it will partition data by passenger count into separate files. these files will be shuffled around the cluster according to their name. then we'll merge files with the same name into a single file. we're going to further partition each passenger count randomly into multiple files to more evenly spread the data around the cluster. we'll make 12 files per passenger count, the same as cluster size.

# partition_by_passengers.py
import sys
import random

cluster_size = int(sys.argv[1])

sys.stdin.readline() # skip the header

files = {}

for line in sys.stdin:
    cols = line.split(',')
    try:
        passengers = int(cols[3])
    except (IndexError, ValueError):
        continue
    else:
        randint = random.randint(0, cluster_size)
        filename = f'passengers_{passengers}_{randint:03d}.csv'
        if filename not in files:
            files[filename] = open(filename, 'w')
        files[filename].write(line)

for name, file in files.items():
    print(name)
    file.close()

>> aws-ec2-scp partition_by_passengers.py :/mnt $name --yes

>> s4 rm -r s4://step

>> time s4 map        s4://inputs/ s4://step1/ 'cat > url && aws s3 cp "$(cat url)" -'

1m16.529s

>> time s4 map        s4://step1/  s4://step2/ 'cut -d, -f1-5'

0m38.528s

>> time s4 map-to-n   s4://step2/  s4://step3/ 'pypy3 /mnt/partition_by_passengers.py 12'

2m11.914s

>> time s4 map-from-n s4://step3/  s4://step4/ 'xargs cat'

0m25.288s

earlier we did a 1:n map, where n=1, sending all results to a single machine. here we did a 1:n map, where n>1, sending results all around the cluster.

earlier we followed that with a n:1 map which ran on a single machine, since only that machine had data. here we followed that with a n:1 map which ran on every machine, since every machine had data, merging the shuffled pieces of data back into single files.

since we partitioned the data in a way that spread it evenly around the cluster, we could see during processing that all machines were busy and then all went idle at the same time. if we hadn't partitioned this way we likely would have seen a few machines staying busy while the rest went idle.

let's take a peak at the data.

>> s4 ls s4://step4/ \
    | awk '{print $3, $4}' \
    | head -n3 \
    | column -t

29120189  passengers_0_000.csv
29084534  passengers_0_001.csv
29021334  passengers_0_002.csv

>> s4 eval s4://step4/passengers_0_000.csv "head -n1"

DDS,2009-01-06 06:46:08,2009-01-06 07:03:10,0,4.2999999999999998

>> s4 eval s4://step4/passengers_5_000.csv "head -n1"

VTS,2009-01-27 14:41:00,2009-01-27 14:48:00,5,1.1299999999999999

s4 eval s4://step4/passengers_5_000.csv "cut -d, -f2 | grep -Eo '^.{4}' | sort | uniq -c | sort -nr"

1145682 2009
1095902 2010
1065193 2011
 927308 2012
 771382 2013
 713021 2014
 609841 2015
 521383 2016
 414996 2017
 353959 2018
 261314 2019
  43358 2020
      1 2008

normally at this point we'd push the results back to s3 to make them durable, but our cluster has read only access, so we won't be doing that.

while we've got a cluster up, let's take a look at performance. what's the biggest and smallest file in the dataset?

>> aws s3 ls --recursive nyc-tlc/ | sort -nk3 | tail -n1

2016-08-15 08:50:21 2994922424 trip data/yellow_tripdata_2012-03.csv

>> aws s3 ls --recursive nyc-tlc/ | sort -nk3 | head -n3

2016-08-11 07:16:22          0 trip data/
2016-08-17 07:54:39          0 misc/
2016-08-17 07:57:08      12322 misc/taxi _zone_lookup.csv

>> s4 ls -r s4://step1/yellow_tripdata_2012-03.csv | awk '{print $3, $4}'

2994922424 yellow_tripdata_2012-03.csv

let's copy the smallest file to s4.

>> aws s3 cp "s3://nyc-tlc/misc/taxi _zone_lookup.csv" - | s4 cp - s4://small/data.csv

let's copy the biggest file from s3 and from s4. we'll run this test on the first machine in the cluster, since the big file doesn't live on that machine.

>> id=$(aws-ec2-id $name | head -n1)

>> aws-ec2-ssh $id --yes --cmd '
       time aws s3 cp "s3://nyc-tlc/trip data/yellow_tripdata_2012-03.csv" - >/dev/null
   '

0m15.018s

>> aws-ec2-ssh $id --yes --cmd '
       time s4 cp s4://step1/yellow_tripdata_2012-03.csv - >/dev/null
   '

0m3.251s

now let's copy the smallest file several times in a loop.

>> aws-ec2-ssh $id --yes --cmd '
       set -x
       time for i in {1..20}; do
           aws s3 cp "s3://nyc-tlc/misc/taxi _zone_lookup.csv" - >/dev/null
       done
   '

0m7.909s

>> aws-ec2-ssh $id --yes --cmd '
       set -x
       time for i in {1..20}; do
           s4 cp "s4://small/data.csv" - >/dev/null
       done
   '

0m2.193s

we're done for now, so let's delete the cluster.

>> aws-ec2-rm $name --yes

clearly s3 and s4 have different performance characteristics, and if we think about their goals, we can understand why.

s3 is durable, elastic, and authenticated. s4 is ephemeral, static, and unauthenticated.

s3 goes slower and almost certainly won't lose data. s4 goes faster and probably won't lose data.

s3 must not fail. s4 may fail and retry strategies must be considered.

these two systems are perfect compliments. we want durability, but we don't need it at every step. we want distributed compute, but we don't want to manually manage the details. we want data shuffle, but we don't want complicated infrastructure or poor performance.

using s4 we can focus more on our data pipelines, and less on low level details of distributed compute. our data pipelines can start, end, and checkpoint to durable data in s3. everywhere in between they can use s4 to map arbitrary commands over ephemeral immutable data in 1:1, 1:n and n:1 operations.

you can find more examples of s4 here, where further analysis of the nyc taxi dataset is done with python and bsv. to verify results and provide a performance baseline the analysis is repeated with presto on emr.

performant batch processing with bsv s4 and presto

full source code is available here.

we looked at scaling python batch processing vertically and horizontally. we refactored the details of distributed compute out of our code. we discovered a reasonable baseline for data processing performance on a single cpu core.

let's build on these experiences and revisit the nyc taxi dataset. we'll use presto as a performance and correctness baseline to evaluate identical analysis with bsv on a s4 cluster.

we'll be using some aws tooling and the official aws cli. one could also use other tools without much trouble.

we're going to only use the first 5 columns, since they are consistent across dataset. we'll create two tables so we can transform the data from csv into orc and get decent performance.

-- schema.hql
CREATE EXTERNAL TABLE IF NOT EXISTS `taxi_csv` (
  `vendor`     string,
  `pickup`     timestamp,
  `dropoff`    timestamp,
  `passengers` integer,
  `distance`   double
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
STORED AS TEXTFILE
LOCATION '/taxi_csv/'
tblproperties("skip.header.line.count"="1");

CREATE EXTERNAL TABLE IF NOT EXISTS `taxi` (
  `vendor`     string,
  `pickup`     timestamp,
  `dropoff`    timestamp,
  `passengers` integer,
  `distance`   double
)
STORED AS ORC
LOCATION '/taxi/';

let's spin up an emr cluster with hive and presto. we'll size it the same as in horizontal scaling.

if you haven't used emr before you may need to create some default iam roles, then we spin up the cluster.

>> export region=us-east-1

>> aws-iam-ensure-common-roles

>> id=$(aws-emr-new --count 12 \
                    --type i3en.2xlarge \
                    --applications hive,presto \
                    test-cluster)

>> time aws-emr-wait-for-state $id --state running

7m37.834s

then we fetch the dataset.

>> time aws-emr-ssh $id --cmd '
       s3-dist-cp --src="s3://nyc-tlc/trip data/" \
                  --srcPattern=".*yellow.*" \
                  --dest=/taxi_csv/
   '

2m52.909s

then we create the tables and translate csv to orc.

>> aws-emr-hive -i $id schema.hql

0m9.091s

-- csv_to_orq.pql
INSERT INTO taxi
SELECT *
FROM taxi_csv;

>> aws-emr-presto -i $id csv_to_orc.pql

2m48.524s

now that we have a cluster with data, we can do our analysis. let's ask a few of questions of different types.

grouping and counting.

-- count_rides_by_passengers.pql
SELECT passengers, count(*) as cnt
FROM taxi
GROUP BY passengers
ORDER BY cnt desc
LIMIT 9;

>> aws-emr-presto -i $id count_rides_by_passengers.pql

          1 | 1135227331
          2 |  239684017
          5 |  103036920
          3 |   70434390
          6 |   38585794
          4 |   34074806
          0 |    6881330
       NULL |     527580
          7 |       2040

0m5.775s

more grouping and counting.

-- count_rides_by_date.pql
SELECT YEAR(pickup), MONTH(pickup), count(*) as cnt
FROM taxi
GROUP BY YEAR(pickup), MONTH(pickup)
ORDER BY cnt desc
LIMIT 9;

>> aws-emr-presto -i $id count_rides_by_date.pql

  2012 |     3 | 16146923
  2011 |     3 | 16066350
  2013 |     3 | 15749228
  2011 |    10 | 15707756
  2009 |    10 | 15604551
  2012 |     5 | 15567525
  2011 |     5 | 15554868
  2010 |     9 | 15540209
  2010 |     5 | 15481351

0m10.556s

grouping and accumulating.

-- sum_distance_by_date.pql
SELECT YEAR(pickup), MONTH(pickup), cast(floor(sum(distance)) as bigint) as dst
FROM taxi
GROUP BY YEAR(pickup), MONTH(pickup)
ORDER BY dst desc
LIMIT 9;

>> aws-emr-presto -i $id sum_distance_by_date.pql

  2013 |     8 | 975457587
  2015 |     4 | 403568758
  2010 |     3 | 372299513
  2015 |    11 | 303443064
  2010 |     2 | 216050426
  2015 |     3 | 210197223
  2015 |     5 | 179394357
  2015 |     1 | 171590254
  2015 |     6 | 145792590

0m9.844s

finding large values.

-- top_n_by_distance.pql
SELECT cast(floor(distance) as bigint)
FROM taxi
ORDER BY distance desc
LIMIT 9;

>> aws-emr-presto -i $id top_n_by_distance.pql

 198623013
  59016609
  19072628
  16201631
  15700000
  15420061
  15420004
  15331800
  15328400

0m5.916s

distributed sort.

-- sort_by_distance.hql
CREATE EXTERNAL TABLE `sorted` (
  `distance` double
)
STORED AS ORC
LOCATION '/sorted/';

-- sort_by_distance.pql
INSERT INTO sorted
SELECT distance
FROM taxi
ORDER BY distance desc;

>> aws-emr-hive   -i $id sort_by_distance.hql

>> aws-emr-presto -i $id sort_by_distance.pql

9m44.334s

finally we shutdown the cluster.

>> aws-emr-rm $id

now let's redo the analysis with bsv and s4.

first we need to install s4 and spin up a cluster. we're going to use an ami instead of live bootstrapping to save time.

>> git clone https://github.com/nathants/s4

>> cd s4

>> python3 -m pip install -r requirements.txt .

>> export region=us-east-1

>> name=s4-cluster

>> time type=i3en.2xlarge ami=s4 num=12 bash scripts/new_cluster.sh $name

3m41.060s

>> export region=us-east-1

>> name=s4-cluster

>> bash scripts/connect_to_cluster.sh $name

let's check the cluster health.

>> s4 health

healthy:   10.0.3.111:8080
healthy:   10.0.2.192:8080
healthy:   10.0.14.51:8080
healthy:   10.0.9.243:8080
healthy:   10.0.15.97:8080
healthy:   10.0.14.223:8080
healthy:   10.0.15.25:8080
healthy:   10.0.5.197:8080
healthy:   10.0.15.201:8080
healthy:   10.0.7.71:8080
healthy:   10.0.5.249:8080
healthy:   10.0.14.19:8080

now we fetch the dataset and convert it to bsv.

# schema.sh
#!/bin/bash
set -euo pipefail

prefix='s3://nyc-tlc/trip data'

keys=$(aws s3 ls "$prefix/" \
        | grep yellow \
        | awk '{print $NF}' \
        | while read key; do
           echo "$prefix/$key";
          done)

i=0
echo "$keys" | while read key; do
    echo $key
    num=$(printf "%03d" $i)
    yearmonth=$(echo $key | tr -dc 0-9 | tail -c6)
    echo $key | s4 cp - s4://inputs/${num}_${yearmonth}
    i=$((i+1))
done

set -x
time s4 map-to-n s4://inputs/ s4://columns/ '
    cat > url
    aws s3 cp "$(cat url)" - \
     | tail -n+2 \
     | bsv \
     | bschema *,*,*,a:i64,a:f64,... --filter \
     | bunzip $filename
'

let's break down what's going on here.

first we find all the s3 keys of the dataset.

prefix='s3://nyc-tlc/trip data'

keys=$(aws s3 ls "$prefix/" \
        | grep yellow \
        | awk '{print $NF}' \
        | while read key; do
           echo "$prefix/$key";
          done)

then we put those keys into s4. since there aren't many keys, we're using numeric prefixes here to ensure the keys are spread evenly across the cluster.

i=0
echo "$keys" | while read key; do
    echo $key
    num=$(printf "%03d" $i)
    yearmonth=$(echo $key | tr -dc 0-9 | tail -c6)
    echo $key | s4 cp - s4://inputs/${num}_${yearmonth}
    i=$((i+1))
done

then we fetch the dataset and convert it to bsv.

time s4 map-to-n s4://inputs/ s4://columns/ '
    cat > url
    aws s3 cp "$(cat url)" - \
     | tail -n+2 \
     | bsv \
     | bschema *,*,*,a:i64,a:f64,... --filter \
     | bunzip $filename
'

let's break that one down a bit more.

we use map-to-n because our pipeline emits file names instead of data.
fetch the data.
skip the csv header.
bsv converts csv to bsv.
bschema filters for rows with at least 5 columns and discards any with less.
bschema keeps the first 5 columns of valid rows.
bschema converts column 4 and 5 from ascii to numerics.
bunzip splits a single stream of 5 columns into 5 streams of 1 column and emits their file names. the original file name is used as prefix.

let's run it.

>> bash schema.sh

1m11.860s

now that we have a cluster with data, we can do our analysis.

grouping and counting.

# count_rides_by_passengers.sh

s4 map-to-n s4://columns/*/*_4 s4://tmp/01/ \
            'bcounteach-hash \
             | bpartition 1'

s4 map-from-n s4://tmp/01/ s4://tmp/02/ \
              'xargs cat \
               | bsumeach-hash i64 \
               | bschema i64:a,i64:a \
               | csv'

s4 eval s4://tmp/02/0 \
        'tr , " " \
         | sort -nrk2 \
         | head -n9'

let's break that down a bit.

we s4 rm because we need blank scratch space.
we s4 map-to-n on a single column, use bcounteach-hash to count the values, then bpartition by 1 sending all results from around the cluster to a single machine.
we s4 map-from-n to merge the results. xargs cat turns file names into data, bsumeach-hash merges the counts, then bschema converts numerics back to ascii, and csv converts the result to csv.
we s4 eval to fetch the result with tr, sort, and head for formatting.

let's run it.

>> bash count_rides_by_passengers.sh

1 1135227331
2 239684017
5 103036920
3 70434390
6 38585794
4 34074806
0 7408814
7 2040
8 1609

0m2.616s

more grouping and counting.

# count_rides_by_date.sh

s4 map-to-n s4://columns/*/*_2 s4://tmp/01/ \
            'bschema 7* \
             | bcounteach-hash \
             | bpartition 1'

s4 map-from-n s4://tmp/01/ s4://tmp/02/ \
              'xargs cat \
               | bsumeach-hash i64 \
               | bschema *,i64:a \
               | csv'

s4 eval s4://tmp/02/0 \
        'tr , " " \
         | sort -nrk2 \
         | head -n9'

let's break that down a bit.

we s4 rm because we need blank scratch space.
we s4 map-to-n on a single column, use bschema to select the first 7 bytes, use bcounteach-hash to count the values, then bpartition by 1 sending all results from around the cluster to a single machine.
we s4 map-from-n to merge the results. xargs cat turns file names into data, bsumeach-hash merges the counts, then bschema converts numerics back to ascii, and csv converts the result to csv.
we s4 eval to fetch the result with tr, sort, and head for formatting.

let's run it.

>> bash count_rides_by_date.sh

2012-03 16146923
2011-03 16066350
2013-03 15749228
2011-10 15707756
2009-10 15604551
2012-05 15567525
2011-05 15554868
2010-09 15540209
2010-05 15481351

0m3.399s

grouping and accumulating.

# sum_distance_by_date.sh

s4 map-from-n s4://columns/ s4://tmp/01/ \
              'bzip 2,5 \
               | bschema 7*,8 \
               | bsumeach-hash f64'

s4 map-to-n s4://tmp/01/ s4://tmp/02/ \
            'bpartition 1'

s4 map-from-n s4://tmp/02/ s4://tmp/03/ \
              'xargs cat \
               | bsumeach-hash f64 \
               | bschema 7,f64:a \
               | csv'

s4 eval s4://tmp/03/0 \
        'tr , " " \
         | sort -nrk2 \
         | head -n9'

let's break that down a bit.

we s4 rm because we need blank scratch space.
we s4 map-from-n to bzip together columns 2 and 5, then use bschema to select the first 7 bytes of column 1, convert column 2 to numerics, then bsumeach-hash to sum column 2 by column 1.
we s4 map-to-n to bpartition by 1 sending all results from around the cluster to a single machine.
we s4 map-from-n to merge the results. xargs cat turns file names into data, bsumeach-hash merges the sums, then bschema converts numerics back to ascii, and csv converts the result to csv.
we s4 eval to fetch the result with tr, sort, and head for formatting.

let's run it.

>> bash sum_distance_by_date.sh

2013-08 975457587.2201815
2015-04 403568758.3299783
2010-03 372299513.2798572
2015-11 303443064.4099629
2010-02 216050426.449974
2015-03 210197223.1599888
2015-05 179394357.3799431
2015-01 171590254.990021
2015-06 145792590.1599617

0m7.130s

finding large values.

# top_n_by_distance.sh

s4 map s4://columns/*/*_5 s4://tmp/01/ \
       'btopn 9 f64'

s4 map-from-n s4://tmp/01/ s4://tmp/02/ \
              'bmerge -r f64'

s4 map-to-n s4://tmp/02/ s4://tmp/03/ \
            'bpartition 1'

s4 map-from-n s4://tmp/03/ s4://tmp/04/ \
              'bmerge -r f64 \
               | bhead 9 \
               | bschema f64:a \
               | csv'

s4 eval s4://tmp/04/0 \
        'cat'

let's break that down a bit.

we s4 rm because we need blank scratch space.
we s4 map to btopn over column 5, accumulating the top 9 values.
we s4 map-from-n to bmerge all results into a single result per machine.
we s4 map-to-n to bpartition by 1 sending all results from around the cluster to a single machine.
we s4 map-from-n to merge the results. bmerge combines the results, bhead takes the top 9, then bschema converts numerics back to ascii, and csv converts the result to csv.
we s4 eval to fetch the result with tr, sort, and head for formatting.

let's run it.

>> bash top_n_by_distance.sh

198623013.6
59016609.3
19072628.8
16201631.4
15700000
15420061
15420004.5
15331800
15328400

0m2.832s

distributed sort.

# sort_by_distance.sh

s4 map s4://columns/*/*_5 s4://tmp/01/ \
      'bsort -r f64'

s4 map-from-n s4://tmp/01/ s4://tmp/02/ \
              'bmerge -r f64'

s4 map-to-n s4://tmp/02/ s4://tmp/03/ \
            'bpartition -l 1'

s4 map-from-n s4://tmp/03/ s4://tmp/04/ \
              'bmerge -lr f64 \
               | blz4'

s4 eval s4://tmp/04/0
        'blz4d \
         | bschema f64:a \
         | csv
         | head -n9'

let's break that down a bit.

we s4 rm because we need blank scratch space.
we s4 map to bsort column 5.
we s4 map-from-n to bmerge all results into a single result per machine.
we s4 map-to-n to bpartition by 1 sending all results from around the cluster to a single machine.
we s4 map-from-n to merge the results. bmerge combines the results.
we s4 eval to fetch the the first few rows with tr, sort, and head for formatting.
we use lz4 compression at several steps to mitigate iowait.

let's run it.

>> bash sort_by_distance.sh

2m10.216s

we're done for now, so let's delete the cluster.

>> aws-ec2-rm $name --yes

let's put our results in a table.

query	presto seconds	s4 seconds
count rides by passengers	6	3
count rides by date	11	3
sum distance by date	10	7
top n by distance	6	3
distributed sort	584	130

so s4 and bsv exceeds our performance baseline. we could use it for batch processing. should we? it depends.

let's look again at one of the queries.

-- sort_by_distance.pql
INSERT INTO sorted
SELECT distance
FROM taxi
ORDER BY distance desc;

# sort_by_distance.sh
s4 map        s4://columns/*/*_5 s4://tmp/01/ 'bsort -r f64'
s4 map-from-n s4://tmp/01/       s4://tmp/02/ 'bmerge -r f64'
s4 map-to-n   s4://tmp/02/       s4://tmp/03/ 'bpartition -l 1'
s4 map-from-n s4://tmp/03/       s4://tmp/04/ 'bmerge -lr f64 | blz4'

the presto query is high level. it expresses what we want to do, not how to do it.

the s4 query is low level. it expresses how to do it, which if correct, results in what we want.

the presto query will be automatically transformed into executable steps by a query planner.

the s4 query is the executable steps, manually planned.

the presto query is difficult to extend in arbitrary ways.

the s4 query is easy to extend in arbitrary ways. any executable or shell snippet can be inserted into the pipeline of an existing step or as a new step.

the presto query has implicit intermediate results, which are not accessible.

the s4 query has explicit intermediate results, which are accessible.

the presto query has multiple implicit steps which are difficult to analyze and measure independently.

the s4 query has multiple explicit steps which are easy to analyze and measure independently. in fact, we omitted it from the results before, but the s4 query timed each step.

>> bash sort_by_distance.sh

+ s4 map 's4://columns/*/*_5' s4://tmp/01/ 'bsort -r f64'
ok ok ok ok ok ok ok ok ok ok ok ok
0m21.215s

+ s4 map-from-n s4://tmp/01/ s4://tmp/02/ 'bmerge -r f64'
ok ok ok ok ok ok ok ok ok ok ok ok
0m1.815s

+ s4 map-to-n s4://tmp/02/ s4://tmp/03/ 'bpartition -l 1'
ok ok ok ok ok ok ok ok ok ok ok ok
0m1.432s

+ s4 map-from-n s4://tmp/03/ s4://tmp/04/ 'bmerge -lr f64 | blz4'
ok
1m43.728s

2m10.216s

as we might expect, the final merge on a single machine is slow. surprisingly, the merge and shuffle steps were very fast. i wonder how much time shuffle took for presto?

presto is excellent, and significantly faster than the previous generation. it should be used, at a minimum, to check the correctness of your batch processing.

s4 and bsv are primitives for distributed data processing. they are low level, high performance, and flexible. they should be used, at a minimum, to establish a performance baseline.

optimizing a bsv data processing pipeline

full source code is available here.

in performant batch processing we composed simple tools into data pipelines. there are many benefits to this. simple tools are easier to write, test, and audit. they can even be shell snippets or existing unix utilities. they can be written in any language and rebuilt as needed. simple tools can compose into arbitrarily complex pipelines, and if something is out of reach you can always add another simple tool. simple tools can even be performant.

there is a cost to composing simple tools into data pipelines. primarily this cost is serialization and copies. efficient data formats and increased pipe sizes mitigate this, but don't eliminate it.

let's install bsv then measure the cost.

>> _gen_bsv 8 12000000 > /tmp/data.bsv

>> ls -lh /tmp/data.bsv | awk '{print $5}'
1.1G

>> time cat /tmp/data.bsv >/dev/null
0m0.124s

>> time cat /tmp/data.bsv | cat >/dev/null
0m0.302s

>> time cat /tmp/data.bsv | cat | cat >/dev/null
0m0.439s

>> time cat /tmp/data.bsv | cat | cat | cat >/dev/null
0m0.537s

>> time bcopy </tmp/data.bsv >/dev/null
0m0.890s

>> time bcopy </tmp/data.bsv | bcopy >/dev/null
0m1.137s

>> time bcopy </tmp/data.bsv | bcopy | bcopy >/dev/null
0m1.228s

>> time bcopy </tmp/data.bsv | bcopy | bcopy | bcopy >/dev/null
0m1.432s

so even when we just copy bytes with cat, we can see that as the pipeline grows, time goes up. the effect is even greater when parsing and serialization is performed at each step with bcopy.

when we are doing distributed compute there will be serialization. it's required before data can go over the network. for convenience, we use it between every process in the pipelines we compose to simplify their interface. the benefit is convenience, the cost is performance. this convenience helps us to quickly prototype pipelines and integrate new tools. once our pipelines have stabilized, we can optimize it out.

first we need to install s4 and spin up a cluster. we're going to use an ami instead of live bootstrapping to save time.

>> git clone https://github.com/nathants/s4

>> cd s4

>> python3 -m pip install -r requirements.txt .

>> export region=us-east-1

>> name=s4-cluster

>> type=i3en.2xlarge ami=s4 num=12 bash scripts/new_cluster.sh $name

>> export region=us-east-1

>> name=s4-cluster

>> bash scripts/connect_to_cluster.sh $name

let's take a look at one of the queries from performant batch processing.

# sum_distance_by_date.sh
s4 map-from-n s4://columns/ s4://tmp/01/   'bzip 2,5 | bschema 7*,8 | bsumeach-hash f64'
s4 map-to-n   s4://tmp/01/  s4://tmp/02/   'bpartition 1'
s4 map-from-n s4://tmp/02/  s4://tmp/03/   'xargs cat | bsumeach-hash f64 | bschema 7,f64:a | csv'
s4 eval       s4://tmp/03/0                'tr , " " | sort -nrk2 | head -n9'

let's run it and see how long each step takes.

>> bash schema.sh

1m9.272s

>> bash sum_distance_by_date.sh

+ s4 map-from-n s4://columns/ s4://tmp/01/ 'bzip 2,5 | bschema 7*,8 | bsumeach-hash f64'
0m5.718s

+ s4 map-to-n s4://tmp/01/ s4://tmp/02/ 'bpartition 1'
0m1.427s

+ s4 map-from-n s4://tmp/02/ s4://tmp/03/ 'xargs cat | bsumeach-hash f64 | bschema 7,f64:a | csv'
0m0.349s

+ s4 eval s4://tmp/03/0 'tr , " " | sort -nrk2 | head -n9'
0m0.161s

0m7.655s

the majority of runtime is in the first step. let's try to replace that pipeline with a single executable. we'll base it off bzip.c, and then insert functionality from bschema.c and bsumeach_hash.c. let's look at the diff of our new step1.c against the original bzip.c.

diff --git a/~/repos/bsv/src/bzip.c b/step1.c
index d393f10..4e12b7a 100644
--- a/~/repos/bsv/src/bzip.c
+++ b/step1.c
@@ -3,6 +3,7 @@
 #include "load.h"
 #include "array.h"
 #include "dump.h"
+#include "hashmap.h"

 #define DESCRIPTION "combine single column inputs into a multi column output\n\n"
 #define USAGE "ls column_* | bzip [COL1,...COLN] [-l|--lz4]\n\n"
@@ -86,6 +87,13 @@ int main(int argc, char **argv) {
     // setup output
     writebuf_t wbuf = wbuf_init((FILE*[]){stdout}, 1, false);

+    // bsumeach-hash state
+    u8 *key;
+    void* element;
+    struct hashmap_s hashmap;
+    f64 *sum_f64;
+    ASSERT(0 == hashmap_create(2, &hashmap), "fatal: hashmap init\n");
+
     // process input row by row
     while (1) {
         for (i32 i = 0; i < ARRAY_SIZE(selected); i++) {
@@ -99,8 +107,36 @@ int main(int argc, char **argv) {
             ASSERT(memcmp(stops, do_stop, ARRAY_SIZE(selected) * sizeof(i32)) == 0, "fatal: all columns didn't end at the same length\n");
             break;
         }
-        dump(&wbuf, &new, 0);
+
+        // bschema 7*,*
+        new.sizes[0] = 7;
+
+        // bsumeach-hash f64
+        ASSERT(new.max >= 1, "fatal: need at least 2 columns\n");
+        ASSERT(8 == new.sizes[1], "fatal: bad data size\n");
+        if (element = hashmap_get(&hashmap, new.columns[0], new.sizes[0])) {
+            *(f64*)element += *(f64*)new.columns[1];
+        } else {
+            MALLOC(key, new.sizes[0]);
+            strncpy(key, new.columns[0], new.sizes[0]);
+            MALLOC(sum_f64, sizeof(f64)); *sum_f64 = *(f64*)new.columns[1];
+            ASSERT(0 == hashmap_put(&hashmap, key, new.sizes[0], sum_f64), "fatal: hashmap put\n");
+        }
+
     }
+
+    // bsumeach-hash f64 dump results
+    for (i32 i = 0; i < hashmap.table_size; i++) {
+        if (hashmap.data[i].in_use) {
+            row.max = 1;
+            row.columns[0] = hashmap.data[i].key;
+            row.sizes[0] = hashmap.data[i].key_len;
+            row.columns[1] = hashmap.data[i].data;
+            row.sizes[1] = sizeof(f64);
+            dump(&wbuf, &row, 0);
+        }
+    }
+
     dump_flush(&wbuf, 0);

 }

let's ship it to the cluster and compile it.

>> aws-ec2-scp step1.c : s4-cluster --yes

>> aws-ec2-ssh s4-cluster -yc "sudo gcc -Ibsv/util -Ibsv/vendor -flto -O3 -march=native -mtune=native -lm -o /usr/local/bin/step1 step1.c bsv/vendor/lz4.c"

our optimized query looks like this.

# sum_distance_by_date.sh
s4 map-from-n s4://columns/ s4://tmp/01/   'step1 2,5'
s4 map-to-n   s4://tmp/01/  s4://tmp/02/   'bpartition 1'
s4 map-from-n s4://tmp/02/  s4://tmp/03/   'xargs cat | bsumeach-hash f64 | bschema 7,f64:a | csv'
s4 eval       s4://tmp/03/0                'tr , " " | sort -nrk2 | head -n9'

let's run it.

>> bash sum_distance_by_date_optimized.sh

+ s4 map-from-n s4://columns/ s4://tmp/01/ 'step1 2,5'
0m2.034s

+ s4 map-to-n s4://tmp/01/ s4://tmp/02/ 'bpartition 1'
0m1.334s

+ s4 map-from-n s4://tmp/02/ s4://tmp/03/ 'xargs cat | bsumeach-hash f64 | bschema 7,f64:a | csv'
0m0.336s

+ s4 eval s4://tmp/03/0 'tr , " " | sort -nrk2 | head -n9'
real    0m0.161s

0m3.866s

looks like step1 went from 6 to 2 seconds and the whole query went from 8 to 4 seconds.

we're done for now, so let's delete the cluster.

>> aws-ec2-rm $name --yes

composing data pipelines from simple tools is an effective way to rapidly prototype.

reusing the same serialization between local and distributed processes we can build and use tools that don't care whether data is coming from or going to a file, a pipe, or a socket.

once our prototypes have stabilized, we can optimize them by collapsing pipelines into a single executable.

data processing performance with python go rust and c

full source code is available here.

performance is important, and yet our intuition about it is often wrong. previously we deployed optimized python across a cluster of machines to analyze the nyc taxi dataset. how was its performance?

let's try to discover a reasonable baseline for data processing performance and build intuition that can guide our decisions. we'll do this by experimenting with simple transformations of generated data using various formats, techniques, and languages on a single cpu core.

whether we are configuring and using off the shelf software or building bespoke systems, we need the ability to intuit problems and detect low hanging fruit.

we'll say that our data is a sequence of rows, that a row is made of 8 columns, and that a column is a random dictionary word.

we'll generate our dataset as csv with the following script.

# gen_csv.py
import sys
import random

words = [
    ...
]

num_rows = int(sys.argv[1])

for _ in range(num_rows):
    row = [random.choice(words) for _ in range(8)]
    print(','.join(row))

our first transformation will be selecting a subset of columns.

let's try python.

# select.py
import sys

for line in sys.stdin:
    columns = line.split(',')
    print(f'{columns[2]},{columns[6]}')

first we need some data.

>> pypy3 gen_csv.py 1000000 > /tmp/data.csv

>> ls -lh /tmp/data.csv | awk '{print $5}'

72M

we're gonna need more data.

>> pypy3 gen_csv.py 15000000 > /tmp/data.csv

>> ls -lh /tmp/data.csv | awk '{print $5}'

1.1G

that'll do. now let's try our selection. we'll make sure a subset of the result is sane, then we'll check the hash of the entire result using xxhsum. all other runs we'll discard the output and time execution.

>> python select.py </tmp/data.csv | head -n3

epigram,Madeleine
strategies,briefed
Doritos,putsch

>> python select.py </tmp/data.csv | xxhsum

12927f314ca6e9eb

seems sane. let's time it.

>> time python select.py </tmp/data.csv >/dev/null

real    0m10.076s
user    0m9.779s
sys     0m0.200s

let's try coreutils cut.

>> cut -d, -f3,7 </tmp/data.csv | xxhsum
12927f314ca6e9eb

>> time cut -d, -f3,7 </tmp/data.csv >/dev/null

real    0m3.534s
user    0m3.341s
sys     0m0.159s

faster. we may need to look at compiled languages for a reasonable baseline.

let's optimize by avoiding allocations and doing as little work as possible. we'll pull rows off a buffered reader, setup columns as offsets into that buffer, and access columns by slicing the row data.

let's try go.

>> go build -o select_go select.go

>> ./select_go </tmp/data.csv | xxhsum

12927f314ca6e9eb

>> time ./select_go </tmp/data.csv >/dev/null

real    0m2.832s
user    0m2.559s
sys     0m0.312s

faster than cut. this is progress.

let's try rust.

>> rustc -O -o select_rust select.rs

>> ./select_rust </tmp/data.csv | xxhsum

12927f314ca6e9eb

>> time ./select_rust </tmp/data.csv >/dev/null

real    0m2.602s
user    0m2.491s
sys     0m0.110s

pretty much the same. let's try c. we'll grab a few header only dependencies for csv parsing and buffered writing.

>> gcc -Iutils -O3 -flto -march=native -mtune=native -o select_c select.c

>> ./select_c </tmp/data.csv | xxhsum

12927f314ca6e9eb

>> time ./select_c </tmp/data.csv >/dev/null

real    0m2.716s
user    0m2.569s
sys     0m0.120s

so rust, go, and c are very similar. we may have established a baseline when working with csv.

let's try a similar optimization with pypy.

>> pypy select_inlined.py </tmp/data.csv | xxhsum

12927f314ca6e9eb

>> time pypy select_inlined.py </tmp/data.csv >/dev/null

real    0m4.491s
user    0m4.293s
sys     0m0.170s

not bad.

let's try using protobuf and go. we'll call the data format psv.

>> (cd psv && protoc -I=row --go_out=row row/row.proto)

>> (cd psv && go build -o psv psv.go)

>> (cd psv && go build -o select select.go)

>> ./psv/psv </tmp/data.csv >/tmp/data.psv

>> ./psv/select </tmp/data.psv | xxhsum

12927f314ca6e9eb  stdin

>> time ./psv/select </tmp/data.psv >/dev/null

real    0m10.424s
user    0m10.465s
sys     0m0.251s

interesting. slower than naive python and csv.

is reading and writing data to some format a majority of the work?

let's think about our optimized code from before. our representation of a row is 3 pieces of data. a byte array of content, an array of column start positions, and an array of column sizes. writing a row as csv was easy, but reading was hard.

what if we made it easier? all we want is an array of bytes and two int arrays.

let's let a row written as bytes be:

the max zero based index
the column sizes
the column data

| u16:max | u16:size | ... | u8[]:column | ... |

this should be easy to write, and more importantly easy to read. we can read max, which tells us how many sizes to read. from the sizes we can reconstruct the offsets and the size of the row data. we can then read the row data, and access the columns by offset and size.

our optimized code also buffered reads and writes into large chunks.

let's let a chunk of rows written as bytes be:

size
data

| i32:size | u8[]:row | ... |

let's constrain a chunk to only contain complete rows and be smaller than some maximum size.

we'll call this format bsv. we'll implement buffered reading and writing of chunks, as well as loading and dumping of rows.

let's implement our transformation using bsv in c.

>> gcc -Iutils -O3 -flto -march=native -mtune=native -o bsv/bsv bsv/bsv.c

>> ./bsv/bsv </tmp/data.csv >/tmp/data.bsv

>> gcc -Iutils -O3 -flto -march=native -mtune=native -o bsv/select bsv/select.c

>> ./bsv/select </tmp/data.bsv | xxhsum

12927f314ca6e9eb

>> time ./bsv/select </tmp/data.bsv >/dev/null

real    0m0.479s
user    0m0.339s
sys     0m0.140s

we've processed the same data, and system time has been fairly consistent, but user time has varied significantly.

let's try a second transformation where we reverse the columns of every row.

we'll implement it with csv in python, pypy and c, then with bsv in c.

>> python reverse.py </tmp/data.csv | xxhsum

e221974c95d356f9

>> time python reverse.py </tmp/data.csv >/dev/null

real    0m13.915s
user    0m13.743s
sys     0m0.170s

>> pypy3 reverse_inlined.py </tmp/data.csv | xxhsum

e221974c95d356f9

>> time pypy3 reverse_inlined.py </tmp/data.csv >/dev/null

real    0m6.141s
user    0m5.880s
sys     0m0.220s

>> gcc -Iutils -O3 -flto -march=native -mtune=native -o reverse reverse.c

>> ./reverse </tmp/data.csv | xxhsum

e221974c95d356f9

>> time ./reverse </tmp/data.csv >/dev/null

real    0m2.890s
user    0m2.719s
sys     0m0.170s

>> gcc -Iutils -O3 -flto -march=native -mtune=native -o bsv/reverse bsv/reverse.c

>> ./bsv/reverse </tmp/data.bsv | xxhsum

e221974c95d356f9

>> time ./bsv/reverse </tmp/data.bsv >/dev/null

real    0m1.052s
user    0m0.891s
sys     0m0.161s

let's try a third transformation where we count every column where the first character of the first column is "f".

we'll implement it with csv in python, pypy and c, then with bsv in c.

>> time python count.py </tmp/data.csv

467002

real    0m6.385s
user    0m6.223s
sys     0m0.160s

>> time pypy3 count_inlined.py </tmp/data.csv

467002

real    0m3.147s
user    0m2.938s
sys     0m0.180s

>> gcc -Iutils -O3 -flto -march=native -mtune=native -o count count.c

>> time ./count </tmp/data.csv

467002

real    0m2.367s
user    0m2.245s
sys     0m0.121s

>> gcc -Iutils -O3 -flto -march=native -mtune=native -o bsv/count bsv/count.c

>> time bsv/count </tmp/data.bsv

467002

real    0m0.260s
user    0m0.135s
sys     0m0.125s

in transformations 2 and 3 we again see significant variance in user time.

let's put our user time results in a table.

first we have our select transformation, which outputs 25% of its input.

format	language	user seconds	gigabytes / second
psv	go	10.4	0.1
csv	python	9.7	0.1
csv	pypy	4.3	0.2
csv	go	2.6	0.4
csv	c	2.6	0.4
csv	rust	2.5	0.4
bsv	c	0.3	3.3

second we have our reverse transformation, which outputs 100% of its input.

format	language	user seconds	gigabytes / second
csv	python	13.7	0.1
csv	pypy	5.8	0.2
csv	c	2.7	0.4
bsv	c	0.9	1.1

third we have our count transformation, which outputs <0.001% of its input.

format	language	user seconds	gigabytes / second
csv	python	6.2	0.2
csv	pypy	2.9	0.3
csv	c	2.2	0.5
bsv	c	0.1	10

interesting. let's take a closer look at the csv and bsv results for c based on the ratio of inputs to outputs.

inputs / outputs	format	language	user seconds	gigabytes / second
1 / 1	csv	c	2.7	0.4
4 / 1	csv	c	2.6	0.4
1000 / 1	csv	c	2.2	0.5

inputs / outputs	format	language	user seconds	gigabytes / second
1 / 1	bsv	c	0.9	1.1
4 / 1	bsv	c	0.3	3.3
1000 / 1	bsv	c	0.1	10

now this is interesting. when dealing with csv, the ratio of inputs to outputs has almost no impact on performance. when dealing with bsv the impact is x3 at each step. this suggests that for csv, parsing the input dominates, while for bsv, writing the output dominates. this asks an interesting question, how can we optimize output? for simplicity, the bsv code is outputting csv. it may be worth experimenting with other output formats, but we'll skip that for now.

do we have enough information to establish a baseline? perhaps.

we've seen python process csv and go process protobuf at 100 megabytes / second.

we've seen c, go, and rust process csv at 400 megabytes / second.

we've seen c process bsv at 1-10 gigabytes / second.

why don't we start with the following baseline. we'll think of it as napkin math.

category	rate
slow	<=100 megabytes / second / cpu core
decent	~500 megabytes / second / cpu core
fast	>=1000 megabytes / second / cpu core

as we do data processing, either by configuring and using off the shelf software, or by building bespoke systems, we can keep these rates in mind.

if you are interested in bsv, you can find it here.

for further experimentation with go, rust, and c, look here.

for examples of applying bsv to distributed compute, look here.

s4

why

s3 is awesome, but can be expensive, slow, and doesn't expose data local compute or efficient shuffle.

what

an s3 cli compatible storage cluster that is cheap and fast, with data local compute and efficient shuffle.

data local compute maps arbitrary commands over immutable keys in 1:1, n:1 and 1:n operations.

data shuffle is implicit in 1:n mappings.

server placement is based on the hash of basename or a numeric prefix.

key	method	placement
s4://bucket/dir/name.txt	int(hash("name.txt"))	?
s4://bucket/dir/000_bucket0.txt	int("000")	0
s4://bucket/dir/000	int("000")	0

keys are strongly consistent and cannot be updated unless first deleted.

when

use this for efficiently processing ephemeral data.

keep durable inputs, outputs, and checkpoints in s3.

how

a ring of servers store files on disk.

a metadata controller on each server orchestrates out of process operations for data transfer and local compute.

a cli client coordinates cluster activity.

non goals

high availability. every key lives on one and only one server.

high durability. data lives on a single disk, and is as durable as that disk.

security. data transfers are checked for integrity, but not encrypted. service access is unauthenticated. secure the network with wireguard if needed.

fine granularity. data should be medium to coarse granularity.

safety for all inputs. service access should be considered to be at the level of root ssh. any user input should be escaped for shell.

cluster resizing. clusters should be short lived and data ephemeral. instead of resizing create a new cluster.

pagination of list results. data layout and partitioning must be considered.

install

go install:

go install github.com/nathants/s4/cmd/s4@latest
go install github.com/nathants/s4/cmd/s4_server@latest
sudo mv -f $(go env GOPATH)/bin/s4 /usr/local/bin/s4
sudo mv -f $(go env GOPATH)/bin/s4_server /usr/local/bin/s4-server

git clone:

git clone https://github.com/nathants/s4
cd s4
git clone go
make -j
sudo mv -fv bin/s4 bin/s4-server /usr/local/bin/

test

>> tox

automatic deployment

cd s4
name=s4-cluster
bash scripts/new_cluster.sh $name

manual deployment

deploy

ssh $server1 "curl -s https://raw.githubusercontent.com/nathants/s4/go/scripts/install_archlinux.sh | bash"
ssh $server2 "curl -s https://raw.githubusercontent.com/nathants/s4/go/scripts/install_archlinux.sh | bash"

configure

echo $server1:8080 >  ~/.s4.conf
echo $server2:8080 >> ~/.s4.conf
scp ~/.s4.conf $server1:
scp ~/.s4.conf $server2:

start

ssh $server1 s4-server
ssh $server2 s4-server

usage

echo hello world | s4 cp - s4://bucket/data.txt
s4 cp s4://bucket/data.txt -
s4 ls s4://bucket --recursive
s4 --help

examples

structured analysis of nyc taxi data with bsv and hive

adhoc exploration of nyc taxi data with python

bsv - a simple and efficient data format for easily manipulating chunks of rows of columns while minimizing allocations and copies.

optimizing a bsv data processing pipeline

performant batch processing with bsv, s4, and presto

discovering a baseline for data processing performance

refactoring common distributed data patterns into s4

scaling python data processing horizontally

scaling python data processing vertically

api

name	description
s4 rm	delete data from s4
s4 eval	eval a bash cmd with key data as stdin
s4 ls	list keys
s4 cp	copy data to or from s4
s4 map	process data
s4 map-to-n	shuffle data
s4 map-from-n	merge shuffled data
s4 config	list the server addresses
s4 health	health check every server

usage

s4 rm

usage: s4 rm [-h] [-r] prefix

    delete data from s4.

    - recursive to delete directories.


positional arguments:
  prefix           -

optional arguments:
  -h       show this help message and exit
  -r       False

s4 eval

usage: s4 eval [-h] key cmd

    eval a bash cmd with key data as stdin


positional arguments:
  key         -
  cmd         -

optional arguments:
  -h  show this help message and exit

s4 ls

usage: s4 ls [-h] [-r] [prefix]

    list keys


positional arguments:
  prefix           -

optional arguments:
  -h, --help       show this help message and exit
  -r, --recursive  False

s4 cp

usage: s4 cp [-h] [-r] src dst

    copy data to or from s4.

    - paths can be:
      - remote:       "s4://bucket/key.txt"
      - local:        "./dir/key.txt"
      - stdin/stdout: "-"
    - use recursive to copy directories.
    - keys cannot be updated, but can be deleted and recreated.
    - note: to copy from s4, the local machine must be reachable by the cluster, otherwise use `s4 eval`.


positional arguments:
  src              -
  dst              -

optional arguments:
  -h       show this help message and exit
  -r       False

s4 map

usage: s4 map [-h] indir outdir cmd

    process data.

    - map a bash cmd 1:1 over every key in indir putting result in outdir.
    - cmd receives data via stdin and returns data via stdout.
    - every key in indir will create a key with the same name in outdir.
    - indir will be listed recursively to find keys to map.


positional arguments:
  indir       -
  outdir      -
  cmd         -

optional arguments:
  -h  show this help message and exit

s4 map-to-n

usage: s4 map-to-n [-h] indir outdir cmd

    shuffle data.

    - map a bash cmd 1:n over every key in indir putting results in outdir.
    - cmd receives data via stdin, writes files to disk, and returns file paths via stdout.
    - every key in indir will create a directory with the same name in outdir.
    - outdir directories contain zero or more files output by cmd.
    - cmd runs in a tempdir which is deleted on completion.


positional arguments:
  indir       -
  outdir      -
  cmd         -

optional arguments:
  -h  show this help message and exit

s4 map-from-n

usage: s4 map-from-n [-h] indir outdir cmd

    merge shuffled data.

    - map a bash cmd n:1 over every key in indir putting result in outdir.
    - indir will be listed recursively to find keys to map.
    - cmd receives file paths via stdin and returns data via stdout.
    - each cmd receives all keys with the same name or numeric prefix
    - output name is that name


positional arguments:
  indir       -
  outdir      -
  cmd         -

optional arguments:
  -h  show this help message and exit

s4 config

usage: s4 config [-h]

    list the server addresses


optional arguments:
  -h  show this help message and exit

s4 health

usage: s4 health [-h]

    health check every server


optional arguments:
  -h  show this help message and exit

runclj

what

tooling around single file clojurescript programs running on node or the browser.

runclj ./rotate_the_logs.cljs

how

transparently converts a single clojurescript file into a temporary shadow-cljs project in .shadow-cljs/.

installation

get the dependencies:

bash
python3
java
node
npm

then:

git clone https://github.com/nathants/runclj
sudo mv runclj/bin/* runclj/bin/.shadow-cljs /usr/local/bin

examples

hello.cljs - node hello world
shell.cljs - node with subprocess, user prompts, and core.async
server.cljs - node express server
client.cljs - client app with material ui (demo)
macros.cljs - node hello world with macros

usage

runclj program.cljs

declare clojure and npm dependencies with meta-data at the start of the file.

#!/usr/bin/env runclj
^{:runclj {:npm [[express "4.16.3"]]
           :deps [[prismatic/schema "1.1.3"]]}}
(ns main
  (:require [schema.core :as schema :include-macros true]))

run the program.

runclj program.cljs

browser dev workflow

runclj-watch examples/client.cljs
open browser to localhost:8000
edit client.cljs and browser auto reloads
optionally connect to repl on localhost:3333

server dev workflow

runclj examples/server.cljs

deployment

runclj-tar program.cljs | ssh server tar xf -

then either install runclj and run normally runclj program.cljs

or run manually:

browser: cd .shadow-cljs/program.cljs/public && python3 -m http.server
server: cd .shadow-cljs/program.cljs/ && node public/main.js

outgrowing runclj

if your project grows too large for a single file, copy the generated shadow-cljs project and continue working on that directly.

runclj program.cljs
cp -r .shadow-cljs/program.cljs program

alternatively lift and shift to a project template like aws-gocljs.

render

why

minimal web content should be easy.

what

a tool for rendering a single github markdown file to a single html file.

install

python3 -m pip install git+https://github.com/nathants/render

usage

render readme.md \
    "render" \
    "render github markdown to html" \
    "github.com/nathants/render" \
    "https://github.com/nathants/render" \
    > index.html

python3 -m http.server &

firefox localhost:8000

>> render -h
usage: render [-h] [-f FOOTER] [-c CSS] github-markdown title subtitle header-link-name header-link-href

    render github markdown to html


positional arguments:
  github-markdown       -
  title                 -
  subtitle              -
  header-link-name      -
  header-link-href      -

optional arguments:
  -h, --help            show this help message and exit
  -f FOOTER, --footer FOOTER
                        ''
  -c CSS, --css CSS     ''

examples

all of nathants.com is created with render.

py-webengine

why

browser testing is annoying, brittle, and slow.

what

easy and fun browser testing from python with pyqt6-webengine.

how

execute javascript.

inspect network requests.

send native mouse and keyboard input.

wait for values to show up on screen.

make assertions.

install

mac

brew install python3
brew install qt6
python3 -m pip install git+https://github.com/nathants/py-webengine

linux

apt-get install -y qt6-webengine-dev                                # debian/ubuntu
pacman -S qt6-webengine                                             # arch
python3 -m pip install git+https://github.com/nathants/py-webengine # all platforms

docker

docker build -t py-webengine .

api

class webengine.Thread:

    action_delay_seconds = .01 # seconds between browser actions

    timeout_seconds = 10 # maximum seconds to wait_attr()

    def js(self, code):
        "execute javascript and return the result as a string"

    def click(self, selector):
        "send native mouse input at the center of the first element matching selector"

    def type(self, value):
        "send native keyboard input, one character at a time"

    def enter(self):
        "send native keyboard input enter"

    def attr(self, selector, attr):
        "return the list of the attribute for all elements matching selector"

    def wait_attr(self, selector, attr, value):
        "wait for the attribute of all elements matching a selector have the given value"

    def load(self, url):
        "load url"

    def screenshot(self, path):
        "save a png or jpg at path"

    def run(self):
        "run the main method"

    def main(self):
        "implement this method as your test"

usage

subclass webengine.Thread and implement main():

host = 'http://localhost:8000'

class Main(webengine.Thread):

    action_delay_seconds = .025

    def main(self):

        # wait for http server to come up and the site to load properly
        for _ in range(100):
            try:
                self.load(host)
                self.wait_attr('a', 'innerText', ['home', 'files', 'api', 'websocket'])
            except:
                print('wait for site to be ready')
                time.sleep(.1)
            else:
                break
        else:
            assert False

        # load the site
        self.load(host)

        # click on files and check contents
        self.click('a#files')
        self.wait_attr("#content p", 'innerText', ["files"])

then invoke your test:

def test():
    # build your webapp
    subprocess.check_call('gunzip --force --keep index.html.gz', shell=True)
    # run your webapp
    server = subprocess.Popen('python3 -m http.server', shell=True)
    try:
        # run webengine
        webengine.run_thread(Main, devtools='horizontal')
    finally:
        # stop webapp
        server.terminate()

if __name__ == '__main__':
    os.chdir(os.path.dirname(os.path.abspath(__file__)))
    sys.exit(pytest.main(['test.py', '-svvx', '--tb', 'native']))

to leave the browser open, insert somewhere in your test:

time.sleep(1000)

to drop into a python repl, first install ipdb:

python3 -m pip install ipdb

then insert somewhere in your test:

import ipdb; ipdb.set_trace()

run x11 docker:

docker run \
    -h $HOSTNAME \
    -e XAUTHORITY=/code/.Xauthority \
    -v $HOME/.Xauthority:/code/.Xauthority \
    -v /tmp/.X11-unix:/tmp/.X11-unix \
    -e DISPLAY \
    --ipc host \
    -v $(pwd)/example:/example \
    py-webengine \
    sh -c 'python3 /example/test.py'

run headless docker:

docker run \
    -it \
    -v $(pwd)/example:/example \
    py-webengine \
    sh -c 'xvfb-run python3 /example/test.py'

docker example:

>> docker run -it --rm py-webengine

wait for: a innerText ['home', 'files', 'api', 'websocket']
wait for: a href ['http://localhost:8000/#/home', 'http://localhost:8000/#/files', 'http://localhost:8000/#/api', 'http://localhost:8000/#/websocket']
wait for: #content innerText ['home']
wait for: #content p innerText ['files']
wait for: #content p innerText 'predicate(x)'
wait for: #content p innerText ['a', 'b', 'c', 'Enter']
wait for: #content innerText ['home']
PASSED

see the example for detailed usage.

in the example we will test the frontend from aws-gocljs.

a live demo of that site is here.

demo

py-web

why

almost always, http servers should be simple and easy.

what

an http library wrapping tornado.

install

git clone https://github.com/nathants/py-web
cd py-web
pip install -r requirements.txt .

http example

#!/usr/bin/env python3
import logging
import tornado.ioloop
import web

logging.basicConfig(level='INFO')

async def handler(request: web.Request) -> web.Response:
    size = int(request['query'].get('size', 0))
    token = request['kwargs']['token']
    return {'code': 200, 'body': f'{token} size: {size}'}

async def fallback_handler(request: web.Request) -> web.Response:
    route = request['args'][0]
    return {'code': 200, 'body': f'no such route: /{route}, try: /hello/xyz?size=123'}

routes = [('/hello/:token', {'get': handler}),
          ('/(.*)',         {'get': fallback_handler})]

app = web.app(routes)
server = tornado.httpserver.HTTPServer(app)
server.bind(8080)
server.start(0)
tornado.ioloop.IOLoop.current().start()

$ curl localhost:8080/hello/world?size=3
world size: 3

https example

#!/usr/bin/env python3
import logging
import tornado.ioloop
import web
import ssl
import subprocess

check_call = lambda *a: subprocess.check_call(' '.join(map(str, a)), shell=True, executable='/bin/bash', stderr=subprocess.STDOUT)

logging.basicConfig(level='INFO')

async def handler(request: web.Request) -> web.Response:
    size = int(request['query'].get('size', 0))
    token = request['kwargs']['token']
    return {'code': 200, 'body': f'{token} size: {size}'}

async def fallback_handler(request: web.Request) -> web.Response:
    route = request['args'][0]
    return {'code': 200, 'body': f'no such route: /{route}, try: /hello/XYZ'}

check_call('openssl req -x509 -nodes -newkey rsa:4096 -keyout ssl.key -out ssl.crt -days 9999 -subj "/CN=localhost/O=Fake Name/C=US"')
options = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
options.load_cert_chain('ssl.crt', 'ssl.key')

routes = [('/hello/:token', {'get': handler}),
          ('/(.*)',         {'get': fallback_handler})]

app = web.app(routes)
server = tornado.httpserver.HTTPServer(app, ssl_options=options)
server.bind(8080)
server.start(0)
tornado.ioloop.IOLoop.current().start()

$ curl --cacert ssl.crt https://localhost:8080/hello/world?size=3
world size: 3

ptop

what

a minimal htop alternative

demo

install

python3 -m pip install git+https://github.com/nathants/ptop

usage

ptop

help

hit h for help

notify

why

sometimes popup messages are for great good.

what

a fullscreen terminal app, typically launched in a new single use terminal, to display a popup message.

an optional y/n prompt will change the exit code.

an optional delay avoids accidental input.

demo

install

python3 -m pip install git+https://github.com/nathants/notify

usage

>> notify -h

usage: notify [-h] [-d DELAY] [-p] msg

    notify the user of a message with a fullscreen popup. hit any key to exit.


positional arguments:
  msg                   message to display

options:
  -h, --help            show this help message and exit
  -d DELAY, --delay DELAY
                        delay seconds before accepting user input for prompt (default: 0.5)
  -p, --prompt          prompt the user y/n then exit 0/1 (default: False)

go-hasdefer

what

a go linter to check that all goroutines have a defer statement.

why

it's too easy to miss panics when they happen inside goroutines, since they exit the defer scope of the caller.

install

go install github.com/nathants/go-hasdefer@latest

usage

>> go-hasdefer $(find test/good/ -name '*.go')

>> go-hasdefer $(find test/bad/ -name '*.go')
missing defer anon func oneliner:        test/bad/bad_onliner.go:4      go func() {}()
missing defer anon func multiliner:      test/bad/bad_multiliner.go:4   go func() {
missing defer top level func multiliner: test/bad/bad_import.go:12 func Foobar() {
missing defer top level func oneliner:   test/bad/bad_import.go:6 func (d *Data) Foobar2() {}
missing defer named func multiliner:     test/bad/bad_imported.go:8     Foobar3 := func() {
missing defer top level func multiliner: test/bad/bad_import.go:8 func (d *Data) Foobar4() {
missing defer named func oneliner:       test/bad/bad_imported.go:12    Foobar4 := func() {}

notes

either of these is valid:

go func() {
    defer func() {}()
}()

go func() {
   // defer func() {}()
}()

the purpose of this linter is to force you to consider what happens if a goroutine panics, not to force you to include empty defers.

this is similar to always considering what to do with an err, even if you decide to assign it to _.

go-hasdefault

what

a go linter to check that all switch statements have a default case.

why

sometimes a missing default is an error.

install

go install github.com/nathants/go-hasdefault@latest

usage

>> go-hasdefault $(find test/good/ -name '*.go')

>> go-hasdefault $(find test/bad/ -name '*.go')
test/bad/bad.go:3: switch statement missing default case

go-dynamolock

why

locking around dynamodb should be simple and easy.

what

a minimal go library for locking around dynamodb.

compared to alternatives it has less code and fewer features.

how

a record in dynamodb uses a uuid and a timetamp to coordinate callers.

to lock, a caller finds the uuid missing and adds it.

while locked, the caller heartbeats the timestamp.

to unlock, the caller removes the uuid.

arbitrary data can be stored atomically in the lock record. it is read via lock, and written via unlock.

manipulation of external state while the lock is held is subject to concurrent updates depending on maxAge, heartbeatInterval, and caller clock drift.

in practice, a small heartbeatInterval, a large maxAge, and reasonable clock drift should be safe.

prefer to store data within the lock when possible.

install

go get github.com/nathants/go-dynamolock

usage

package main

import (
	"context"
	"time"
	"github.com/nathants/go-dynamolock"
	"github.com/aws/aws-sdk-go/service/dynamodb/dynamodbattribute"
)


type Data struct {
    Value string
}

func main() {
	ctx := context.Background()

	// dynamodb table
	table := "table"

	// dynamodb key
	id := "lock1"

	// after a failure to unlock/heartbeat, this much time must pass before lock is available
	maxAge := time.Second * 30

	// how often to heartbeat lock timestamp
	heartbeat := time.Second * 1

	// lock and read data
	unlock, item, err := dynamolock.Lock(ctx, table, id, maxAge, heartbeat)
	if err != nil {
		// TODO handle lock contention
		panic(err)
	}
	data := &Data{}
	err = dynamodbattribute.UnmarshalMap(item, data)
	if err != nil {
		panic(err)
	}

	// do work with the lock
	time.Sleep(time.Second * 1)
	data.Value = "updated"

	// unlock and write data
	item, err = dynamodbattribute.MarshalMap(data)
	if err != nil {
		panic(err)
	}
	err = unlock(item)
	if err != nil {
		panic(err)
	}
}

c-argh

why

minimal argument parsing shouldn't require a hundred lines of code.

what

header only argument parsing for c inspired by the simplicity of argh.

example

#include <stdbool.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "argh.h"

#define USAGE "example [-l|--lz4] [-h N|--head N] [-p|--prefix] POS1 ... POSN"

int main(int argc, char **argv) {
    bool prefix = false;
    bool lz4 = false;
    int head = 0;
    ARGH_PARSE {
        ARGH_NEXT();
        if      ARGH_BOOL("-p", "--prefix") { prefix = true;}
        else if ARGH_BOOL("-l", "--lz4")    { lz4 = true; }
        else if ARGH_FLAG("-h", "--head")   { head = atol(ARGH_VAL()); }
    }
    printf("head: %d, prefix: %d, lz4: %d\n", head, prefix, lz4);
    for (int i = 0; i < ARGH_ARGC; i++)
        printf("positional arg %d: %s\n", i, ARGH_ARGV[i]);
}

>> make

>> ./example -lph5 asdf 123
head: 5, prefix: 1, lz4: 1
pos arg 0: asdf
pos arg 1: 123

>> ./example --lz4 asdf -p 123 --head 5
head: 5, prefix: 1, lz4: 1
pos arg 0: asdf
pos arg 1: 123


>> ./example asdf 123 --head 5 --lz4
head: 5, prefix: 0, lz4: 1
pos arg 0: asdf
pos arg 1: 123

backup

why

backups should be simple and easy.

how

easily create immutable, trustless backups with revision history, compression, and file deduplication.

what

the index, tracked in git, contains filesystem metadata.
the index is a sorted tsv file of: path, tarball, hash, size, mode
for every line of metadata in the index, there is one and only one tarball containing a file with that hash.
duplicate files, by blake2b hash, are never stored.
the index is encrypted with git-remote-gcrypt.
the tarballs are split into chunks, compressed with lz4, then encrypted with gpg.
all remote storage is handled via rclone on any backend it supports.
the ignore file, tracked in git, contains one regex per line of file paths to ignore.
a clean restore will clone the git repo, checkout a revision, select file paths by regex, gather needed tarball names, fetch tarballs from storage, and extract the selected files.

usage

backup-add - scan the filesystem for changes.
backup-diff - inspect the uncommitted backup diff.
backup-ignore - if needed, edit the ignore regexes, then goto backup-add.
backup-commit - commit the backup diff to remote storage.
backup-find - search for files in the index by regex at revision.
backup-restore - restore files from remote storage by regex at revision.

dependencies

awk
bash
cat
git
git-remote-gcrypt
gpg
grep
lz4
python3
rclone

installation

put bin/ on $PATH

sudo mv bin/* /usr/local/bin

setup

add some environment variables to your bashrc:

export BACKUP_ROOT=~ - root directory to backup

export BACKUP_RCLONE_REMOTE=$REMOTE - a remote setup with rclone config

export BACKUP_DESTINATION=$BUCKET/backups/$(hostname) - where to rclone data to

export BACKUP_CHUNK_MEGABYTES=100 - approximate size of each tarball before compression

have a gpg key and a gpg.conf that looks like the following:

>> cat ~/.gnupg/gpg.conf

default-key YOUR@EMAIL.COM
default-recipient YOUR@EMAIL.COM

personal-cipher-preferences AES256
personal-digest-preferences SHA512
personal-compress-preferences Uncompressed
default-preference-list SHA512 AES256 Uncompressed
cert-digest-algo SHA512
s2k-cipher-algo AES256
s2k-digest-algo SHA512
s2k-mode 3
s2k-count 65011712
disable-cipher-algo 3DES
weak-digest SHA1
force-mdc

api

modify backup state:

backup-add - scan the filesystem for changes
backup-commit - commit the backup diff to remote storage
backup-ignore - edit the ignore file in $EDITOR
backup-reset - clear uncommited backup state

view backup state:

backup-additions-sizes - show large files in the uncommited backup diff
backup-additions - inspect the uncommited backup diff, additions only
backup-diff - inspect the uncommited backup diff
backup-find - find files by regex at revision
backup-index - view the backup index
backup-log - view the git log

restore backup content:

backup-restore - restore files from remote storage by regex at revision

test

export BACKUP_TEST_RCLONE_REMOTE=$REMOTE
export BACKUP_TEST_DESTINATION=$BUCKET/test
tox

aws-gocljs

why

fullstack web should be easy and fun.

how

start with working implementations of everything, then tinker and tweak until your app is complete!

fast and reliable automation.

easy browser testing.

what

a project scaffold for a fullstack webapp on aws with an infrastructure set ready-to-deploy with libaws.

the project scaffold contains:

a go lambda backend.
a clojurescript and react frontend.
s3 and dynamodb for state.
http and websocket apis.
low latency logging.
automated devops.

a live demo on aws is here.

lambda zip

the lambda zip contains only 3 files:

>> ls -lh | awk '{print $9, $5}' | column -t

favicon.png    2.7K # favicon
index.html.gz  296K # web app
main           15M  # go binary

the index.html.gz:

<!DOCTYPE html>
<html>
    <head>
      <meta charset="utf-8">
      <meta http-equiv="Content-Security-Policy" content="script-src 'sha256-${JS_SHA256}'">
      <link rel="icon" href="/favicon.png">
    </head>
    <body>
        <div id="app"></div>
        <script type="text/javascript">
          ${JS}
        </script>
    </body>
</html>

the lambda zip itself:

>> ls -lh | awk '{print $9, $5}'
lambda.zip 4.6M

sdlc demo

dependencies

use the included Dockerfile or install the following dependencies:

npm
jdk
go
bash
entr
libaws

aws prerequisites

aws route53 has the domain or its parent from env.sh
aws acm has a wildcard cert for the domain or its parent from env.sh

install

git clone https://github.com/nathants/aws-gocljs
cd aws-gocljs
cp env.sh.template env.sh # update values

usage

bash bin/check.sh         # lint
bash bin/preview.sh       # preview changes to aws infra
bash bin/ensure.sh        # ensure aws infra
bash bin/dev.sh           # iterate on backend and frontend
bash bin/logs.sh          # tail the logs
bash bin/delete.sh        # delete aws infra

usage with bad upload bandwidth:

# bash bin/dev.sh         # this needs upload bandwidth
bash bin/dev_frontend.sh  # iterate on localhost frontend
bash bin/relay.sh         # iterate on backend via ec2 relay

aws-exec

why

building services on lambda should be easy and fun.

what

a project scaffold for a backend service on aws with an infrastructure set ready-to-deploy with libaws.

the project scaffold makes it easy to:

authenticate callers.
implement fast synchronous apis that return all results immediately.
implement slow asynchronous apis with streaming logs, exit code, and 15 minutes max duration.
use the web admin interface, even from a phone.
use the cli admin interface, executing locally or on lambda.
use the api interface, calling efficiently from other backend services.

how

synchronous apis are normal http on lambda.

asynchronous apis are a http post that triggers an async lambda which invokes a command via rpc or subprocess and stores the results in s3.

each invocation creates 3 objects in s3:
- log: all stdout and stderr, updated in its entirety every second.
- exit: the exit code of the command, written once.
- size: the size in bytes of the log after the final update, written once, written last.
objects are stored in either:
- aws-exec private s3.
- presigned s3 put urls provided by the caller.
to follow invocation status, the caller:
- polls the log object with increasing range-start.
- stops when the size object exists and range-start equals size.
- returns the exit object.

there are three ways to invoke an asynchronous api:

api invoke via rpc, this is faster.
cli invoke via subprocess, this is slower.
web invoke via subprocess, this is slower.

add a new synchronous functionality

add to api/.

duplicate the httpExecGet or httpExecPost handler and modify it to introduce new functionality.

add a new asynchronous functionality

add to cmd/.

duplicate the listdir command and modify it to introduce new functionality.

web demo

cli demo

api demo

mobile demo

dependencies

use the included Dockerfile or install the following dependencies:

npm
jdk
go
bash
entr
libaws

aws prerequisites

aws route53 has the domain or its parent from env.sh
aws acm has a wildcard cert for the domain or its parent from env.sh

usage

go install github.com/nathants/libaws@latest
export PATH=$PATH:$(go env GOPATH)/bin

cp env.sh.template env.sh # update values
bash bin/check.sh         # lint
bash bin/preview.sh       # preview changes to aws infra
bash bin/ensure.sh        # ensure aws infra
bash bin/dev.sh           # iterate on backend and frontend
bash bin/logs.sh          # tail the logs
bash bin/delete.sh        # delete aws infra
bash bin/cli.sh -h        # interact with the service via the cli

usage with bad upload bandwidth:

# bash bin/dev.sh         # this needs upload bandwidth
bash bin/dev_frontend.sh  # iterate on localhost frontend
bash bin/relay.sh         # iterate on backend via ec2 relay

usage with docker

cp env.sh.template env.sh # update values
docker build -t aws-exec:latest .
docker run -it --rm \
    -v $(pwd):/code \
    -e AWS_DEFAULT_REGION \
    -e AWS_ACCESS_KEY_ID \
    -e AWS_SECRET_ACCESS_KEY \
    aws-exec:latest \
    bash -c '
        cd /code
        bash bin/ensure.sh
    '

create auth

bash bin/cli.sh auth-new test-user

install and use cli

go install github.com/nathants/aws-exec@latest
export PATH=$PATH:$(go env GOPATH)/bin

export AUTH=$AUTH
export PROJECT_DOMAIN=$DOMAIN
aws-exec exec -- whoami

install and use api

go get github.com/nathants/aws-exec@latest

package cmd

import (
	"context"
	"encoding/json"
	"fmt"
	"os"

	awsexec "github.com/nathants/aws-exec/exec"
)

func main() {
	val, err := json.Marshal(map[string]interface{}{
		"path": ".",
	})
	if err != nil {
	    panic(err)
	}
	exitCode, err := awsexec.Exec(context.Background(), &awsexec.Args{
		Url:     "https://%s" + os.Getenv("PROJECT_DOMAIN"),
		Auth:    os.Getenv("AUTH"),
		RpcName: "listdir",
		RpcArgs: string(val),
		LogDataCallback: func(logs string) {
			fmt.Print(logs)
		},
	})
	if err != nil {
		panic(err)
	}
	os.Exit(exitCode)
}

aws-ensure-route53

why

managing your dns in route53 across multiple accounts should be easy.

what

tooling to make managing your dns in route53 across multiple accounts simple and easy using libaws.

install

go install github.com/nathants/libaws@latest

export PATH=$PATH:$(go env GOPATH)/bin

or use the dockerfile

usage

bootstrap

clone this repo, and setup a new private remote. you will version your dns data here. you probably don't want this on public github.

setup your credentials using: libaws creds-add -h

you can now list all your credentials with: libaws creds-ls

initialize

pull all your dns records across all accounts with: bash bin/pull.sh

commit this initial data.

your repo now looks like:

>> tree
├── accounts
│   ├── work-prod
│   │   └── dns.txt
│   ├── work-staging
│   │   └── dns.txt
│   ├── work-scratch
│   │   └── dns.txt
│   ├── personal-prod
│   │   └── dns.txt
│   └── personal-scratch
│       └── dns.txt
└── bin
    ├── ensure_all.sh
    ├── ensure.sh
    ├── preview_all.sh
    ├── preview.sh
    └── pull.sh

the dns.txt files contain entries created by route53-ls that look like:

example.com example.com       Type=A     TTL=60 Value=1.1.1.1 Value=2.2.2.2
example.com cname.example.com Type=CNAME TTL=60 Value=about.us-west-2.domain.example.com
example.com alias.example.com Type=Alias        Value=d-XXX.execute-api.us-west-2.amazonaws.com     HostedZoneId=XXX

update

you can now modify or add entires to these files, and deploy them.

you could make a change like:

>> git diff
diff --git a/accounts/work-prod/dns.txt b/accounts/work-prod/dns.txt
index 4b959e4..67415b7 100644
--- a/accounts/work-prod/dns.txt
+++ b/accounts/work-prod/dns.txt
@@ -1,4 +1,4 @@
-example.com foo.example.com Type=CNAME TTL=300 Value=bar
+example.com foo.example.com Type=CNAME TTL=300 Value=barr

preview

to preview those changes, use bash bin/preview_all.sh or bash bin/preview.sh work-prod, which looks like:

>> bash bin/preview_all.sh
preview dns: work-prod
lib/route53.go:258: preview: route53 update Values for foo.example.com: ["bar"] => ["barr"]
preview dns: work-staging
preview dns: work-scratch
preview dns: personal-prod
preview dns: personal-scratch

no output means no changes.

deploy

to deploy those changes using route53-ensure-record, use bash bin/ensure_all.sh or bash bin/ensure.sh work-prod, which looks like:

>> bash bin/ensure_all.sh
ensure dns: work-prod
lib/route53.go:258: route53 update Values for foo.example.com: ["bar"] => ["barr"]
lib/route53.go:284: route53 updated record: foo.example.com
ensure dns: work-staging
ensure dns: work-scratch
ensure dns: personal-prod
ensure dns: personal-scratch

delete

like all the ensure functions in libaws, ensure creates or updates infrastructure as needed, but does not remove it.

to delete a record, remove it from its dns.txt:

>> git diff
diff --git a/accounts/dns/dns.txt b/accounts/dns/dns.txt
index 4b959e4..dd68522 100644
--- a/accounts/work-prod/dns.txt
+++ b/accounts/work-prod/dns.txt
@@ -1,4 +1,3 @@
-example.com foo.example.com Type=CNAME TTL=300 Value=barr

then preview the delete using route53-rm-record:

>> libaws route53-rm-record --preview example.com foo.example.com Type=CNAME TTL=300 Value=barr
lib/route53.go:85: preview: route53 deleted record foo.example.com: TTL=300 Type=CNAME Value=barr

then perform the delete using route53-rm-record:

>> libaws route53-rm-record example.com foo.example.com Type=CNAME TTL=300 Value=barr
lib/route53.go:85: route53 deleted record foo.example.com: TTL=300 Type=CNAME Value=barr

monitoring

add pull.sh to your crontab to keep track of changes to your dns:

0 15 * * * bash -c 'cd ~/repos/aws-ensure-route53 && bash bin/pull.sh'

when you notice uncommited changes in git status, you can either commit them, or investigate them. foo likely should not be barr.

agr

why

repo wide search and replace should be easier.

what

like ack or ag, but for search and replace.

can climb to the root of a git repo before running.

shows a preview of the replacements to be made.

prompts to continue globally or at each change site.

install

python3 -m pip install git+https://github.com/nathants/agr

usage

agr '(\w+)_factory' '\1_factory_factory'

dependencies

examples

$ agr '(\w+)_factory' '\1_factory_factory'
> lib.py:14: def poodle_factory_factory(): => def poodle_factory_factory_factory():
> main.py:3: def dog_factory_factory(): => def dog_factory_factory_factory():
> proceed? y/n

$ agr '(\w+)_factory' --delete
> lib.py:14: def poodle_factory_factory(): => DELETED!
> main.py:3: def dog_factory_factory(): => DELETED!
> proceed? y/n

>> agr -h
usage: agr [-h] [-d] [-p] [-s] [-u] [-n] [-y] [-e] pattern [replacement]

positional arguments:
  pattern             regex to match
  replacement         replacement for matches (default: -)

optional arguments:
  -h, --help          show this help message and exit
  -d, --delete        rather than substitute replacement, delete the matched line (default: False)
  -p, --preview       show diffs and then exit without prompting for commit (default: False)
  -s, --short         show shorter diffs (default: False)
  -u, --unrestricted  process all files, not just code files (default: False)
  -n, --no-climb      no climbing upwards until a .git dir is found (default: False)
  -y, --yes           commit without prompting (default: False)
  -e, --each          prompt for y/n at each change site (default: False)

go-libsodium

why

libsodium should be easy.

how

a minimal cgo interface to the following libsodium constructs:

what


func Init()

func StreamKeygen() (key []byte, err error)

func StreamEncrypt(key []byte, plainText io.Reader, cipherText io.Writer) error

func StreamDecrypt(key []byte, cipherText io.Reader, plainText io.Writer) error

func StreamEncryptRecipients(publicKeys [][]byte, plainText io.Reader, cipherText io.Writer) error

func StreamDecryptRecipients(secretKey []byte, cipherText io.Reader, plainText io.Writer) error

func BoxKeypair() (publicKey, secretKey []byte, err error)

func BoxSealedEncrypt(plainText, recipientPublicKey []byte) (cipherText []byte, err error)

func BoxSealedDecrypt(cipherText, recipientSecretKey []byte) (plainText []byte, err error)

func BoxEasyEncrypt(plainText, recipientPublicKey, senderSecretKey []byte) (cipherText []byte, err error)

func BoxEasyDecrypt(cipherText, senderPublicKey, recipientSecretKey []byte) (plainText []byte, err error)

func SignKeypair() (publicKey, secretKey []byte, err error)

func Sign(plainText, signerSecretKey []byte) (signedText []byte, err error)

func SignVerify(signedText, plainText, signerPublicKey []byte) error

install

brew install         go     libsodium     # homebrew
sudo pacman -S       go     libsodium     # arch
sudo apk add         go     libsodium-dev # alpine
sudo apt-get install golang libsodium-dev # ubuntu/debian

go get github.com/nathants/go-libsodium

usage

package main

import (
	"bytes"
	"fmt"

	"github.com/nathants/go-libsodium"
)

func Stream() {
	libsodium.Init()
	key, err := libsodium.StreamKeygen()
	if err != nil {
		panic(err)
	}
	value := []byte("hello world")
	var cipher bytes.Buffer
	err = libsodium.StreamEncrypt(key, bytes.NewReader(value), &cipher)
	if err != nil {
		panic(err)
	}
	var plain bytes.Buffer
	err = libsodium.StreamDecrypt(key, bytes.NewReader(cipher.Bytes()), &plain)
	if err != nil {
		panic(err)
	}
	fmt.Println("stream", bytes.Equal(value, plain.Bytes()))
}

func StreamRecipients() {
	libsodium.Init()
	pk1, sk1, err := libsodium.BoxKeypair()
	if err != nil {
		panic(err)
	}
	pk2, sk2, err := libsodium.BoxKeypair()
	if err != nil {
		panic(err)
	}
	value := []byte("hello world")
	var cipher bytes.Buffer
	err = libsodium.StreamEncryptRecipients([][]byte{pk1, pk2}, bytes.NewReader(value), &cipher)
	if err != nil {
		panic(err)
	}
	var plain bytes.Buffer
	err = libsodium.StreamDecryptRecipients(sk1, bytes.NewReader(cipher.Bytes()), &plain)
	if err != nil {
		panic(err)
	}
	fmt.Println("recipient1", bytes.Equal(value, plain.Bytes()))
	plain.Reset()
	err = libsodium.StreamDecryptRecipients(sk2, bytes.NewReader(cipher.Bytes()), &plain)
	if err != nil {
		panic(err)
	}
	fmt.Println("recipient2", bytes.Equal(value, plain.Bytes()))
}

func BoxSeal() {
	libsodium.Init()
	value := []byte("hello world")
	pk, sk, err := libsodium.BoxKeypair()
	if err != nil {
	    panic(err)
	}
	cipher, err := libsodium.BoxSealedEncrypt(value, pk)
	if err != nil {
	    panic(err)
	}
	plain, err := libsodium.BoxSealedDecrypt(cipher, sk)
	if err != nil {
	    panic(err)
	}
	fmt.Println("seal", bytes.Equal(value, plain))
}

func BoxEasy() {
	value := []byte("hello world")
	pk1, sk1, err := libsodium.BoxKeypair()
	if err != nil {
	    panic(err)
	}
	pk2, sk2, err := libsodium.BoxKeypair()
	if err != nil {
	    panic(err)
	}
	cipher, err := libsodium.BoxEasyEncrypt(value, pk2, sk1)
	if err != nil {
	    panic(err)
	}
	plain, err := libsodium.BoxEasyDecrypt(cipher, pk1, sk2)
	if err != nil {
	    panic(err)
	}
	fmt.Println("easy", bytes.Equal(value, plain))
}

func Sign() {
	value := []byte("hello world")
	pk, sk, err := libsodium.SignKeypair()
	if err != nil {
	    panic(err)
	}
	signature, err := libsodium.Sign(value, sk)
	if err != nil {
	    panic(err)
	}
	err = libsodium.SignVerify(signature, value, pk)
	if err != nil {
	    panic(err)
	}
	fmt.Println("signature")
}

func main() {
	Stream()
	StreamRecipients()
	BoxSeal()
	BoxEasy()
	Sign()
}

git-remote-aws

why

encrypted git hosting should be easy.

how

encrypted git bundles are stored in s3.

compare and swap against dynamodb updates an ordered list of bundles. this enables multiple writers to safely collaborate on a single remote.

each remote can hold one and only one branch.

bundles in s3 are immutable, and force push is not allowed.

bundles are encrypted with libsodium secretstream. user keys are libsodium box keypairs. authorized user public keys are added to a .publickeys file in the git repository. to add or remove authorized users, update the publickeys file, then create and push to a new remote or delete s3 data and recreate an existing remote.

metadata is stored unencrypted:

branch name
remote name
git hash for the start and end of each bundle

data is stored encrypted:

git bundles

both git sha1 and sha256 hashing algorithms are supported.

private s3 buckets and dynamodb tables are created ondemand if they do not already exist.

what

a custom git remote adding support for remotes like:

git remote add origin aws://${s3_bucket}+${dynamo_table}/${remote_name}

the git remote binary provides a keygen for libsodium box keypairs:

git-remote-aws --keygen ~/.git-remote-aws/publickey ~/.git-remote-aws/secretkey

the default path for your secret key is ~/.git-remote-aws/secretkey. this can be changed via environment variable GIT_REMOTE_AWS_SECRETKEY.

install

install go and libsodium from your package manager:

brew install         go     libsodium     # homebrew
sudo pacman -S       go     libsodium     # arch
sudo apk add         go     libsodium-dev # alpine
sudo apt-get install golang libsodium-dev # ubuntu/debian

install the binary and update PATH:

go install github.com/nathants/git-remote-aws@latest

export PATH=$PATH:$(go env GOPATH)/bin

usage

>> git init

>> git remote add origin aws://${bucket}+${table}/myrepo

>> mkdir -p ~/.git-remote-aws

>> git-remote-aws --keygen ~/.git-remote-aws/publickey ~/.git-remote-aws/secretkey

>> cat ~/.git-remote-aws/publickey >> .publickeys

>> git add .

>> git commit -m init

>> git push -u origin master

creating private s3 bucket: $bucket
lib/s3.go:329: created bucket: $bucket
lib/s3.go:367: created bucket tags for: $bucket
lib/s3.go:415: created public access block for $bucket: private
lib/s3.go:657: created encryption for $bucket: true
lib/s3.go:688: put bucket metrics for: $bucket
created private s3 bucket: $bucket
creating private dynamodb table: $table
lib/dynamodb.go:481: created table: $table
lib/dynamodb.go:974: waiting for table active: $table
lib/dynamodb.go:974: waiting for table active: $table
created private dynamodb table: $table
get dynamodb://$table/$bucket/myrepo
get dynamodb://$table/$bucket/myrepo
get s3://$bucket/
git bundle: 0000000000000000000000000000000000000000..daf8ea23a2aa082a3eeffacbdda04917d14916cc
put s3://$bucket/myrepo/0000000000000000000000000000000000000000..daf8ea23a2aa082a3eeffacbdda04917d14916cc
put s3://$bucket/myrepo/bundles_daf8ea23a2aa082a3eeffacbdda04917d14916cc
put dynamodb://$table/$bucket/myrepo
To aws://$bucket+$table/myrepo
 * [new branch]      master -> master

>> libaws s3-ls $bucket/ -r

770 myrepo/0000000000000000000000000000000000000000..daf8ea23a2aa082a3eeffacbdda04917d14916cc
 82 myrepo/bundles_daf8ea23a2aa082a3eeffacbdda04917d14916cc

>> libaws dynamodb-item-scan $table | jq .

{
  "branch": "master",
  "bundles": "myrepo/bundles_daf8ea23a2aa082a3eeffacbdda04917d14916cc",
  "id": "$bucket/myrepo",
  "uid": null,
  "unix": 0
}

>> cd $(mktemp -d)

>> git clone aws://${bucket}+${table}/myrepo

Cloning into 'myrepo'...
get dynamodb://$table/$bucket/myrepo
get s3://$bucket/myrepo/bundles_daf8ea23a2aa082a3eeffacbdda04917d14916cc
get dynamodb://$table/$bucket/myrepo
get s3://$bucket/myrepo/bundles_daf8ea23a2aa082a3eeffacbdda04917d14916cc
get s3://$bucket/myrepo/0000000000000000000000000000000000000000..daf8ea23a2aa082a3eeffacbdda04917d14916cc
git unbundle: 0000000000000000000000000000000000000000..daf8ea23a2aa082a3eeffacbdda04917d14916cc
get dynamodb://$table/$bucket/myrepo
get s3://$bucket/myrepo/bundles_daf8ea23a2aa082a3eeffacbdda04917d14916cc

libaws

why

aws is amazing, but it's hard to see the forest for the trees.

aws should:

have fewer knobs
have sane defaults
be easy to use
be hard to screw up
be fast
be fun
have a tldr

it should be easy for a lambda to react to:

docker push to ecr
s3 put object
dynamodb put item
sqs send message
time passing
http requests
websocket messages

it should be easy to create:

vpcs
security groups
instance profiles
keypairs

how

declare and deploy groups of related aws infrastructure as infrastructure sets:

that contain:
- lambdas
- s3 buckets
- dynamodb tables
- sqs queues
- vpcs
- security groups
- instance profiles
- keypairs
that react to lambda triggers:
- http apis
- websocket messages
- s3 bucket writes
- dynamodb table writes
- sqs queue puts
- cron schedules
- ecr docker pushes

what

a simpler way to declare aws infrastructure that is easy to use and extend.

there are two ways to use it:

yaml and the cli
go structs and the go api

the primary entrypoints are:

infra-ensure: deploy an infrastructure set.

libaws infra-ensure ./infra.yaml --preview
libaws infra-ensure ./infra.yaml

infra-ls: view infrastructure sets.
```
libaws infra-ls
```

infra-ensure --quick: quickly update lambda code.

libaws infra-ensure ./infra.yaml --quick LAMBDA_NAME

infra-rm: remove an infrastructure set.

libaws infra-rm ./infra.yaml --preview
libaws infra-rm ./infra.yaml

infra-ensure is a positive assertion. it asserts that some named infrastructure exists, and is configured correctly, creating or updating it if needed.

many other entrypoints exist, and can be explored by type. they fall into two categories:

mutate aws state:

>> libaws -h | grep ensure | wc -l
19

>> libaws -h | grep new | wc -l
1

>> libaws -h | grep rm | wc -l
26

view aws state:

>> libaws -h | grep ls | wc -l
33

>> libaws -h | grep describe | wc -l
6

>> libaws -h | grep get | wc -l
16

>> libaws -h | grep scan | wc -l
1

aws sdk, pulumi, terraform, cloudformation, and serverless

compared to the full aws api, systems declared as infrastructure sets:

have simpler examples.
have fewer knobs.
are easier to use.
are harder to screw up.
are almost always enough, and easy to extend.
are more fun.

if you want to use the full aws api, there are many great tools:

readme index

install
- cli
- go api
tldr
usage
infrastructure set
typical usage
design
tradeoffs
infra.yaml
- environment variable substitution
- name
- s3
- dynamodb
- sqs
- keypair
- vpc
  - security group
- instance profile
- lambda
  - entrypoint
  - attr
  - policy
  - allow
  - env
  - include
  - require
  - trigger
    - api
    - websocket
    - s3
    - dynamodb
    - sqs
    - schedule
    - ecr
bash completion
extending
testing

install

cli

go install github.com/nathants/libaws@latest

export PATH=$PATH:$(go env GOPATH)/bin

go api

go get github.com/nathants/libaws@latest

tldr

define an infrastructure set

>> cd examples/simple/go/s3 && tree
.
├── infra.yaml
└── main.go

name: test-infraset-${uid}

s3:
  test-bucket-${uid}:
    attr:
      - acl=private

lambda:
  test-lambda-${uid}:
    entrypoint: main.go
    attr:
      - concurrency=0
      - memory=128
      - timeout=60
    policy:
      - AWSLambdaBasicExecutionRole
    trigger:
      - type: s3
        attr:
          - test-bucket-${uid}

package main

import (
	"context"
	"fmt"
	"github.com/aws/aws-lambda-go/events"
	"github.com/aws/aws-lambda-go/lambda"
)

func handleRequest(_ context.Context, e events.S3Event) (events.APIGatewayProxyResponse, error) {
	for _, record := range e.Records {
		fmt.Println(record.S3.Object.Key)
	}
	return events.APIGatewayProxyResponse{StatusCode: 200}, nil
}

func main() {
	lambda.Start(handleRequest)
}

ensure the infrastructure set

view the infrastructure set

depth based colors by yaml

trigger the infrastructure set

quickly update lambda code

delete the infrastructure set

usage

explore the cli

>> libaws -h | grep ensure | head

codecommit-ensure             - ensure a codecommit repository
dynamodb-ensure               - ensure a dynamodb table
ec2-ensure-keypair            - ensure a keypair
ec2-ensure-sg                 - ensure a sg
ecr-ensure                    - ensure ecr image
iam-ensure-ec2-spot-roles     - ensure iam ec2 spot roles that are needed to use ec2 spot
iam-ensure-instance-profile   - ensure an iam instance-profile
iam-ensure-role               - ensure an iam role
iam-ensure-user-api           - ensure an iam user with api key
iam-ensure-user-login         - ensure an iam user with login

explore a cli entrypoint

>> libaws s3-ensure -h

ensure a s3 bucket

example:
 - libaws s3-ensure test-bucket acl=public versioning=true

optional attrs:
 - acl=VALUE        (values = public | private, default = private)
 - versioning=VALUE (values = true | false,     default = false)
 - metrics=VALUE    (values = true | false,     default = true)
 - cors=VALUE       (values = true | false,     default = false)
 - ttldays=VALUE    (values = 0 | n,            default = 0)

setting 'cors=true' uses '*' for allowed origins. to specify one or more explicit origins, do this instead:
 - corsorigin=http://localhost:8080
 - corsorigin=https://example.com

Usage: s3-ensure [--preview] NAME [ATTR [ATTR ...]]

Positional arguments:
  NAME
  ATTR

Options:
  --preview, -p
  --help, -h             display this help and exit

explore the go api

package main

import (
	"github.com/nathants/libaws/lib"
)

func main() {
    lib. (TAB =>)
      |--------------------------------------------------------------------------------|
      |f AcmClient func() *acm.ACM (Function)                                          |
      |f AcmClientExplicit func(accessKeyID string, accessKeySecret string, region stri|
      |f AcmListCertificates func(ctx context.Context) ([]*acm.CertificateSummary, erro|
      |f Api func(ctx context.Context, name string) (*apigatewayv2.Api, error) (Functio|
      |f ApiClient func() *apigatewayv2.ApiGatewayV2 (Function)                        |
      |f ApiClientExplicit func(accessKeyID string, accessKeySecret string, region stri|
      |f ApiList func(ctx context.Context) ([]*apigatewayv2.Api, error) (Function)     |
      |f ApiListDomains func(ctx context.Context) ([]*apigatewayv2.DomainName, error) (|
      |f ApiUrl func(ctx context.Context, name string) (string, error) (Function)      |
      |f ApiUrlDomain func(ctx context.Context, name string) (string, error) (Function)|
      |...                                                                             |
      |--------------------------------------------------------------------------------|
}

explore simple examples

api: python, go, docker
dynamodb: python, go, docker
ecr: python, go, docker
includes: python, go
s3: python, go, docker
schedule: python, go, docker
sqs: python, go, docker
websocket: python, go, docker

explore complex examples

s3-ec2:
- write to s3 in-bucket
- which triggers lambda
- which launches ec2 spot
- which reads from in-bucket, writes to out-bucket, and terminates

explore external examples

infrastructure set

an infrastructure set is defined by yaml or go struct and contains:

stateful infrastructure:
- s3
- dynamodb
- sqs
ec2 infrastructure:
lambdas:
- triggers:
  - api
  - websocket
  - s3
  - dynamodb
  - sqs
  - schedule
  - ecr

typical usage

use infra-ensure to deploy an infrastructure set.

libaws infra-ensure ./infra.yaml --preview
libaws infra-ensure ./infra.yaml

use infra-ls to view infrastructure sets.
```
libaws infra-ls
```
use infra-ensure --quick LAMBDA_NAME to quickly update lambda code.
```
libaws infra-ensure ./infra.yaml --quick LAMBDA_NAME
```

use infra-rm to remove an infrastructure set.

libaws infra-rm ./infra.yaml --preview
libaws infra-rm ./infra.yaml

design

there is no implicit coordination.
- if you aren't already serializing your infrastructure mutations, lock around dynamodb.
there are only two state locations:
- aws.
- your code.
aws infrastructure is uniquely identified by name.
- all aws infrastructure share a private namespace scoped to account/region. use good names.
- except s3, which shares a public namespace scoped to earth. use better names.
mutative operations manipulate aws state.
- mutative operations are idempotent. if they fail due to a transient error, run them again.
- mutative operations can --preview. no output means no changes.
ensure are mutative operations that create or update infrastructure.
rm are mutative operations that delete infrastructure.
ls, get, scan, and describe operations are non-mutative.
multiple infrastructure sets can be deployed into the same account/region.

tradeoffs

no attempt is made to avoid vendor lock-in.
- migrating between cloud providers will always be non-trivial.
- attempting to mitigate future migrations has more cost than benefit in the typical case.
ensure operations are positive assertions. they assert that some named infrastructure exists, and is configured correctly, creating or updating it if needed.
- positive assertions CANNOT remove top level infrastructure, but CAN remove configuration from them.
- removing a trigger, policy, or allow WILL remove that from the lambda.
- removing policy, or allow WILL remove that from the instance-profile.
- removing a security-group WILL remove that from the vpc.
- removing a rule WILL remove that from the security-group.
- removing an attr WILL remove that from a sqs, s3, dynamodb, or lambda.
- removing a keypair, vpc, instance-profile, sqs, s3, dynamodb, or lambda WON'T remove that from the account/region.
  - the operator decides IF and WHEN top level infrastructure should be deleted, then uses an rm operation to do so.
  - as a convenience, infra-rm will remove ALL infrastructure CURRENTLY declared in an infra.yaml.
when using ensure operations, no output means no changes.
- for large infrastructure sets, this can mean a minute or two without output if no changes are needed.
- to see a lot of output instead of none, set this environment variable:
```
export DEBUG=yes
```
infra-ls is designed to list aws accounts managed with infra-ensure. it will not work well in other scenarios.

infra.yaml

use an infra.yaml file to declare an infrastructure set. the schema is as follows:

name: VALUE
lambda:
  VALUE:
    entrypoint: VALUE
    policy:     [VALUE ...]
    allow:      [VALUE ...]
    attr:       [VALUE ...]
    require:    [VALUE ...]
    env:        [VALUE ...]
    include:    [VALUE ...]
    trigger:
      - type: VALUE
        attr: [VALUE ...]
s3:
  VALUE:
    attr: [VALUE ...]
dynamodb:
  VALUE:
    key:  [VALUE ...]
    attr: [VALUE ...]
sqs:
  VALUE:
    attr: [VALUE ...]
vpc:
  VALUE:
    security-group:
      VALUE:
        rule: [VALUE ...]
keypair:
  VALUE:
    pubkey-content: VALUE
instance-profile:
  VALUE:
    allow: [VALUE ...]
    policy: [VALUE ...]

environment variable substitution

anywhere in infra.yaml you can substitute environment variables from the caller's environment:

example:

s3:
  test-bucket-${uid}:
    attr:
      - versioning=${versioning}

the following variables are defined during deployment, and are useful in allow declarations:

${API_ID} the id of the apigateway v2 api created by an api trigger.
${WEBSOCKET_ID} the id of the apigateway v2 websocket created by a websocket trigger.

name

defines the name of the infrastructure set.

schema:
```
name: VALUE
```
example:
```
name: test-infraset
```

s3

defines a s3 bucket:

the following attributes can be defined:
- acl=VALUE, values: public | private, default: private
- versioning=VALUE, values: true | false, default: false
- metrics=VALUE, values: true | false, default: true
- cors=VALUE, values: true | false, default: false
- ttldays=VALUE, values: 0 | n, default: 0
setting cors=true uses * for allowed origins. to specify one or more explicit origins, do this instead:
- corsorigin=http://localhost:8080
- corsorigin=https://example.com
schema:
```
s3:
  VALUE:
    attr:
      - VALUE
```

example:

s3:
  test-bucket:
    attr:
      - versioning=true
      - acl=public

dynamodb

defines a dynamodb table:

specify key as:
- NAME:ATTR_TYPE:KEY_TYPE
the following attributes can be defined:
- read=VALUE, provisioned read capacity, default: 0
- write=VALUE, provisioined write capacity, default: 0
on global indices the following attributes can be defined:
- projection=VALUE, provisioned read capacity, default: ALL
- read=VALUE, provisioned read capacity, default: 0
- write=VALUE, provisioined write capacity, default: 0
on local indices the following attributes can be defined:
- projection=VALUE, provisioned read capacity, default: ALL

schema:

dynamodb:
  VALUE:
    key:
      - NAME:ATTR_TYPE:KEY_TYPE
    attr:
      - VALUE
    global-index:
      VALUE:
        key:
          - NAME:ATTR_TYPE:KEY_TYPE
        non-key:
          - NAME
        attr:
          - VALUE
    local-index:
      VALUE:
        key:
          - NAME:ATTR_TYPE:KEY_TYPE
        non-key:
          - NAME
        attr:
          - VALUE

example:

dynamodb:
  stream-table:
    key:
      - userid:s:hash
      - timestamp:n:range
    attr:
      - stream=keys_only
  auth-table:
    key:
      - id:s:hash
    attr:
      - write=50
      - read=150

example global secondary index:

dynamodb:
  test-table:
    key:
      - id:s:hash
    global-index:
      test-index:
        key:
          - hometown:s:hash

example local secondary index:

dynamodb:
  test-table:
    key:
      - id:s:hash
    local-index:
      test-index:
        key:
          - hometown:s:hash

sqs

defines a sqs queue:

the following attributes can be defined:
- delay=VALUE, delay seconds, default: 0
- size=VALUE, maximum message size bytes, default: 262144
- retention=VALUE, message rentention period seconds, default: 345600
- wait=VALUE, receive wait time seconds, default: 0
- timeout=VALUE, visibility timeout seconds, default: 30
schema:
```
sqs:
  VALUE:
    attr:
      - VALUE
```

example:

sqs:
  test-queue:
    attr:
      - delay=20
      - timeout=300

keypair

defines an ec2 keypair.

example:

keypair:
  VALUE:
    pubkey-content: VALUE

example:

keypair:
  test-keypair:
    pubkey-content: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICVp11Z99AySWfbLrMBewZluh7cwLlkjifGH5u22RXor

vpc

defines a default-like vpc with an internet gateway and public access.

schema:
```
vpc:
  VALUE: {}
```
example:
```
vpc:
  test-vpc: {}
```

security group

defines a security group on a vpc

schema:

vpc:
  VALUE:
    security-group:
      VALUE:
        rule:
          - PROTO:PORT:SOURCE

example:

vpc:
  test-vpc:
    security-group:
      test-sg:
        rule:
          - tcp:22:0.0.0.0/0

instance profile

defines an ec2 instance profile.

schema:

instance-profile:
  VALUE:
    allow:
      - SERVICE:ACTION ARN
    policy:
      - VALUE

example:

instance-profile:
  test-profile:
    allow:
      - s3:* *
    policy:
      - AWSLambdaBasicExecutionRole

lambda

defines a lambda.

schema:
```
lambda:
  VALUE: {}
```
example:
```
lambda:
  test-lambda: {}
```

entrypoint

defines the code of the lambda. it is one of:

a python file.
a go file.
an ecr container uri.
schema:
```
lambda:
  VALUE:
    entrypoint: VALUE
```

example:

lambda:
  test-lambda:
    entrypoint: main.go

attr

defines lambda attributes. the following can be defined:

concurrency defines the reserved concurrent executions, default: 0
memory defines lambda ram in megabytes, default: 128
timeout defines the lambda timeout in seconds, default: 300
logs-ttl-days defines the ttl days for cloudwatch logs, default: 7

schema:

lambda:
  VALUE:
    attr:
      - KEY=VALUE

example:

lambda:
  test-lambda:
    attr:
      - concurrency=100
      - memory=256
      - timeout=60
      - logs-ttl-days=1

policy

defines policies on the lambda's iam role.

schema:

lambda:
  VALUE:
    policy:
      - VALUE

example:

lambda:
  test-lambda:
    policy:
      - AWSLambdaBasicExecutionRole

allow

defines allows on the lambda's iam role.

schema:

lambda:
  VALUE:
    allow:
      - SERVICE:ACTION ARN

example:

lambda:
  test-lambda:
    allow:
      - s3:* *
      - dynamodb:* arn:aws:dynamodb:*:*:table/test-table

env

defines environment variables on the lambda:

schema:

lambda:
  VALUE:
    env:
      - KEY=VALUE

example:

lambda:
  test-lambda:
    env:
      - kind=production

include

defines extra content to include in the lambda zip:

this is ignored when entrypoint is an ecr container uri.

schema:

lambda:
  VALUE:
    include:
      - VALUE

example:

lambda:
  test-lambda:
    include:
      - ./cacerts.crt
      - ../frontend/public/*

require

defines dependencies to install with pip in the virtualenv zip.

this is ignored unless the entrypoint is a python file.

schema:

lambda:
  VALUE:
    require:
      - VALUE

example:

lambda:
  test-lambda:
    require:
      - fastapi==0.76.0

trigger

defines triggers for the lambda:

schema:

lambda:
  VALUE:
    trigger:
      - type: VALUE
        attr:
          - VALUE

example:

lambda:
  test-lambda:
    trigger:
      - type: dynamodb
        attr:
          - test-table

trigger types

api

defines an apigateway v2 http api:

add a custom domain with attr: domain=api.example.com
add a custom domain and update route53 with attr: dns=api.example.com
- this domain, or its parent domain, must already exist as a hosted zone in route53.
- this domain, or its parent domain, must already have an acm certificate with subdomain wildcard.

schema:

lambda:
  VALUE:
    trigger:
      - type: api
        attr:
          - VALUE

example:

lambda:
  test-lambda:
    trigger:
      - type: api
        attr:
          - dns=api.example.com

websocket

defines an apigateway v2 websocket api:

add a custom domain with attr: domain=ws.example.com
add a custom domain and update route53 with attr: dns=ws.example.com
- this domain, or its parent domain, must already exist as a hosted zone in route53-ls.
- this domain, or its parent domain, must already have an acm certificate with subdomain wildcard.

schema:

lambda:
  VALUE:
    trigger:
      - type: websocket
        attr:
          - VALUE

example:

lambda:
  test-lambda:
    trigger:
      - type: websocket
        attr:
          - dns=ws.example.com

s3

defines an s3 trigger:

the only attribute must be the bucket name.
object creation and deletion invoke the trigger.

schema:

lambda:
  VALUE:
    trigger:
      - type: s3
        attr:
          - VALUE

example:

lambda:
  test-lambda:
    trigger:
      - type: s3
        attr:
          - test-bucket

dynamodb

defines a dynamodb trigger:

the first attribute must be the table name.
the following trigger attributes can be defined:
- batch=VALUE, maximum batch size, default: 100
- parallel=VALUE, parallelization factor, default: 1
- retry=VALUE, maximum retry attempts, default: -1
- window=VALUE, maximum batching window in seconds, default: 0
- start=VALUE, starting position

schema:

lambda:
  VALUE:
    trigger:
      - type: dynamodb
        attr:
          - VALUE

example:

lambda:
  test-lambda:
    trigger:
      - type: dynamodb
        attr:
          - test-table
          - start=trim_horizon

sqs

defines a sqs trigger:

the first attribute must be the queue name.
the following trigger attributes can be defined:
- batch=VALUE, maximum batch size, default: 10
- window=VALUE, maximum batching window in seconds, default: 0

schema:

lambda:
  VALUE:
    trigger:
      - type: sqs
        attr:
          - VALUE

example:

lambda:
  test-lambda:
    trigger:
      - type: sqs
        attr:
          - test-queue

schedule

defines a schedule trigger:

the only attribute must be the schedule expression.

schema:

lambda:
  VALUE:
    trigger:
      - type: schedule
        attr:
          - VALUE

example:

lambda:
  test-lambda:
    trigger:
      - type: schedule
        attr:
          - rate(24 hours)

ecr

defines an ecr trigger:

successful image actions to any ecr repository will invoke the trigger.

schema:

lambda:
  VALUE:
    trigger:
      - type: ecr

example:

lambda:
  test-lambda:
    trigger:
      - type: ecr

bash completion

source completions.d/libaws.sh

extending

drop down to the aws go sdk and implement what you need.

extend an existing mutative operation or add a new one.

make sure that mutative operations are IDEMPOTENT and can be PREVIEWED.

you will find examples in cmd/ and lib/ that can provide a good place to start.

you can reuse many existing operations like:

alternatively, lift and shift to other infrastructure automation tooling. ls and describe operations will give you all the information you need.

testing

run all integration tests aws with tox:

export LIBAWS_TEST_ACCOUNT=$ACCOUNT_NUM

tox

run one integration test aws with tox:

export LIBAWS_TEST_ACCOUNT=$ACCOUNT_NUM

tox -- bash -c 'cd examples/simple/python/api/ && python test.py'

bsv

why

it should be simple and easy to process data at the speed of sequential io.

what

a simple and efficient data format for easily manipulating chunks of rows of columns while minimizing allocations and copies.

minimal cli tools for rapidly composing performant data flow pipelines.

how

column: 0-65536 bytes.

row: 0-65536 columns.

chunk: up to 5MB containing 1 or more complete rows.

note: row data cannot exceed chunk size.

layout

chunk:

| i32:size | u8[]:row | ... |

row:

| u16:max | u16:size | ... | u8[]:column | ... |

note: column bytes are always followed by a single null byte.

note: max is the maximum zero based index into the row.

install

>> curl https://raw.githubusercontent.com/nathants/bsv/master/scripts/install_archlinux.sh | bash

>> git clone https://github.com/nathants/bsv
>> cd bsv
>> make -j
>> sudo mv -fv bin/* /usr/local/bin

note: for best pipeline performance increase maximum pipe size

>> sudo sysctl fs.pipe-max-size=5242880

test

>> tox

>> docker build -t bsv:debian -f Dockerfile.debian .

>> docker run -v $(pwd):/code --rm -it bsv:debian bash -c 'cd /code && py.test -vvx --tb native -n auto test/'

>> docker build -t bsv:alpine -f Dockerfile.alpine .

>> docker run -v $(pwd):/code --rm -it bsv:alpine bash -c 'cd /code && py.test -vvx --tb native -n auto test/'

increase the number of generated tests cases with environment variable: TEST_FACTOR=5

example

add bsumall.c to bsv/src/:

#include "util.h"
#include "load.h"
#include "dump.h"

#define DESCRIPTION "sum columns of u16 as i64\n\n"
#define USAGE "... | bsumall \n\n"
#define EXAMPLE ">> echo '\n1,2\n3,4\n' | bsv | bschema a:u16,a:u16 | bsumall i64 | bschema i64:a,i64:a | csv\n4,6\n"

int main(int argc, char **argv) {

    // setup state
    SETUP();
    readbuf_t rbuf = rbuf_init((FILE*[]){stdin}, 1, false);
    writebuf_t wbuf = wbuf_init((FILE*[]){stdout}, 1, false);
    i64 sums[MAX_COLUMNS] = {0};
    row_t row;

    // process input row by row
    while (1) {
        load_next(&rbuf, &row, 0);
        if (row.stop)
            break;
        for (i32 i = 0; i <= row.max; i++) {
            ASSERT(sizeof(u16) == row.sizes[i], "fatal: bad data\n");
            sums[i] += *(u16*)row.columns[i];
        }
    }

    // generate output row
    row.max = -1;
    for (i32 i = 0; i < MAX_COLUMNS; i++) {
        if (!sums[i])
            break;
        row.sizes[i] = sizeof(i64);
        row.columns[i] = &sums[i];
        row.max++;
    }

    // dump output
    if (row.max >= 0)
        dump(&wbuf, &row, 0);
    dump_flush(&wbuf, 0);
}

build and run:

>> ./scripts/makefile.sh

>> make bsumall

>> bsumall -h
sum columns of u16 as i64

usage: ... | bsumall

>> echo '
1,2
3,4
' | bsv | bschema a:u16,a:u16 | bsumall i64 | bschema i64:a,i64:a | csv
4,6

non goals

support of hardware other than little endian.

types and schemas as a part of the data format.

testing methodology

quickcheck style testing with python implementations to verify correct behavior for arbitrary inputs and varying buffer sizes.

experiments

performance experiments and alternate implementations.

s4 - a storage cluster that is cheap and fast, with data local compute and efficient shuffle.

optimizing a bsv data processing pipeline

performant batch processing with bsv, s4, and presto

discovering a baseline for data processing performance

refactoring common distributed data patterns into s4

scaling python data processing horizontally

scaling python data processing vertically

more examples

structured analysis of nyc taxi data with bsv and hive

tools

name	description
bcat	cat some bsv files to csv
bcombine	prepend a new column by combining values from existing columns
bcounteach	count as i64 each contiguous identical row by the first column
bcounteach-hash	count as i64 by hash of the first column
bcountrows	count rows as i64
bcut	select some columns
bdedupe	dedupe identical contiguous rows by the first column, keeping the first
bdedupe-hash	dedupe rows by hash of the first column, keeping the first
bdropuntil	for sorted input, drop until the first column is gte to VALUE
bhead	keep the first n rows
blz4	compress bsv data
blz4d	decompress bsv data
bmerge	merge sorted files from stdin
bpartition	split into multiple files by consistent hash of the first column value
bquantile-merge	merge ddsketches and output quantile value pairs as f64
bquantile-sketch	collapse the first column into a single row ddsketch
bschema	validate and converts row data with a schema of columns
bsort	timsort rows by the first column
bsplit	split a stream into multiple files
bsum	sum the first column
bsumeach	sum the second column of each contiguous identical row by the first column
bsumeach-hash	sum as i64 the second column by hash of the first column
bsv	convert csv to bsv
btake	take while the first column is VALUE
btakeuntil	for sorted input, take until the first column is gte to VALUE
btopn	accumulate the top n rows in a heap by first column value
bunzip	split a multi column input into single column outputs
bzip	combine single column inputs into a multi column output
csv	convert bsv to csv
xxh3	xxh3_64 hash stdin

bcat

cat some bsv files to csv

usage: bcat [-l|--lz4] [-p|--prefix] [-h N|--head N] FILE1 ... FILEN

>> for char in a a b b c c; do
     echo $char | bsv >> /tmp/$char
   done

>> bcat --head 1 --prefix /tmp/{a,b,c}
/tmp/a:a
/tmp/b:b
/tmp/c:c

bcombine

prepend a new column by combining values from existing columns

usage: ... | bcombine COL1,...,COLN

>> echo a,b,c | bsv | bcombine 3,2 | csv
b:a,a,b,c

bcounteach

count as i64 each contiguous identical row by the first column

usage: ... | bcounteach

echo '
a
a
b
b
b
a
' | bsv | bcounteach | bschema *,i64:a | csv
a,2
b,3
a,1

bcounteach-hash

count as i64 by hash of the first column

usage: ... | bcounteach-hash

echo '
a
a
b
b
b
a
' | bsv | bcounteach-hash | bschema *,i64:a | bsort | csv
a,3
b,3

bcountrows

count rows as i64

usage: ... | bcountrows

>> echo '
1
2
3
4
' | bsv | bcountrows | csv
4

bcut

select some columns

usage: ... | bcut COL1,...,COLN

>> echo a,b,c | bsv | bcut 3,3,3,2,2,1 | csv
c,c,c,b,b,a

bdedupe

dedupe identical contiguous rows by the first column, keeping the first

usage: ... | bdedupe

>> echo '
a
a
b
b
a
a
' | bsv | bdedupe | csv
a
b
a

bdedupe-hash

dedupe rows by hash of the first column, keeping the first

usage: ... | bdedupe-hash

>> echo '
a
a
b
b
a
a
' | bsv | bdedupe-hash | csv
a
b

bdropuntil

for sorted input, drop until the first column is gte to VALUE

usage: ... | bdropuntil VALUE [TYPE]

>> echo '
a
b
c
d
' | bsv | bdropuntil c | csv
c
d

bhead

keep the first n rows

usage: ... | bhead N

>> echo '
a
b
c
' | bsv | btail 2 | csv
a
b

blz4

compress bsv data

usage: ... | blz4

>> echo a,b,c | bsv | blz4 | blz4d | csv
a,b,c

blz4d

decompress bsv data

usage: ... | blz4d

>> echo a,b,c | bsv | blz4 | blz4d | csv
a,b,c

bmerge

merge sorted files from stdin

usage: echo FILE1 ... FILEN | bmerge [TYPE] [-r|--reversed] [-l|--lz4]

>> echo -e 'a
c
e
' | bsv > a.bsv
>> echo -e 'b
d
f
' | bsv > b.bsv
>> echo a.bsv b.bsv | bmerge
a
b
c
d
e
f

bpartition

split into multiple files by consistent hash of the first column value

usage: ... | bpartition NUM_BUCKETS [PREFIX] [-l|--lz4]

>> echo '
a
b
c
' | bsv | bpartition 10 prefix
prefix03
prefix06

bquantile-merge

merge ddsketches and output quantile value pairs as f64

usage: ... | bquantile-merge QUANTILES

>> seq 1 100 | bsv | bschema a:i64 | bquantile-sketch i64 | bquantile-merge .2,.5,.7 | bschema f64:a,f64:a | csv
0.2,19.88667024086646
0.5,49.90296094906742
0.7,70.11183939140405

bquantile-sketch

collapse the first column into a single row ddsketch

usage: ... | bquantile-sketch TYPE [-a|--alpha] [-b|--max-bins] [-m|--min-value]

>> seq 1 100 | bsv | bschema a:i64 | bquantile-sketch i64 | bquantile-merge .2,.5,.7 | bschema f64:a,f64:a | csv
0.2,19.88667024086646
0.5,49.90296094906742
0.7,70.11183939140405

bschema

validate and converts row data with a schema of columns

usage: ... | bschema SCHEMA [--filter]

  --filter remove bad rows instead of erroring

  example schemas:
    *,*,*             = 3 columns of any size
    8,*               = a column with 8 bytes followed by a column of any size
    8,*,...           = same as above, but ignore any trailing columns
    a:u16,a:i32,a:f64 = convert ascii to numerics
    u16:a,i32:a,f64:a = convert numerics to ascii
    4*,*4             = keep the first 4 bytes of column 1 and the last 4 of column 2

>> echo aa,bbb,cccc | bsv | bschema 2,3,4 | csv
aa,bbb,cccc

bsort

timsort rows by the first column

usage: ... | bsort [-r|--reversed] [TYPE]

>> echo '
3
2
1
' | bsv | bschema a:i64 | bsort i64 | bschema i64:a | csv
1
2
3

bsplit

split a stream into multiple files

usage: ... | bsplit PREFIX [chunks_per_file=1]

>> echo -n a,b,c | bsv | bsplit prefix
prefix_0000000000

bsum

sum the first column

usage: ... | bsum TYPE

>> echo '
1
2
3
4
' | bsv | bschema a:i64 | bsum i64 | bschema i64:a | csv
10

bsumeach

sum the second column of each contiguous identical row by the first column

usage: ... | bsumeach TYPE

echo '
a,1
a,2
b,3
b,4
b,5
a,6
' | bsv | bschema *,a:i64 | bsumeach i64 | bschema *,i64:a | csv
a,3
b,12
a,6

bsumeach-hash

sum as i64 the second column by hash of the first column

usage: ... | bsumeach-hash i64

echo '
a,1
a,2
b,3
b,4
b,5
a,6
' | bsv | bschema *,a:i64 | bsumeach-hash i64 | bschema *,i64:a | csv
a,3
b,12
a,6

bsv

convert csv to bsv

usage: ... | bsv

>> echo a,b,c | bsv | bcut 3,2,1 | csv
c,b,a

btake

take while the first column is VALUE

usage: ... | btake VALUE

>> echo '
a
b
c
d
' | bsv | bdropntil c | btake c | csv
c

btakeuntil

for sorted input, take until the first column is gte to VALUE

usage: ... | btakeuntil VALUE [TYPE]

>> echo '
a
b
c
d
' | bsv | btakeuntil c | csv
a
b

btopn

accumulate the top n rows in a heap by first column value

usage: ... | btopn N [TYPE] [-r|--reversed]

>> echo '
1
3
2
' | bsv | bschema a:i64 | btopn 2 i64 | bschema i64:a | csv
3
2

bunzip

split a multi column input into single column outputs

usage: ... | bunzip PREFIX [-l|--lz4]

>> echo '
a,b,c
1,2,3
' | bsv | bunzip col && echo col_1 col_3 | bzip | csv
a,c
1,3

bzip

combine single column inputs into a multi column output

usage: ls column_* | bzip [COL1,...COLN] [-l|--lz4]

>> echo '
a,b,c
1,2,3
' | bsv | bunzip column && ls column_* | bzip 1,3 | csv
a,c
1,3

csv

convert bsv to csv

usage: ... | csv

>> echo a,b,c | bsv | csv
a,b,c

xxh3

xxh3_64 hash stdin

usage: ... | xxh3 [--stream|--int]

  --stream pass stdin through to stdout with hash on stderr

  --int output hash as int not hash

>> echo abc | xxh3
079364cbfdf9f4cb

tiny-snitch

why

it should be easy to monitor and control inbound and outbound connections.

what

an interactive firewall for inbound and outbound connections.

the rules are global, but the prompt always shows the pid/path/args of the program requesting a new rule.

based on the excellent opensnitch.

demo

a split screen monitoring setup for a second monitor using ptop, color, pys, and a oneliner to tail tiny-snitch logs into a small and colorful format.

dependencies

there are two components with separate dependencies:

tiny-snitch:
tiny-snitch-prompt
- python3
- pyqt5

install

setup nftables with sudo nft -f nftables.conf

build with: make

put tiny-snitch/ on your $PATH.

usage

tiny-snitch should be launched with sudo -E, so the qt5 prompt can use your DISPLAY.

either run it in a background terminal: sudo -E tiny-snitch

or automatically run it with cron: * * * * * sudo -E auto-restart tiny-snitch 2>&1 | rotate-logs /tmp/tinynitch.log

auto-restart and rotate-logs are not required.

rules

permanent rules are stored in /etc/tiny-snitch.rules and /etc/tiny-snitch.adblock. edit those files and tiny-snitch will reload.

some example rules:

action address port proto

allow google.com             443 tcp
deny  *.google-analytics.com *   tcp
allow 172.17.*.*             *   tcp
allow 172.17.*.*             *   udp

temporary rules can be added by appending lines to /tmp/tiny-snitch.temp, which will be loaded and then truncated.

some example temporary rules:

action duration address port proto

1-hour   allow google.com             443 tcp
9-minute deny  *.google-analytics.com *   tcp
24-hour  allow 172.17.*.*             *   tcp
1-minute allow 172.17.*.*             *   udp

mighty-snitch

why

noticing and preventing network requests should be easy.

how

interactively filter network requests with rules and visual prompts.

what

a linux security module communicates via netlink with the userspace snitch on each sendmsg/recvmsg.

snitch decides whether to allow or deny the network request.

rules are checked. if no rule exists, a visual prompt is displayed to the user.

finally snitch responds to the kernel and the request is allowed or denied.

demo

hardware

the primary test environments are alpine, arch, and postmarketos.

the primary test devices are thinkpad x1, rog x670e-i, and oneplus 6t.

prior art

little-snitch which introduced me to this concept.

open-snitch which introduced me to nfq.

tiny-snitch which helped me understand what is possible with nfq and bpftrace.

uslm which helped me understand what is possible with lsm.

design

mighty-snitch uses lsm instead of nfq to filter network requests.

the primary advantage is that it has direct access to the pid, executable, and commandline of the process making the request.

the primary disadvantage is that it requires a custom kernel.

the visual prompt is a terminal application which responds to keyboard input. a new terminal is launched for each prompt and exits after y/n are pressed. st is used on x86_64 and foot is used on arm64, though any terminal should work.

the systems fails closed. when snitch isn't running, network requests are not possible.

dns packets received on udp 53 are read via nfq so that rules can specify domains in addition to ipv4 addresses.

constraints

the following are simplifying constraints. other configurations should be possible.

ipv6 is disabled.
io_uring is disabled.
nftables rules are replaced when snitch starts.
iptables rules should be empty.
all other lsm are disabled.
kernel commandline parameters for lsm are ignored.

rules

snitch creates a rules file: ~/.snitch.rules

when this file is edited, snitch reloads the rules.

typically rules are created by choosing the forever duration in the visual prompt, but can also be directly added to the rules file.

address can be a wildcard up to three subdomains.

commandline can be a wildcard.

here are the rules for firefox to deny all the unprompted connections it makes:

send  deny  /usr/lib/firefox/firefox  content-signature-2.cdn.mozilla.net    443  tcp  /usr/lib/firefox/firefox
send  deny  /usr/lib/firefox/firefox  content-signature-2.cdn.mozilla.net    80   tcp  /usr/lib/firefox/firefox
send  deny  /usr/lib/firefox/firefox  contile.services.mozilla.com           443  tcp  /usr/lib/firefox/firefox
send  deny  /usr/lib/firefox/firefox  firefox.settings.services.mozilla.com  443  tcp  /usr/lib/firefox/firefox
send  deny  /usr/lib/firefox/firefox  firefox.settings.services.mozilla.com  443  udp  /usr/lib/firefox/firefox
send  deny  /usr/lib/firefox/firefox  getpocket.cdn.mozilla.net              443  tcp  /usr/lib/firefox/firefox
send  deny  /usr/lib/firefox/firefox  location.services.mozilla.com          443  tcp  /usr/lib/firefox/firefox
send  deny  /usr/lib/firefox/firefox  mozilla.cloudflare-dns.com             443  tcp  /usr/lib/firefox/firefox
send  deny  /usr/lib/firefox/firefox  normandy.cdn.mozilla.net               443  tcp  /usr/lib/firefox/firefox
send  deny  /usr/lib/firefox/firefox  push.services.mozilla.com              443  tcp  /usr/lib/firefox/firefox
send  deny  /usr/lib/firefox/firefox  shavar.services.mozilla.com            443  tcp  /usr/lib/firefox/firefox

install alpine

copy latest wget urls from: https://github.com/nathants/mighty-snitch/releases

cd /tmp
wget linux-edge-*.apk
wget linux-edge-dev-*.apk
wget me@nathants.com-*.rsa.pub
sudo mv *.pub /etc/apk/keys/
sudo apk add *.apk
sudo reboot

cd ~
git clone https://github.com/nathants/mighty-snitch

cd ~/mighty-snitch/snitch-prompt
sudo pip install .

cd ~/mighty-snitch/snitch
bash snitch.sh

install postmarketos

copy latest wget urls from: https://github.com/nathants/mighty-snitch/releases

cd /tmp
wget linux-postmarketos-qcom-sdm845-*.apk
wget pmos@local-*.rsa.pub
sudo mv *.pub /etc/apk/keys/
sudo apk add *.apk
sudo reboot

cd ~
git clone https://github.com/nathants/mighty-snitch

cd ~/mighty-snitch/snitch-prompt
sudo pip install .

cd ~/mighty-snitch/snitch
bash snitch.sh

install arch

copy latest wget urls from: https://github.com/nathants/mighty-snitch/releases

cd /tmp
wget linux-*.zst
wget linux-docs-*.zst
wget linux-headers-*.zst
sudo pacman -U *.zst
sudo reboot

cd ~
git clone https://github.com/nathants/mighty-snitch

cd ~/mighty-snitch/snitch-prompt
sudo pip install .

cd ~/mighty-snitch/snitch
bash snitch.sh

build alpine on aws and install

sudo apk add go
go install github.com/nathants/libaws@latest
export PATH=$PATH:$(go env GOPATH)/bin

export MIGHTY_SNITCH_S3_BUCKET=$NAME
export MIGHTY_SNITCH_AWS_ACCOUNT=$ACCOUNT_NUMBER
export MIGHTY_SNITCH_PUBKEY_CONTENT=$(cat ~/.ssh/id_ed25519.pub)

cd ~
git clone https://github.com/nathants/mighty-snitch

cd ~/mighty-snitch/kernel/alpine
bash build.sh
sudo mv /tmp/abuild/*.pub /etc/apk/keys/
sudo apk add /tmp/packages/*/*/*.apk
sudo reboot

cd ~/mighty-snitch/snitch-prompt
sudo pip install .

cd ~/mighty-snitch/snitch
bash snitch.sh

build alpine on aws and install

sudo apk add go
go install github.com/nathants/libaws@latest
export PATH=$PATH:$(go env GOPATH)/bin

export MIGHTY_SNITCH_S3_BUCKET=$NAME
export MIGHTY_SNITCH_AWS_ACCOUNT=$ACCOUNT_NUMBER
export MIGHTY_SNITCH_PUBKEY_CONTENT=$(cat ~/.ssh/id_ed25519.pub)

cd ~
git clone https://github.com/nathants/mighty-snitch

cd ~/mighty-snitch/kernel/alpine-sdm845
bash build.sh
sudo mv /tmp/*.pub /etc/apk/keys/
sudo apk add /tmp/*.apk
sudo reboot

cd ~/mighty-snitch/snitch-prompt
sudo pip install .

cd ~/mighty-snitch/snitch
bash snitch.sh

build arch and install

cd ~
git clone https://github.com/nathants/mighty-snitch

cd ~/mighty-snitch/kernel/arch
makepkg -sCf
sudo pacman -U *.zst

cd ~/mighty-snitch/snitch-prompt
sudo pip install .

cd ~/mighty-snitch/snitch
bash snitch.sh