aws-exec

aws-exec

building services on lambda should be easy and fun

github.com/nathants/aws-exec

why

building services on lambda should be easy and fun.

what

a project scaffold for a backend service on aws with an infrastructure set ready-to-deploy with libaws.

the project scaffold makes it easy to:

  • authenticate callers.

  • implement fast synchronous apis that return all results immediately.

  • implement slow asynchronous apis with streaming logs, exit code, and 15 minutes max duration.

  • use the web admin interface, even from a phone.

  • use the cli admin interface, executing locally or on lambda.

  • use the api interface, calling efficiently from other backend services.

how

synchronous apis are normal http on lambda.

asynchronous apis are a http post that triggers an async lambda which invokes a command via rpc or subprocess and stores the results in s3.

  • each invocation creates 3 objects in s3:

    • log: all stdout and stderr, updated in its entirety every second.
    • exit: the exit code of the command, written once.
    • size: the size in bytes of the log after the final update, written once, written last.
  • objects are stored in either:

    • aws-exec private s3.
    • presigned s3 put urls provided by the caller.
  • to follow invocation status, the caller:

    • polls the log object with increasing range-start.
    • stops when the size object exists and range-start equals size.
    • returns the exit object.

there are three ways to invoke an asynchronous api:

add a new synchronous functionality

add to api/.

duplicate the httpExecGet or httpExecPost handler and modify it to introduce new functionality.

add a new asynchronous functionality

add to cmd/.

duplicate the listdir command and modify it to introduce new functionality.

web demo

cli demo

api demo

mobile demo

dependencies

use the included Dockerfile or install the following dependencies:

aws prerequisites

  • aws route53 has the domain or its parent from env.sh

  • aws acm has a wildcard cert for the domain or its parent from env.sh

usage

go install github.com/nathants/libaws@latest
export PATH=$PATH:$(go env GOPATH)/bin

cp env.sh.template env.sh # update values
bash bin/check.sh         # lint
bash bin/preview.sh       # preview changes to aws infra
bash bin/ensure.sh        # ensure aws infra
bash bin/dev.sh           # iterate on backend and frontend
bash bin/logs.sh          # tail the logs
bash bin/delete.sh        # delete aws infra
bash bin/cli.sh -h        # interact with the service via the cli

usage with bad upload bandwidth:

# bash bin/dev.sh         # this needs upload bandwidth
bash bin/dev_frontend.sh  # iterate on localhost frontend
bash bin/relay.sh         # iterate on backend via ec2 relay

usage with docker

cp env.sh.template env.sh # update values
docker build -t aws-exec:latest .
docker run -it --rm \
    -v $(pwd):/code \
    -e AWS_DEFAULT_REGION \
    -e AWS_ACCESS_KEY_ID \
    -e AWS_SECRET_ACCESS_KEY \
    aws-exec:latest \
    bash -c '
        cd /code
        bash bin/ensure.sh
    '

create auth

bash bin/cli.sh auth-new test-user

install and use cli

go install github.com/nathants/aws-exec@latest
export PATH=$PATH:$(go env GOPATH)/bin

export AUTH=$AUTH
export PROJECT_DOMAIN=$DOMAIN
aws-exec exec -- whoami

install and use api

go get github.com/nathants/aws-exec@latest
package cmd

import (
	"context"
	"encoding/json"
	"fmt"
	"os"

	awsexec "github.com/nathants/aws-exec/exec"
)

func main() {
	val, err := json.Marshal(map[string]interface{}{
		"path": ".",
	})
	if err != nil {
	    panic(err)
	}
	exitCode, err := awsexec.Exec(context.Background(), &awsexec.Args{
		Url:     "https://%s" + os.Getenv("PROJECT_DOMAIN"),
		Auth:    os.Getenv("AUTH"),
		RpcName: "listdir",
		RpcArgs: string(val),
		LogDataCallback: func(logs string) {
			fmt.Print(logs)
		},
	})
	if err != nil {
		panic(err)
	}
	os.Exit(exitCode)
}