User Guide for Streamlio Community Edition, GCP Marketplace

How to deploy and operate Streamlio Community Edition on Google Cloud

Install the Streamlio application

Streamlio Community Edition can be installed using either of the following approaches:

Using the Google Cloud Platform Marketplace

Get up and running with a few clicks! Install this Kubernetes app to a Google Kubernetes Engine cluster using Google Cloud Marketplace. Follow the on-screen instructions.

Using the command line

Prerequisites

Set up command-line tools

You’ll need the following tools in your development environment:

Configure gcloud as a Docker credential helper:

gcloud auth configure-docker

You can install Streamlio in an existing GKE cluster or create a new GKE cluster.

  • If you want to create a new Google GKE cluster, follow the instructions from the section Create a GKE cluster onwards.
  • If you have an existing GKE cluster, ensure that the cluster nodes have a minimum of 2 vCPU and running k8s version 1.9 and follow the instructions from section Install the application resource definition onwards.
Create a GKE cluster

Streamlio requires a minimum 3 node cluster with each node having a minimum of 2 vCPU and k8s version 1.9. Available machine types can be seen here.

Create a new cluster from the command line:

# set the name of the Kubernetes cluster
export CLUSTER=streamlio-ce-cluster

# set the zone to launch the cluster
export ZONE=us-west1-a

# set the machine type for the cluster
export MACHINE_TYPE=n1-standard-2

# create the cluster using google command line tools
gcloud container clusters create "$CLUSTER" --zone "$ZONE" ---machine-type "$MACHINE_TYPE"

Configure kubectl to connect to the new cluster:

gcloud container clusters get-credentials "$CLUSTER" --zone "$ZONE"
Install the application resource definition

An application resource is a collection of individual Kubernetes components, such as services, stateful sets, deployments, and so on, that you can manage as a group.

To set up your cluster to understand application resources, run the following command:

kubectl apply -f "https://raw.githubusercontent.com/GoogleCloudPlatform/marketplace-k8s-app-tools/master/crd/app-crd.yaml"

You need to run this command once.

The application resource is defined by the Kubernetes SIG-apps community. The source code can be found on github.com/kubernetes-sigs/application.

Prerequisites for using Role-Based Access Control

You must grant your user the ability to create roles in Kubernetes by running the following command.

kubectl create clusterrolebinding cluster-admin-binding \
  --clusterrole cluster-admin \
  --user $(gcloud config get-value account)

You need to run this command once.

Install the Application

Clone this repo
git clone https://github.com/streamlio/streamlio-gke-marketplace.git
Pull deployer image

Configure gcloud as a Docker credential helper:

gcloud auth configure-docker

Pull the deployer image to your local docker registry

docker pull gcr.io/cloud-marketplace/streamlio-public/streamlio/deployer:0.8
Run installer script

Set your application instance name, the Kubernetes namespace and size used to deploy:

# set the application instance name
export APP_INSTANCE_NAME=streamlio-ce

# set the Kubernetes namespace the application was originally installed
export NAMESPACE=<namespace>

# set the size of the Streamlio deployment
# available values are small,medium,large
export SIZE=<size>

Create the namespace:

kubectl create namespace $NAMESPACE

Run the install script:

./scripts/install --name=$APP_INSTANCE_NAME --namespace=$NAMESPACE --size=$SIZE

Watch the deployment come up with:

kubectl get pods -n $NAMESPACE --watch

Basic Usage

Download and Install Streamlio Tools

You can interact with the Apache Pulsar running in Streamlio Community Edition using Streamlio tools. Download and install the streamlio tools using the following command.

curl -fsSL http://get.streaml.io | bash

This will install Streamlio tools at $HOME/.streamlio and add $HOME/.streamlio/bin to your path.

Expose the ports for data ingestion and admininstration

In order to ingest and consume, you need to expose the ports

# set the application instance name
export APP_INSTANCE_NAME=streamlio-ce

# set the Kubernetes namespace the application was originally installed
export NAMESPACE=<namespace>

# port forward for data ingestion (6650), operations (8080) and websockets (9090)
kubectl port-forward svc/$APP_INSTANCE_NAME-proxy -n $NAMESPACE 6650 8080 9090

Data ingestion and consumption

Streamlio tools ship with Apache Pulsar binaries pulsar-admin and pulsar-perf. pulsar-admin is used to interact with Apache Pulsar while pulsar-perf is used to ingest and consume data.

For producing data, start the pulsar-perf producer using the following command:

pulsar-perf produce my-topic

For consuming data, start the pulsar-perf consumer using the following command:

pulsar-perf consume -s my-sub my-topic

Delete the Application

There are two approaches to deleting the Streamlio Community Edition

Using the Google Cloud Platform Console

  1. In the GCP Console, open Kubernetes Applications.
  2. From the list of applications, click Streamlio.
  3. On the Application Details page, click Delete.

Using the command line

Set your application instance name and the Kubernetes namespace used to deploy:

# set the application instance name
export APP_INSTANCE_NAME=streamlio-ce

# set the Kubernetes namespace the application was originally installed
export NAMESPACE=<namespace>

Delete the resources

Delete the resources using types and a label:

kubectl delete statefulset,secret,service,configmap,serviceaccount,role,rolebinding,application \
  --namespace $NAMESPACE \
  --selector app.kubernetes.io/name=$APP_INSTANCE_NAME

Delete the persistent volumes of your installation

By design, removal of stateful sets (used by bookie pods) in Kubernetes does not remove the persistent volume claims that are attached to their pods. This prevents your installations from accidentally deleting stateful data.

To remove the persistent volume claims with their attached persistent disks, run the following kubectl commands:

# remove all the persistent volumes or disks
for pv in $(kubectl get pvc --namespace $NAMESPACE \
  --selector app.kubernetes.io/name=$APP_INSTANCE_NAME \
  --output jsonpath='{.items[*].spec.volumeName}');
do
  kubectl delete pv/$pv --namespace $NAMESPACE
done

# remove all the persistent volume claims
kubectl delete persistentvolumeclaims \
  --namespace $NAMESPACE \
  --selector app.kubernetes.io/name=$APP_INSTANCE_NAME

Delete the GKE cluster

Optionally, if you don’t need the deployed application or the GKE cluster, delete the cluster using this command:

# replace with the cluster name that you used
export CLUSTER=streamlio-ce-cluster

# replace with the zone that you used
export ZONE=us-west1-a

# delete the cluster using gcloud command
gcloud container clusters delete "$CLUSTER" --zone "$ZONE"