Deploying Intel-Owl on GKE

‘ Intel Owl’ is a one-stop destination for all your threat intelligence needs. This application, itself was designed on the idea of scalability and provides docker configurations for the same.

What if, one needs to deploy it for their organisation, which actively performs threat intelligence, and need high performance application for the same?

Well, Intel Owl deployed over GKE (Google Kubernetes Engine) might be able to perform well for that!!

NOTE: This is for versions prior to v2 of Intel Owl and specific to Google Kubernetes Engine.

For latest and more generic implementation, see https://gitlab.com/mostwanted002/intel-owl-gke

Prerequisites

1. A GCP (Google Cloud Platform) account.
2. Kubernetes Engine, Service Account Creation, and Cloud SQL API, Cloud Repository accesses, and Cloud SQL Admin API enabled.
3. gcloud, kubectl, docker installed on a host.
4. Internet, some coffee and common sense 🙂

Phase 1: Preparing Infrastructure

1. Login to your GCP Console and head over Kubernetes Engine → Clusters, and create a cluster according to your/company’s needs.
2. While the cluster is being prepared and provisioned, head over to Storage→SQL and create a PostgresQL Instance.
3. And while both the above steps are being processed, clone IntelOwl and download intel-owl-gke files to your host with gcloud and kubectl installed. Extract the latter files into intel-owl-gke folder
4. After all GCP Resources are provisioned, head over to APIs & Services→Library and search for Cloud SQL Admin API, and see if it’s enabled. If not, enable the API.
5. Go to your Cloud SQL instance, and add a database user and password which will be used to access database by IntelOwl. After that, create a database, that’ll be used by IntelOwl. Keep a note of these, as they’ll be mentioned in Secrets file kustomization.yaml for deployment.
6. Head over to IAM & Admin→Service Accounts and create a service account, with Role of Cloud SQL Client. We’ll be using this service account Cloud SQL Proxy to make our web app’s connection secure to database.

Phase 2: Preparing deployment files and configuration

1. Copy all the files from intel-owl-gke folder to IntelOwl repo folder.

2. Authenticate your gcloud application, and set your project. ( Quick Guide)

3. Authenticate your docker to be able to push images to Cloud Repository by running the following command:

   gcloud auth configure-docker
   

4. Fetch Cluster credentials and configure kubectl with:

   gcloud container clusters get-credentials <CLUSTER_NAME> --zone/region <ZONE/REGION>
   

5. Change directory into IntelOwl folder, and run the following command to configure intel_owl_nginx_http for deployment:

   sed -i -e 's/server uwsgi/server backend-webapp-service.default.svc.cluster.local/g' ./configuration/intel_owl_nginx_http
   

6. To set your API Keys, make the desired changes in kustomization.yaml. Set DB_HOST to localhost. Also change the DB_NAME, DB_USER and DB_PASSWORD accordingly. (Similar to Deployment Preparation ).

Phase 3: Deployment

1. Run the following script to build, tag and push container images to Cloud Repository:

   ./build-tag-push.sh <PROJECT-ID>
   

   By default, images will be pushed to gcr.io. To change it, modify [build-tag-push.sh](http://build-tag-push.sh) accordingly. And use the same modifications in further steps.

2. Run the next command, to set image location in .yaml deployment files. Make sure to change <PLACE_YOUR_PROJECT_ID_HERE> with your Project ID.

   sed -i -e 's/<project-id>/<PLACE_PROJECT_ID_HERE>/g' *.yaml
   

3. First, we’ll create Secrets, (ENV Variables, and Cloud Proxy Credentials). Run kubectl apply -k . to create ENV Variable secrets. After Secrets has been created, run kubectl get secrets and copy the name of secret that begins with env-app-secrets- . Run the following command to place this value in deployment. Replace the env-app-secrets-xxxxxx with the value you copied.

   sed -i -e 's/<PLACE_SECRETS>/env-app-secrets-xxxxxx/g' backend-webapp.yaml
   

4. Run the script secrets-for-cloudproxy.sh to create Secrets, to be used by Cloud SQL Proxy Container, as following:

   ./secrets-for-cloudproxy.sh <SERVICE_ACCOUNT_EMAIL> <SECRETS_NAME>
   

   Replace <SECRETS_NAME> with a suitable name. Use this name in next commands to configure it in backend-webapp.yaml:

   sed -i -e 's/<YOUR-SA-SECRET>/<SECRETS_NAME>/g' backend-webapp.yaml
   sed -i -e 's/<YOUR-SA-SECRET-VOLUME>/<SECRET_NAME>-volume/g' backend-webapp.yaml
   

5. Head over to SQL Instance and copy the Instance Connection name, and use it in this command, in place of <SQL_INSTANCE> to configure Cloud SQL Instance connection details in backend-webapp.yaml

   sed -i -e 's/<INSTANCE_CONNECTION_NAME>/<SQL_INSTANCE>/g' backend-webapp.yaml
   

   Also, use the port 5432 (default for PostgresQL instances) in place of <SQL_PORT> in this command:

   sed -i -e 's/<DB_PORT>/<SQL_PORT>/g' backend-webapp.yaml
   

   Almost there now!

6. Deploy the broker, rabbitmq-daemon.yaml by running:

   kubectl apply -f ./rabbitmq-daemon.yaml
   

   Deploy the services:

   kubectl apply -f ./services.yaml
   

   Deploy the backend:

   kubectl apply -f ./backend-webapp.yaml
   

   Wait for couple of minutes, before it deploys. After the pods are running, proceed.

   Deploy the frontend:

   kubectl apply -f ./frontend-deployment.yaml
   

Voila!! To get the IP Address for reaching the app, run kubectl get svc and Public IP will be available for Load Balancer Service.

To create admin user, first get the list of pods with kubectl get pods -l app=backend-webapp. Then run kubectl exec -it <ANY_POD_NAME_FROM_PREVIOUS_COMMAND> -c uwsgi python manage.py createsuperuser , and follow the instructions to create an admin. Login and Enjoy!