2 - Jenkins and AWX Deployment

1. Check out repository senofi/openidl-devops

2. Create a new folder under openidl-devops/aws-infrastructure/environments/ by copying the sample folder openidl-devops/aws-infrastructure/environments/sample-env 

Make sure you have an AWS profile set in your ~/.aws/config  and ~/.aws/credentials

[profile tf-user]
region = us-east-2
external_id = terraform

[profile tf-role]
external_id = terraform
source_profile = tf-user
role_arn = arn:aws:iam::<aws-account-number>:role/tf_automation
region = us-east-2

[tf-user]
aws_access_key_id = AKI...
aws_secret_access_key = r3AB...

Find the name of the Kubernetes cluster and update the local config with it

export AWS_PROFILE=tf-role
aws eks update-kubeconfig --name ops-k8s

1. Install Nginx Ingress controller

   kubectl create ns ingress-nginx 
   helm install -n ingress-nginx lb ingress-nginx/ingress-nginx

Use the helm chart for installing Jenkins onto the Kubernetes cluster created above.

cd <devops-repo>/jenkins 
kubectl create ns jenkins 
helm repo add jenkins https://charts.jenkins.io 
helm upgrade --install -n jenkins jenkins jenkins/jenkins --values values.yaml

Wait for Jenkins to start up. 

To view the Jenkins admin password: 

kubectl exec --namespace jenkins -it svc/jenkins -c jenkins -- /bin/cat /run/secrets/additional/chart-admin-password && echo

Set up a cloud-provisioned Jenkins node as defined in the Kubernetes plugin config in Jenkins.

Create the AWX DB by connecting to the RDS PostgreSQL instance created via Terraform.

1. Create an SSH Tunnel. Lookup the RDS DB DNS and the EC2 instance that is the AWX target public DNS and replace them in the command line template: 

   ssh -i <env-folder>/awx-target.pem -N -L 5432:ops-tools-db.<instance-id>.us-east-2.rds.amazonaws.com:5432 ubuntu@<awx-target-ec2>.us-east-2.compute.amazonaws.com -vv

2. Connect with DBeaver (or another PostgreSQL client) on localhost port 5432 and run the following SQL after replacing <pass> with an actual password (as defined under environments/<env>/org-vars.yaml ) 

   create database awx;
   create user awxuser with encrypted password '<pass>';
   grant all privileges on database awx to awxuser;

3. Configure the Kustomize script awx-custom.yaml by replacing the DB settings in awx-operator folder under openidl-devops Git repository.

Install AWX with the Kustomize command.

helm repo add awx-operator https://ansible.github.io/awx-operator/
cd awx-operator 
kustomize build . | kubectl apply -f -

Watch for the script failing and if it does run it again (timing issue due to the creation of the AWX RBAC)

We need to maintain two workspaces - one for the Fabric Kubernetes cluster and one for the openIDL applications.

To create the workspaces use the tool located in senofi/openidl-devops:

1. Go to openidl-devops/aws-infrastructure/environments/<env-folder>/terraform-cloud and run 

   terragrunt plan

   If everything looks ok, execute terragrunt apply. This should create two workspaces and a var set in Terraform Cloud.

2. Create a new KMS key (symetric, encrypt/decrypt) in the AWS console. The name is not important but use a meaningful name that will associate it with this environment. Use it to populate the property in the next step

3. Go to openidl-devops/automation/terraform-cloud and update configuration.properties Make sure that the varset name

4. Create SSH keys 

   ssh-keygen -t rsa -f app_eks_worker_nodes_ssh_key.pem ssh-keygen -t rsa -f blk_eks_worker_nodes_ssh_key.pem ssh-keygen -t rsa -f bastion_ssh_key.pem

5. Populate the variable set by executing the following command in openidl-devops/automation/terraform-cloud 

   pip install -r requirements.txt 
   python populate-variable-set.py

6. Copy the contents of the public keys and populate them in Terraform Cloud UI under Variable Sets → <the newly created varset>

1. Set Jenkins node label ‘openidl’ in Kubernetes Cloud by going to Manage Jenkins → Manage Nodes and Clouds → Configure Clouds. Make sure that under Pod Template details the labels field contains the value ‘openidl’.

   

   Also, remove the prepopulated ‘sleep’ command if it is set on the pod template:

   

2. Create the Terraform Job Template

   1. Terraform Token Secret - Login to Jenkins go to Manage Jenkins → Manage Credentials → Stores scoped to Jenkins (Jenkins) → Global Credentials (unrestricted) → Add credentials

      

   2. Choose Kind as secret text, enter secret text like Token in “secret” field and name the secret ID as unique since it will be used in pipeline code.

   3. Git Credentials - Add a new credential

3. Terraform Job

   1. Go to Jenkins → New Item. Use a name such as Terraform Job

   2. Select job type as PIPELINE and proceed.

   3. Select Definition as Pipeline Script from SCM

   4. Select SCM as Git

   5. Key in the Infrastructure code repository (openidl-gitops) URL.

   6. Select the Git credential created above

   7. Specify the relevant branch “refs/heads/<branch-name>”.

   8. Set script path to jenkins-jobs/jenkinsfile-tf