How to Install Confluent Kafka Cluster by using Ansible

Overview
The rise of micro-services brings another level of software architecture, which is a event driven architecture. One of the tools out there to support this mechanism is Apache Kafka. Today’s article will speak about how to install Kafka in the server by using Ansible Playbook.

Confluent Kafka Playbook
This playbook will install Confluent Kafka into 3 cluster nodes. Each node will contain one Kafka broker and one Zookeeper instance. They will in sync one another.

REFER:

Testing and Deploying Spark Jobs : Ansible example playbook

Security Patching : Ansible

EXAMPLE: 1

The following playbook was run against 100+ servers and patched the bash vulnerability in less than 10 minutes. The below example updates both Debian and Red Hat Linux variants. It will first run on half of all the hosts that are defined in an inventory file.

- hosts: all
  gather_facts: yes
  remote_user: craun
  serial: "50%"
  sudo: yes
  tasks:
    - name: Update Shellshock (Debian)
      apt: name=bash
           state=latest
           update_cache=yes
      when: ansible_os_family == "Debian"

    - name: Update Shellshock (RedHat)
      yum: name=bash
           state=latest
           update_cache=yes
      when: ansible_os_family == "RedHat"

EXAMPLE: 2

The below example updates both Debian and RedHat linux variants. It will patch and reboot 25% of the servers at a time until all of the hosts defined in the inventory file are updated.

- hosts: all
  gather_facts: yes
  remote_user: craun
  serial: "25%"
  sudo: yes
  tasks:
    - name: Update OpenSSL and OpenSSH (Debian)
      apt: name={{ item }}
           state=latest
           update_cache=yes
      with_items:
        - openssl
        - openssh-client
        - openssh-server
      when: ansible_os_family == "Debian"

    - name: Update OpenSSL and OpenSSH (RedHat)
      yum: name={{ item }}
           state=latest
           update_cache=yes
      with_items:
        - openssl
        - openssh-client
        - openssh-server
      when: ansible_os_family == "RedHat"
  post_tasks:
    - name: Reboot servers
      command: reboot

Setting up Tomcat : Ansible Playbook

Use the below playbook for reference for installing tomcat.

Create mentioned handlers separately.

---
	- name: Install Java 1.7
	yum: name=java-1.7.0-openjdk state=present
	- name: add group "tomcat"
	group: name=tomcat

Difference between Shell & Command module : Ansible

• The command module takes the command name followed by a list of space-delimited arguments.
• The given command will be executed on all selected nodes. It will not be processed through the shell, so variables like $HOME and operations like “<”, “>”, “|”, ”;” and “&” will not work (use the shell module if you need these features).

Command args (common for shell & command)

chdir - Change into this directory before running the command.

creates - A filename or (since 2.0) glob pattern, when it already exists, this step will not be run.

removes - A filename or (since 2.0) glob pattern, when it does not exist, this step will not be run.

args: only for shell module

executable - change the shell used to execute the command. Should be an absolute path to the executable.

Adding pre-tasks and post-tasks to playbooks

---
- hosts: www
  remote_user: vagrant
  sudo: yes
  pre_tasks:
     - shell: echo 'I":" Beginning to configure web server..'
  roles:
     - nginx
  post_tasks:
     - shell: echo 'I":" Done configuring nginx web server...'

System information: Adding colors to Shell script

#!/bin/bash
# Sample script written for Part 4 of the RHCE series
# This script will return the following set of system information:
# -Hostname information:
echo -e "\e[31;43m***** HOSTNAME INFORMATION *****\e[0m"
hostnamectl
echo ""
# -File system disk space usage:
echo -e "\e[31;43m***** FILE SYSTEM DISK SPACE USAGE *****\e[0m"
df -h
echo ""
# -Free and used memory in the system:
echo -e "\e[31;43m ***** FREE AND USED MEMORY *****\e[0m"
free
echo ""
# -System uptime and load:
echo -e "\e[31;43m***** SYSTEM UPTIME AND LOAD *****\e[0m"
uptime
echo ""
# -Logged-in users:
echo -e "\e[31;43m***** CURRENTLY LOGGED-IN USERS *****\e[0m"
who
echo ""
# -Top 5 processes as far as memory usage is concerned
echo -e "\e[31;43m***** TOP 5 MEMORY-CONSUMING PROCESSES *****\e[0m"
ps -eo %mem,%cpu,comm --sort=-%mem | head -n 6
echo ""
echo -e "\e[1;32mDone.\e[0m"

Run this script in bash shell and you will get an output similar below.

Note that the headers of each section are shown in color for better visualization:

That functionality is provided by this command:

echo -e "\e[COLOR1;COLOR2m<YOUR TEXT HERE>\e[0m"

Where COLOR1 and COLOR2 are the foreground and background colors, respectively (more info and options are explained in this entry from the Arch Linux Wiki) and <YOUR TEXT HERE> is the string that you want to show in color.