Monday, December 11, 2017

How to Install Confluent Kafka Cluster by using Ansible

The rise of micro-services brings another level of software architecture, which is a event driven architecture. One of the tools out there to support this mechanism is Apache Kafka. Today’s article will speak about how to install Kafka in the server by using Ansible Playbook.

Confluent Kafka Playbook
This playbook will install Confluent Kafka into 3 cluster nodes. Each node will contain one Kafka broker and one Zookeeper instance. They will in sync one another.

Monday, August 28, 2017

Security Patching : Ansible


The following playbook was run against 100+ servers and patched the bash vulnerability in less than 10 minutes. The below example updates both Debian and Red Hat Linux variants. It will first run on half of all the hosts that are defined in an inventory file.
- hosts: all
  gather_facts: yes
  remote_user: craun
  serial: "50%"
  sudo: yes
    - name: Update Shellshock (Debian)
      apt: name=bash
      when: ansible_os_family == "Debian"

    - name: Update Shellshock (RedHat)
      yum: name=bash
      when: ansible_os_family == "RedHat"

The below example updates both Debian and RedHat linux variants. It will patch and reboot 25% of the servers at a time until all of the hosts defined in the inventory file are updated.
- hosts: all
  gather_facts: yes
  remote_user: craun
  serial: "25%"
  sudo: yes
    - name: Update OpenSSL and OpenSSH (Debian)
      apt: name={{ item }}
        - openssl
        - openssh-client
        - openssh-server
      when: ansible_os_family == "Debian"

    - name: Update OpenSSL and OpenSSH (RedHat)
      yum: name={{ item }}
        - openssl
        - openssh-client
        - openssh-server
      when: ansible_os_family == "RedHat"
    - name: Reboot servers
      command: reboot

Setting up Tomcat : Ansible Playbook

Use the below playbook for reference for installing tomcat.
Create mentioned handlers separately.
- name: Install Java 1.7
yum: name=java-1.7.0-openjdk state=present
- name: add group "tomcat"
group: name=tomcat

Friday, July 28, 2017

Difference between Shell & Command module : Ansible

  • The command module takes the command name followed by a list of space-delimited arguments.
  • The given command will be executed on all selected nodes. It will not be processed through the shell, so variables like $HOME and operations like “<”“>”“|””;” and “&” will not work (use the shell module if you need these features).
Command args (common for shell & command)

chdir - Change into this directory before running the command.
creates - A filename or (since 2.0) glob pattern, when it already exists, this step will not be run.
removes - A filename or (since 2.0) glob pattern, when it does not exist, this step will not be run.

args: only for shell module
executable - change the shell used to execute the command. Should be an absolute path to the executable.

Tuesday, July 18, 2017

Adding pre-tasks and post-tasks to playbooks

- hosts: www
  remote_user: vagrant
  sudo: yes
     - shell: echo 'I":" Beginning to configure web server..'
     - nginx
     - shell: echo 'I":" Done configuring nginx web server...'

Wednesday, June 28, 2017

System information: Adding colors to Shell script

# Sample script written for Part 4 of the RHCE series
# This script will return the following set of system information:
# -Hostname information:
echo -e "\e[31;43m***** HOSTNAME INFORMATION *****\e[0m"
echo ""
# -File system disk space usage:
echo -e "\e[31;43m***** FILE SYSTEM DISK SPACE USAGE *****\e[0m"
df -h
echo ""
# -Free and used memory in the system:
echo -e "\e[31;43m ***** FREE AND USED MEMORY *****\e[0m"
echo ""
# -System uptime and load:
echo -e "\e[31;43m***** SYSTEM UPTIME AND LOAD *****\e[0m"
echo ""
# -Logged-in users:
echo -e "\e[31;43m***** CURRENTLY LOGGED-IN USERS *****\e[0m"
echo ""
# -Top 5 processes as far as memory usage is concerned
echo -e "\e[31;43m***** TOP 5 MEMORY-CONSUMING PROCESSES *****\e[0m"
ps -eo %mem,%cpu,comm --sort=-%mem | head -n 6
echo ""
echo -e "\e[1;32mDone.\e[0m"
Run this script in bash shell and you will get an output similar below.

Note that the headers of each section are shown in color for better visualization:
That functionality is provided by this command:
echo -e "\e[COLOR1;COLOR2m<YOUR TEXT HERE>\e[0m"
Where COLOR1 and COLOR2 are the foreground and background colors, respectively (more info and options are explained in this entry from the Arch Linux Wiki) and <YOUR TEXT HERE> is the string that you want to show in color.

Wednesday, June 7, 2017

Enable direct ssh access to EC2 instance without .pem key

Enable password authentication by editing /etc/ssh/sshd_config: change PasswordAuthentication no to PasswordAuthentication yes

Restart ssh:

sudo /etc/init.d/ssh restart

systemctl  restart sshd (for RHEL7)

Sunday, May 28, 2017

Ansible : Privilege escalation (become)

Ansible uses the becomebecome_user, and become_method directives to achieve privilege escalation. You can apply them to an entire play or playbook, set them in an included playbook, or set them for a particular task.
- name: checkout repo
  git: repo= version=master dest={{ dst }}
  become: yes
  become_user: some_user
You can use become_with to specify how the privilege escalation is achieved, the default being sudo.
More Examples:
- name: Ensure the httpd service is running
    name: httpd
    state: started
  become: yes
To run a command as the apache user:
- name: Run a command as the apache user
  command: somecommand
  become: yes
  become_user: apache
To do something as the nobody user when the shell is nologin:
- name: Run a command as nobody
  command: somecommand
  become: yes
  become_method: su
  become_user: nobody
  become_flags: '-s /bin/sh'

Saturday, May 27, 2017

Setting up Ansible in AWS Linux RHEL

Enable PING
Edit Security Groups>Add Rule>>ICMP

Connect to AWS using putty
Follow the steps mentioned here/

To convert the .pem to .ppk file use the latest PuttyGen otherwise you will encounter the issues like 'Cant load the private key'

Modify the shell prompt

vi /etc/bashrc
Edit the below line as you like
[ "$PS1" = "\\s-\\v\\\$ " ] && PS1="[\u@MyServer>\w]\\$ "

Enable Root Login in AWS Linux machine

Uncomment PermitRootLogin in /etc/ssh/sshd_config

Setup SSH Passwordless authentication

copy the generated .pem file to he server using winscp (if you are using windows)
1. As root run ssh-keygen (generate the key to
2. As ec2-user copy this file to Target server:/home/ec2-user
scp -i "/home/ec2-user/clients.pem" /root/.ssh/ ec2-user@x.x.x.x:/home/ec2-user/
2. Login to target Server
    cat /home/ec2-user/ >> /root/.ssh/authorized_keys

Now you have setup the SSH passwordless login for Ansible.

Wednesday, March 29, 2017

Daily Task Automation: High load notification

The below code will trigger an email if the load is higher than 5
load=`w | head -n1 | awk '{print$10}' | cut -d"," -f1`
if [ $load -gt 5 ]; then
echo "High Load Found." | mail -s "High Load Found."

Tuesday, March 28, 2017

Ansible - wait_for module

wait_for - Waits for a condition before continuing


- name: sleep for 300 seconds and continue with play
wait_for: timeout=300
delegate_to: localhost

- name: Wait 300 seconds for port 8000 to become open on the host, don't start checking for 10 seconds
port: 8000
delay: 10

- name: Wait 300 seconds for port 8000 of any IP to close active connections, don't start checking for 10 seconds
port: 8000
delay: 10
state: drained

- name: Wait 300 seconds for port 8000 of any IP to close active connections, ignoring connections for specified hosts
port: 8000
state: drained

- name: Wait until the file /tmp/foo is present before continuing
path: /tmp/foo

- name: Wait until the string "completed" is in the file /tmp/foo before continuing
path: /tmp/foo
search_regex: completed

- name: Wait until the lock file is removed
path: /var/lock/file.lock
state: absent

- name: Wait until the process is finished and pid was destroyed
path: /proc/3466/status
state: absent

- name: Output customized message when failed
path: /tmp/foo
state: present
msg: Timeout to find file /tmp/foo

Monday, March 6, 2017

Monitoring script & html output

Here we are monitoring the file system usage by 'df -h' command and output of this command can be accessible via web browser.
# Sample script to demonstrate the creation of an HTML report using shell scripting
# Web directory
df -h > /var/www/html/test.txt
# A little CSS and table layout to make the report look a little nicer
echo "<HTML>
.titulo{font-size: 1em; color: white; background:#0863CE; padding: 0.1em 0.2em;}
table, td, th
border:1px solid black;
<meta http-equiv='Content-Type' content='text/html; charset=UTF-8' />
<BODY>" > $WEB_DIR/report.html
# View hostname and insert it at the top of the html body
echo "Filesystem usage for host <strong>$HOST</strong><br>
Last updated: <strong>$(date)</strong><br><br>
<table border='1'>
<tr><th class='titulo'>Filesystem</td>
<th class='titulo'>Size</td>
<th class='titulo'>Use %</td>
</tr>" >> $WEB_DIR/report.html
# Read the output of df -h line by line
while read line; do
echo "<tr><td align='center'>" >> $WEB_DIR/report.html
echo $line | awk '{print $1}' >> $WEB_DIR/report.html
echo "</td><td align='center'>" >> $WEB_DIR/report.html
echo $line | awk '{print $2}' >> $WEB_DIR/report.html
echo "</td><td align='center'>" >> $WEB_DIR/report.html
echo $line | awk '{print $5}' >> $WEB_DIR/report.html
echo "</td></tr>" >> $WEB_DIR/report.html
done < /var/www/html/test.txt
echo "</table></BODY></HTML>" >> $WEB_DIR/report.html
Open the report.html file in a web browser. You will see output similar to below.

You can add to that report as much information as you want. To run the script every day at 1:30 pm, add the following crontab entry:
30 13 * * * /root/scripts/

Tuesday, February 14, 2017

Create EC2 instances using Ansible Playbook

Log into your AWS account to get your “AWS_ACCESS_KEY_ID” and “AWS_SECRET_ACCESS_KEY”. Go to “Identity and Access Management”. Create a new user or select an exiting one. Go to “Security Credentials” and click “Create Access Key”. Here’s an example of what you’ll end up with:
Keep those safe – download when asked. Use the above values to create environment variables. Copy and paste the following (with your values replacing mine) into your shell:
Create the “~/hosts” file with the following contents:

Now we build our YML file for Ansible to run through. Here’s a sample that will create a basic EC2 with a public IP address and your public SSH key. Put the following into the file “~/ec2-basic.yml”
  - name: Provision an EC2 Instance
    hosts: local
    connection: local
    gather_facts: False
    tags: provisioning
    # Necessary Variables for creating/provisioning the EC2 Instance
      instance_type: t2.micro
      security_group: ansible-webserver # Change the security group name here
      image: ami-719fb712 # This is an AMI i created myself
      keypair: agix-key # This is one of my keys that i already have in AWS
      region: ap-southeast-2 # Change the Region
      count: 1

    # Task that will be used to Launch/Create an EC2 Instance

      - name: Create a security group
          module: ec2_group
          name: "{{ security_group }}"
          description: Security Group for webserver Servers
          region: "{{ region }}"
            - proto: tcp
              from_port: 22
              to_port: 22
            - proto: tcp
              from_port: 80
              to_port: 80
            - proto: tcp
              from_port: 443
              to_port: 443
            - proto: all
        register: basic_firewall

      - name: Launch the new EC2 Instance
        local_action: ec2 
                      group={{ security_group }} 
                      instance_type={{ instance_type}} 
                      image={{ image }} 
                      region={{ region }} 
                      keypair={{ keypair }}
        register: ec2

      - name: Add the newly created EC2 instance(s) to the local host group (located inside the directory)
        local_action: lineinfile 
                      regexp={{ item.public_ip }} 
                      insertafter="[webserver]" line={{ item.public_ip }}
        with_items: ec2.instances

      - name: Wait for SSH to come up
        local_action: wait_for 
                      host={{ item.public_ip }} 
        with_items: ec2.instances

      - name: Add tag to Instance(s)
        local_action: ec2_tag resource={{ }} region={{ region }} state=present
        with_items: ec2.instances
            Name: webserver
Being the provisioning (spin it up):
ansible-playbook -i ./hosts ec2-basic.yml
And finally log into your new ec2 instance:
Keypair->EC2>NETWORK & SECURITY, choose Key Pairs

Wednesday, February 1, 2017

Daily tasks automation : low memory

Email notification if memory is low

subject="Server Memory Status Alert"
##sending mail as
## sending mail to
## send carbon copy to

## get total free memory size in megabytes(MB) 
free=$(free -mt | grep Total | awk '{print $4}')
## check if free memory is less or equals to  100MB
if [[ "$free" -le 100  ]]; then
## get top processes consuming system memory and save to temporary file 
ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%mem | head >/tmp/top_proccesses_consuming_memory.txt
## send email if system memory is running low
echo -e "Warning, server memory is running low!\n\nFree memory: $free MB" | mailx -a "$file" -s "$subject" -r "$from" -c "$to" "$also_to"
exit 0

Sunday, January 29, 2017

Daily Task Automation : Read the logs and notify

Below script will continuously read the log and will trigger an email if it founds a match for the string



while true
if tail -f /path/to/file | grep $string
echo -e "String found on $HOSTNAME" | mail -s "Subject"

Wednesday, January 18, 2017

Installing GUI in AWS EC2 - RHEL7

There are three sections involved in the whole setup. Follow all the three sections explained below to successfully configure the GUI.

sudo yum -y update
Install the gnome GUI components using the following command.
sudo yum groupinstall -y "Server with GUI"
Issue the following commands to start the GUI during boot.
sudo systemctl set-default

sudo systemctl default

Now we have all the essential GUI components installed on the server. In the next section, we will install the xrdp components to enable remote desktop connections.

Setting Up XRDP

Add the xrdp repository to your instance using the following command.
Install xrdp and tiger VNC server.
sudo yum install -y xrdp tigervnc-server
Setup SELINUX security using the following commands.
chcon --type=bin_t /usr/sbin/xrdp
chcon --type=bin_t /usr/sbin/xrdp-sesman
Start and enable the xrdp service.
Enable RDP port on the instance using the following firewall commands.
sudo firewall-cmd --permanent --add-port=3389/tcp
sudo firewall-cmd --reload
Set a password for ec2-user . This password will be used to connect to the RDP session.
Set password for root as you will be prompted for cloud user password for network proxy and color. Login as root and set the password.
Now we have the xdrp components and all instance level settings in the right place. Now let’s test out the RDP connection from an RDP client. In this tutorial, i am using windows RDP client.

Connecting The Instance Using RDP

Note: Make sure you have opened RDP port in your instance security group.
1. Open RDP client and type in the public IP of your instance and click connect.
2. If you get a warning message about remote identity, just click yes.
3. Now you will get a xrdp authentication window, enter the credentials and click ok.
Note: The username is “ec2-user” and the password is the password you set for ec2-user in step 6.
4. You will be prompted to enter the password again. Provide the password and proceed to the steps to configure the initial desktop.
5. If it prompts for “cloud user password” provide the root user password you set in step 7.
6. That it, you will get a GUI session as shown below. If you face any errors do let me know in the comment session.
Source :