Dealing with failed hosts: Ansible playbooks - Part 2

Maximum Failure Percentage

By default, Ansible will continue executing actions as long as there are hosts in the batch that have not yet failed. The batch size for a play is determined by the serial parameter. If serial is not set, then batch size is all the hosts specified in the hosts: field. In some situations, such as with the rolling updates described above, it may be desirable to abort the play when a certain threshold of failures have been reached. To achieve this, you can set a maximum failure percentage on a play as follows:

- hosts: webservers
  max_fail_percentage: 30
  serial: 10

In the above example, if more than 3 of the 10 servers in the group were to fail, the rest of the play would be aborted.

Dealing with failed hosts: Ansible playbooks - Part 1

Summary:

When using the serial keyword a single failed host aborts the entire playbook with the error "FATAL: all hosts have already failed -- aborting"

Removing the serial keyword and all hosts are evaluated and a single failure does not abort the playbook run.

Steps To Reproduce:

Create an inventory with a "test" group with three hosts, A, B, and C. Save the following playbook.

# test_fail.yml

---
- hosts: host1:host2:host3
  gather_facts: no
  tasks:
    - ping:
    - fail:
      when: inventory_hostname == 'host3'

- hosts: host4:host5
  gather_facts: no
  tasks:
    - ping:

- hosts: host6
  gather_facts: no
  tasks:
    - ping:

- hosts: host7:host8
  gather_facts: no
  tasks:
    - ping:

How to Install Confluent Kafka Cluster by using Ansible

Overview
The rise of micro-services brings another level of software architecture, which is a event driven architecture. One of the tools out there to support this mechanism is Apache Kafka. Today’s article will speak about how to install Kafka in the server by using Ansible Playbook.

Confluent Kafka Playbook
This playbook will install Confluent Kafka into 3 cluster nodes. Each node will contain one Kafka broker and one Zookeeper instance. They will in sync one another.

REFER:

Testing and Deploying Spark Jobs : Ansible example playbook

Security Patching : Ansible

EXAMPLE: 1

The following playbook was run against 100+ servers and patched the bash vulnerability in less than 10 minutes. The below example updates both Debian and Red Hat Linux variants. It will first run on half of all the hosts that are defined in an inventory file.

- hosts: all
  gather_facts: yes
  remote_user: craun
  serial: "50%"
  sudo: yes
  tasks:
    - name: Update Shellshock (Debian)
      apt: name=bash
           state=latest
           update_cache=yes
      when: ansible_os_family == "Debian"

    - name: Update Shellshock (RedHat)
      yum: name=bash
           state=latest
           update_cache=yes
      when: ansible_os_family == "RedHat"

EXAMPLE: 2

The below example updates both Debian and RedHat linux variants. It will patch and reboot 25% of the servers at a time until all of the hosts defined in the inventory file are updated.

- hosts: all
  gather_facts: yes
  remote_user: craun
  serial: "25%"
  sudo: yes
  tasks:
    - name: Update OpenSSL and OpenSSH (Debian)
      apt: name={{ item }}
           state=latest
           update_cache=yes
      with_items:
        - openssl
        - openssh-client
        - openssh-server
      when: ansible_os_family == "Debian"

    - name: Update OpenSSL and OpenSSH (RedHat)
      yum: name={{ item }}
           state=latest
           update_cache=yes
      with_items:
        - openssl
        - openssh-client
        - openssh-server
      when: ansible_os_family == "RedHat"
  post_tasks:
    - name: Reboot servers
      command: reboot

Setting up Tomcat : Ansible Playbook

Use the below playbook for reference for installing tomcat.

Create mentioned handlers separately.

---
	- name: Install Java 1.7
	yum: name=java-1.7.0-openjdk state=present
	- name: add group "tomcat"
	group: name=tomcat

Difference between Shell & Command module : Ansible

• The command module takes the command name followed by a list of space-delimited arguments.
• The given command will be executed on all selected nodes. It will not be processed through the shell, so variables like $HOME and operations like “<”, “>”, “|”, ”;” and “&” will not work (use the shell module if you need these features).

Command args (common for shell & command)

chdir - Change into this directory before running the command.

creates - A filename or (since 2.0) glob pattern, when it already exists, this step will not be run.

removes - A filename or (since 2.0) glob pattern, when it does not exist, this step will not be run.

args: only for shell module

executable - change the shell used to execute the command. Should be an absolute path to the executable.

Adding pre-tasks and post-tasks to playbooks

---
- hosts: www
  remote_user: vagrant
  sudo: yes
  pre_tasks:
     - shell: echo 'I":" Beginning to configure web server..'
  roles:
     - nginx
  post_tasks:
     - shell: echo 'I":" Done configuring nginx web server...'

Ansible : Privilege escalation (become)

Ansible uses the become, become_user, and become_method directives to achieve privilege escalation. You can apply them to an entire play or playbook, set them in an included playbook, or set them for a particular task.

- name: checkout repo
  git: repo=https://github.com/some/repo.git version=master dest={{ dst }}
  become: yes
  become_user: some_user

You can use become_with to specify how the privilege escalation is achieved, the default being sudo.

More Examples:

- name: Ensure the httpd service is running
  service:
    name: httpd
    state: started
  become: yes

To run a command as the apache user:

- name: Run a command as the apache user
  command: somecommand
  become: yes
  become_user: apache

To do something as the nobody user when the shell is nologin:

- name: Run a command as nobody
  command: somecommand
  become: yes
  become_method: su
  become_user: nobody
  become_flags: '-s /bin/sh'

Setting up Ansible in AWS Linux RHEL

Enable PING
Edit Security Groups>Add Rule>>ICMP

Connect to AWS using putty
Follow the steps mentioned here/
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/putty.html

To convert the .pem to .ppk file use the latest PuttyGen otherwise you will encounter the issues like 'Cant load the private key'

Modify the shell prompt

vi /etc/bashrc
Edit the below line as you like
[ "$PS1" = "\\s-\\v\\\$ " ] && PS1="[\u@MyServer>\w]\\$ "

Enable Root Login in AWS Linux machine

Uncomment PermitRootLogin in /etc/ssh/sshd_config

Setup SSH Passwordless authentication

copy the generated .pem file to he server using winscp (if you are using windows)
1. As root run ssh-keygen (generate the key to id_rsa.pub)
2. As ec2-user copy this file to Target server:/home/ec2-user
scp -i "/home/ec2-user/clients.pem" /root/.ssh/id_rsa.pub ec2-user@x.x.x.x:/home/ec2-user/
2. Login to target Server
    cat /home/ec2-user/id_rsa.pub >> /root/.ssh/authorized_keys

Now you have setup the SSH passwordless login for Ansible.

Ansible - wait_for module

wait_for - Waits for a condition before continuing

EXAMPLES

- name: sleep for 300 seconds and continue with play
wait_for: timeout=300
delegate_to: localhost


- name: Wait 300 seconds for port 8000 to become open on the host, don't start checking for 10 seconds
wait_for:
port: 8000
delay: 10


- name: Wait 300 seconds for port 8000 of any IP to close active connections, don't start checking for 10 seconds
wait_for:
host: 0.0.0.0
port: 8000
delay: 10
state: drained


- name: Wait 300 seconds for port 8000 of any IP to close active connections, ignoring connections for specified hosts
wait_for:
host: 0.0.0.0
port: 8000
state: drained
exclude_hosts: 10.2.1.2,10.2.1.3


- name: Wait until the file /tmp/foo is present before continuing
wait_for:
path: /tmp/foo


- name: Wait until the string "completed" is in the file /tmp/foo before continuing
wait_for:
path: /tmp/foo
search_regex: completed


- name: Wait until the lock file is removed
wait_for:
path: /var/lock/file.lock
state: absent


- name: Wait until the process is finished and pid was destroyed
wait_for:
path: /proc/3466/status
state: absent


- name: Output customized message when failed
wait_for:
path: /tmp/foo
state: present
msg: Timeout to find file /tmp/foo

Create EC2 instances using Ansible Playbook

Log into your AWS account to get your “AWS_ACCESS_KEY_ID” and “AWS_SECRET_ACCESS_KEY”. Go to “Identity and Access Management”. Create a new user or select an exiting one. Go to “Security Credentials” and click “Create Access Key”. Here’s an example of what you’ll end up with:

Access Key ID: NUHKOIJFOJF9GFJDO
Secret Access Key: LSDJKFODSJF9SDJF8UH3U3HFKW

Keep those safe – download when asked. Use the above values to create environment variables. Copy and paste the following (with your values replacing mine) into your shell:

export AWS_ACCESS_KEY_ID="NUHKOIJFOJF9GFJDO" 
export AWS_SECRET_ACCESS_KEY="LSDJKFODSJF9SDJF8UH3U3HFKW"

Create the “~/hosts” file with the following contents:

[local]
localhost

[webserver]

Now we build our YML file for Ansible to run through. Here’s a sample that will create a basic EC2 with a public IP address and your public SSH key. Put the following into the file “~/ec2-basic.yml”

---
  - name: Provision an EC2 Instance
    hosts: local
    connection: local
    gather_facts: False
    tags: provisioning
    # Necessary Variables for creating/provisioning the EC2 Instance
    vars:
      instance_type: t2.micro
      security_group: ansible-webserver # Change the security group name here
      image: ami-719fb712 # This is an AMI i created myself
      keypair: agix-key # This is one of my keys that i already have in AWS
      region: ap-southeast-2 # Change the Region
      count: 1

    # Task that will be used to Launch/Create an EC2 Instance
    tasks:

      - name: Create a security group
        local_action: 
          module: ec2_group
          name: "{{ security_group }}"
          description: Security Group for webserver Servers
          region: "{{ region }}"
          rules:
            - proto: tcp
              from_port: 22
              to_port: 22
              cidr_ip: 0.0.0.0/0
            - proto: tcp
              from_port: 80
              to_port: 80
              cidr_ip: 0.0.0.0/0
            - proto: tcp
              from_port: 443
              to_port: 443
              cidr_ip: 0.0.0.0/0
          rules_egress:
            - proto: all
              cidr_ip: 0.0.0.0/0
        register: basic_firewall

      - name: Launch the new EC2 Instance
        local_action: ec2 
                      group={{ security_group }} 
                      instance_type={{ instance_type}} 
                      image={{ image }} 
                      wait=true 
                      region={{ region }} 
                      keypair={{ keypair }}
                      count={{count}}
        register: ec2

      - name: Add the newly created EC2 instance(s) to the local host group (located inside the directory)
        local_action: lineinfile 
                      dest="./hosts" 
                      regexp={{ item.public_ip }} 
                      insertafter="[webserver]" line={{ item.public_ip }}
        with_items: ec2.instances


      - name: Wait for SSH to come up
        local_action: wait_for 
                      host={{ item.public_ip }} 
                      port=22 
                      state=started
        with_items: ec2.instances

      - name: Add tag to Instance(s)
        local_action: ec2_tag resource={{ item.id }} region={{ region }} state=present
        with_items: ec2.instances
        args:
          tags:
            Name: webserver

Being the provisioning (spin it up):

ansible-playbook -i ./hosts ec2-basic.yml

And finally log into your new ec2 instance:

https://github.com/docker/machine/issues/287
Keypair->EC2>NETWORK & SECURITY, choose Key Pairs