Challenges of Upgrading Splunk Universal Forwarders and Ansible Automation

Introduction

In large deployments, upgrading the Splunk Universal Forwarders (UFs) is not as easy as it looks. Most people have a number of servers in play, different configurations, and it can be challenging to implement a consistent setup across all your systems. Therefore, even a small error during an upgrade can turn into a nightmare.

We do provide an app called Remote Upgrade Forwarders which attempts to do some of what this tool does. It has its own set of limitations (it requires that things already be set up on the forwarders it operates on) and there may be scenarios where it is not useable due to automated setup policies or connectivity restrictions at your site.

When working with Eclipse UF teams, there are many ways to keep plugins and the framework up toiled. Ansible is one such approach that allows you to automate and repeat the process on many machines with little manual work. In this blog we will cover some common pain points when doing UF framework updates and then look at how to do a simple upgrade with Ansible.

Challenges in Upgrading Universal Forwarders

Large scale upgrades typically hit a roadblock after the first 10% of servers are upgraded due to unforeseen issues. When upgrading Universal Forwarders in large batches, there are several roadblocks that organizations experience. We will go over a few of the common pains and ways to ease the process in a later post.

Pre-configuration dependency: You must pre-configure the Remote Upgrade app on every forwarder. This is not realistic.

Environmental restrictions: Certain organisations have limitations imposed upon them which mean automated upgrades may not be permitted or require additional approvals.

Manual intensive process: With so many forwarders to upgrade, one by one it just gets to be too time consuming. You may have multiple UF versions coexisting within your environment. Be sure to verify compatibility before upgrading any version.

Best Practices: Ensure Log Forwarding Is Not Down for Too Long During an Upgrade, especially in highly monitoring-intensive environments.

Alternative: Using Ansible for UF Upgrades

Most tools come with an upgrade feature built in, but with Ansible, you can roll your own and make it even better than the default tool. For example, you might add a task to check the current version, to stop and then start the service, to make a backup, to install the updated version, and so on.

Playbook automation takes one headache off your plate with a pre-written playbook; you only need to run it once to upgrade all your servers consistently and accurately. There is little room for human error in missing a crucial step, which is often the bugbear of manual upgrades. 

Step-by-Step Ansible Playbook for UF Upgrade

Below is a sample Ansible playbook for upgrading Splunk  Universal Forwarders to version 9.4.2:

name: Upgrade Splunk Universal Forwarders to version 9.4.2
  hosts: splunk_forwarders
  become: yes
  vars:
    splunk_version: '9.4.2'
    splunk_url: 'https://download.splunk.com/products/universalforwarder/releases/9.4.2/linux/splunkforwarder-9.4.2-e9664af3d956-linux-amd64.tgz'
    splunk_install_dir: '/opt/splunkforwarder'
    splunk_user: 'splunker'
    splunk_group: 'splunker'
  tasks:
    - name: Get Splunk UF version
      command: '{{ splunk_install_dir }}/bin/splunk version'
      become_user: '{{ splunk_user }}'
      register: splunk_version_cmd
      changed_when: false
      failed_when: false
    - name: Extract numeric version
      set_fact:
        current_version: '{{ (splunk_version_cmd.stdout | regex_search("[0-9]+\.[0-9]+\.[0-9]+")) | default("unknown") }}'
    - name: Skip upgrade if already at target version
      meta: end_host
      when: current_version == splunk_version
    - name: Stop Splunk UF service
      command: '/opt/splunkforwarder/bin/splunk stop'
      become_user: '{{ splunk_user }}'
    - name: Backup Splunk UF configuration
      command: 'tar -czf /tmp/splunk_uf_backup_{{ ansible_hostname }}.tar.gz -C {{ splunk_install_dir }} .'
      register: backup
      failed_when: backup.rc != 0
    - name: Download the Splunk UF package
      get_url:
        url: '{{ splunk_url }}'
        dest: '/tmp/splunkforwarder-9.4.2-e9664af3d956-linux-amd64.tgz'
        mode: '0644'
    - name: Extract the Splunk UF package
      unarchive:
        src: '/tmp/splunkforwarder-9.4.2-e9664af3d956-linux-amd64.tgz'
        dest: '/opt'
        remote_src: yes
        extra_opts: ['--overwrite']
    - name: Set correct ownership for Splunk UF directory
      file:
        path: '{{ splunk_install_dir }}'
        owner: '{{ splunk_user }}'
        group: '{{ splunk_group }}'
        recurse: yes
    - name: Start Splunk UF service
      command: '/opt/splunkforwarder/bin/splunk start --accept-license --answer-yes --no-prompt'
      become_user: '{{ splunk_user }}'

Inventory File (hosts.ini) for Ansible

The hosts.ini file tells Ansible which servers to run the playbook on:

[splunk_forwarders]
uf1 ansible_host=10.2x2.xx.1xx ansible_user=splunk ansible_password=pass
uf2 ansible_host=10.2x2.xx.1xx ansible_user=splunk ansible_password=pass

[splunk_forwarders:vars]
ansible_become=yes
ansible_become_method=sudo
ansible_become_password=hello

Notes:
- `ansible_host`: IP or hostname of the UF server.
- `ansible_user`: SSH username to connect.
- `ansible_password`: Only if SSH password authentication is used (SSH keys are recommended).

Conclusion

Even though there is a great app to do Remote Upgrade Forwarders for Splunk Universal Forwarders, there may be certain configuration dependencies or customer restrictions that are not met by that approach. Using automation tools such as Ansible to perform the upgrade of forwarders allows for repeatable, scalable, and error-reduced processes that also provide audit trails for what may have gone wrong during the automation process. The following blog post is the continuation of the process outlined earlier and includes a sample playbook and inventory file to give you a jump start on automating the Splunk UF upgrade processes for your organization.

For more information, explore our https://positka.com/services/splunk-professional-services/splunk-upgrade