Merge branch 'delivery-ansible-scripts' into 'master'

Delivery ansible scripts

See merge request medina/security-monitoring!6

Makefile

@@ -4,6 +4,9 @@ ENV_DIR = $(DEPLOY_DIR)/environments/$(ENVIRONMENT)
 ANSIBLE_DIR = $(DEPLOY_DIR)/ansible
 include $(ENV_DIR)/$(ENVIRONMENT).mk
 
+CUSTOM_PROVISIONING_DIR = $(DEPLOY_DIR)/custom-provision
+include $(CUSTOM_PROVISIONING_DIR)/custom-provision.mk
+
 ANSIBLE_ARGS = -i $(ENV_DIR)/inventory.txt \
 							-e ansible_dir=$(ANSIBLE_DIR) \
 							-e environment_dir=$(ENV_DIR)

README.md

@@ -4,9 +4,15 @@ This project is meant for quickly setting up a demo of evidence collection with
 
 Project is deployed using Ansible scripts on top of infrastructure provisioned with Vagrant.
 
-It creates 4 VMs: a Wazuh server (manager), two machines acting as Wazuh agents, the Evidence Collector, and Clouditor.
+It creates 5 CentOS virtual machines (if ran in `full-setup` mode): 
+- Wazuh server (manager), 
+- 2x machines acting as Wazuh agents,
+- Evidence Collector, 
+- Clouditor.
 
-In addition to Wazuh, ClamAV is also installed on agent machines (for testing purposes).
+In addition to Wazuh, ClamAV is also installed on agent machines.
+
+---
 
 ## Requirements
 
@@ -14,45 +20,54 @@ In addition to Wazuh, ClamAV is also installed on agent machines (for testing pu
  * Ansible 2.9.16
  * (optional / integrations) `npm` / `npx` in order to run the simple HTTP server for the integrations
 
+---
+
 ## Setting up the demo
 
-First, checkout Wazuh's tag `v4.1.5` into the current directory:
+1. Checkout Wazuh's tag `v4.1.5` into the current directory:
 
     ```
     $ make clone-wazuh
     ```
-or
-```
-$ git clone https://github.com/wazuh/wazuh-ansible.git
-$ git checkout tags/v4.1.5
-```
 
-Then select your `ENVIRONMENT` in `Makefile`. Set it to `full-setup` or `no-collector`.
+2. Select your `ENVIRONMENT` in `Makefile`. Set it to `full-setup` or `no-collector` (for development purposes, when `evidence-collector` runs on local machine).
+
+3. Create and provision VMs:
 
-If the case of `full-setup`, installation of additional Ansible roles is required:
     ```
-$ ansible-galaxy install --roles-path ansible/ gantsign.golang
+    $ make create provision
     ```
 
-1. Provision Wazuh server and Wazuh agents:
+---
+
+## Using demo components
+
+### Alert forwarding
+
+To test Wazuh's alert forwarding, run HTTP Simple server using `npx`: 
 
 ```
-$ make create provision
+$ PORT=8088 npx http-echo-server
 ```
 
-2. Check the running instances:
+### Clouditor
 
-Navigate browser to: `https://192.168.33.10:5601`, login with default credentials `admin:changeme`. Navigate to `wazuh` section on the left hand-side.
+Clouditor starts automatically when Clouditor VM is provisioned.
 
-You should see 2 agents registered and running with Wazuh. 
+To see Clouditor's output, `ssh` to its machine and examine the log file:  
 
-3. Run HTTP Simple server using `npx` 
+```
+$ make ssh-clouditor
+$ tail /var/log/clouditor.log
+```
+
+To manually (re)start Clouditor (normally not needed), you can use the following command on the Clouditor VM (inside `/home/vagrant/clouditor`):
 
 ```
-$ PORT=8088 npx http-echo-server
+$ make run
 ```
 
-## Using Evidence Collector
+### Evidence Collector
 
 To see Evidence Collector's output, `ssh` to its machine and open Docker logs:
 
@@ -61,29 +76,130 @@ $ make ssh-evidence-collector
 $ docker logs -ft evidence-collector
 ```
 
-## Testing with Clouditor
+### Wazuh
 
-Clouditor starts automatically when clouditor VM is provisioned.
+To check running instances (via Wazuh web interface):
 
-To see Clouditor's output, `ssh` to its machine and examine the log file:  
+1. Navigate browser to: `https://192.168.33.10:5601`.
+
+2. Login with default credentials `admin:changeme`. 
+
+3. Navigate to `Wazuh` section on the left hand-side.
+
+You should see 2 agents registered and running with Wazuh. 
+
+---
+
+## Vagrant & Ansible environment configuration
+
+Vagrant boxes (and variables later used by Ansible) are defined inside `/environments/` folder. Each environment contains 3 main files:
+
+- `inventory.txt`:
+
+    contains environment variables/configs that will be used by Ansible when provisioning.
+
+- `Makefile`:
+
+    named the same as the environment (for easier referencing in the main `Makefile` in root directory), adds additional commands that are environment specific.
+
+- `Vagrantfile`:
+
+    contains Vagrant configuration. IPs, hostnames etc. of machines have to match those defined in the corresponding `inventory.txt`.
+
+> Note: `full-setup` environment contains additional `.env` file containing environment variables required by `evidence-collector`.
+
+To deploy to some other existing machines (assuming they run same/similar Linux distro etc.), use `custom-provision` functionality.
+
+---
+
+## Provision existing machines
+
+Ansible playbooks allow for easy installation and set-up of Wazuh (both manager and agents) and Evidence collector. 
+
+As part of the Wazuh agent deploy, machines will also have ClamAV installed.
+
+Wazuh manager and Evidence collector should be installed on the same, clean machine, while Wazuh agents can be (and should be) installed onto existing machines with other software running.
+
+> Note: this functionality was developed primarily for CentOS based machines (as it uses YUM package manager).
+
+> Possible problems: CentOS 7 versions with RHEL 7.9 could have problems starting Docker containers due to `libseccomp-devel` package deprecation.
+
+1. Generate SSH key-pair on the remote server(s) as well as on your local machine (if you haven't yet done so or want to use separate credentials):
 
     ```
-$ make ssh-clouditor
-$ tail /var/log/clouditor.log
+    $ ssh-keygen -t rsa
     ```
 
-To manually start clouditor (normally not needed), you can use the following command on the clouditor VM (in `/home/vagrant/`):
+2. Copy your SSH public key to remote server's `authorized_keys` file:
+
     ```
-$ make run
+    $ ssh-copy-id root@192.168.0.13
     ```
 
+    > Note: this will copy your default SSH pub-key from `~/.ssh/id_rsa.pub`.
+
+3. Add machine info to `/custom-provision/custom-inventory.txt` file (see `/environments/.../inventory.txt` file for example).
+
+    Make sure to set correct variables:
+    
+    | Variable | Description |
+    | -------- | -------- |
+    | `public_ip` | Machine's IP address. |
+    | `ansible_sudo_pass` | Machine's root password. |
+    | `ansible_ssh_user` | Username used to SSH (and later used by Ansible). |
+    | `ansible_ssh_pass` | SSH password (corresponding to `ansible_ssh_user`). |
+    | `ansible_ssh_private_key_file` | Location of your private key (corresponding to public key set in previous step). |
+
+    Example (user: `root`, password: `admin`, @ `192.168.0.13`): 
+
+    ```
+    192.168.0.13 public_ip=192.168.0.13 ansible_sudo_pass=admin ansible_ssh_pass=admin ansible_ssh_user=root ansible_ssh_private_key_file=~/.ssh/id_rsa
+    ```
+
+4. Set `evidence-collector` environment variables in `/custom-provision/.env`. See [Evidence collector's documentation](https://git.code.tecnalia.com/medina/wp3/task_3.2/evidence-collector) for more information.
+
+    If you're installing both Evidence collector as well as Wazuh manager on the same machine (as intended), you have to set only `clouditor_host`, `elastic_host` & `wazuh_host` variables (where `elastic_host` & `wazuh_host` are the same).
+
+    > Note: empty line in `.env` file can cause `Invalid line in environment file` Docker error. This happens only on certain Docker builds - distro dependant.
+
+5. Set variables in `/ansible/globals/globals.yml`:
+
+    | Variable | Description |
+    | ---- | ---- |
+    | `elasticsearch_host_ip` | IP of the machine running Elasticsearch (same as Wazuh manager). |
+    | `wazuh_manager_ip` | IP of the machine running Wazuh manager. |
+
+6. Provision:
+
+    ```
+    $ make -B custom-provision
+    ```
+
+### Wazuh troubleshooting
+
+Depending on your machine and network configuration, Wazuh could have problem connecting agents to manager. Check Wazuh's web interface to see if agents work corrrectly. 
+
+If not, check the logs in `/var/ossec/logs/ossec.log` and consult [official troubleshooting manual](https://documentation.wazuh.com/current/user-manual/agent-enrollment/troubleshooting.html). 
+
+Two of the most common problems are [missing open ports](https://documentation.wazuh.com/current/user-manual/agent-enrollment/troubleshooting.html#testing-communication-with-wazuh-manager) and [invalid agent names](https://documentation.wazuh.com/current/user-manual/agent-enrollment/troubleshooting.html#invalid-agent-name-for-enrollment) (if agent machine's hostname matches hostname of any already existing Wazuh machine).
+
+### Minimum hardware requirements
+
+| Component | Wazuh manager + <br>Evidence collector machine  | Wazuh agent<br>machine |
+| ---- | ---- | ---- |
+| Memory | 2 GB | 1 GB |
+| CPU | 2 | 1 |
+| Storage | 10 GB | 10 GB |
+
+---
+
 ## Potential issues
  
 ### ClamAV (re)start failed/timed out
 
-ClamAV restart can time-out due to slow disk read/write speeds (if using HDD) and lack of memory. To resolve this, provide the machine with more RAM. Current implementation has it set to `1024` MB (which should suffice for the majoirty of host machine configurations). If you're using SSD, you can lower it to `512` MB.
+ClamAV restart can time-out due to slow disk read/write speeds (if using HDD) and lack of memory. To resolve this, provide the machine with more RAM. Current implementation has it set to `1024 MB` (which should suffice for the majoirty of host machine configurations). If you're using SSD, you can lower it to `512 MB`.
 
-### Vagrant issue:
+### Vagrant issue
 
 ```
 The following SSH command responded with a non-zero exit status.
@@ -94,14 +210,15 @@ Stdout from the command:
 
 Stderr from the command:
 umount: /mnt: not mounted.
-
 ```
+
 Solved:
+
 ```
 $ vagrant plugin uninstall vagrant-vbguest
 ```
  
-### Ansible failing due to ssh issues.
+### Ansible failing due to ssh issues
 
 This is important for `manager` and `agents` - VMs need to be running already.
 

ansible/docker/tasks/main.yml

 ---
-- name: add Docker CE repository
+- name: Remove container tools
+  yum: 
+    name: 
+      - buildah
+      - podman
+      - runc
+    state: absent
+
+- name: Install (updated) runc
+  yum: 
+    name: 
+      - runc
+    state: present
+
+- name: Add Docker CE repository
   yum_repository:
     name: docker-ce-stable
     file: docker
@@ -9,17 +23,25 @@
     gpgcheck: yes
     gpgkey: https://download.docker.com/linux/centos/gpg
 
-- name: create a docker group
+- name: Create a docker group
   group: name=docker
 
-- name: install Docker CE from repository
-  yum: name=docker-ce state=installed
+- name: Install Docker CE from repository
+  yum: 
+    name: docker-ce 
+    state: installed
 
-- name: add curent '{{ ansible_user }}' to docker groups
-  user: name={{ ansible_user }} groups=docker append=yes
+- name: Add curent "{{ ansible_user }}" to docker groups
+  user: 
+    name: "{{ ansible_user }}"
+    groups: docker
+    append: yes
 
-- name: enable docker service
-  service: name=docker enabled=yes state=started
+- name: Enable docker service
+  service:
+    name: docker 
+    enabled: yes 
+    state: started
 
-- name: reset connection to apply group permissions
+- name: Reset connection to apply group permissions
   meta: reset_connection
\ No newline at end of file

ansible/provision-evidence-collector.yml

@@ -15,11 +15,28 @@
       - name: Install docker-py
         pip:
           name: docker-py
+          executable: pip3
+        vars:
+          ansible_python_interpreter: /usr/bin/python3.6
+      - name: Create 'evidence-collector' directory
+        file: 
+          path: /home/security-monitoring
+          state: directory
+          owner: "{{ ansible_ssh_user }}"
+          group: "{{ ansible_ssh_user }}"
+      - name: Copy .env file
+        copy:
+          src: "{{ environment_dir }}/.env"
+          dest: /home/security-monitoring/.env
+          owner: "{{ ansible_ssh_user }}"
+          group: "{{ ansible_ssh_user }}"
       - name: Log in to Docker image registry
         docker_login:
           registry: "{{ docker_registry }}"
           username: "{{ docker_username }}"
           password: "{{ docker_token }}"
+        vars:
+          ansible_python_interpreter: /usr/bin/python3.6
       - name: Run Docker container
         docker_container:
           name: evidence-collector
@@ -27,3 +44,6 @@
           state: started
           pull: yes
           restart_policy: always
+          env_file: /home/security-monitoring/.env
+        vars:
+          ansible_python_interpreter: /usr/bin/python3.6
\ No newline at end of file

custom-provision/.env

+demo_mode=false
+wazuh_host=192.168.0.12
+wazuh_port=55000
+wazuh_username=wazuh-wui
+wazuh_password=wazuh-wui
+elastic_host=192.168.0.12
+elastic_port=9200
+elastic_username=admin
+elastic_password=changeme
+redis_host=localhost
+redis_port=6379
+redis_queue=low
+clouditor_host=192.168.0.13
+clouditor_port=9090
\ No newline at end of file

custom-provision/custom-inventory.txt

+#
+#   Ansible hosts
+#
+
+[wazuh_managers]
+192.168.0.12 public_ip=192.168.0.12 ansible_sudo_pass=admin ansible_ssh_pass=admin ansible_ssh_user=root ansible_ssh_private_key_file=~/.ssh/is_rsa
+
+[wazuh_agents]
+192.168.0.13 public_ip=192.168.0.13 ansible_sudo_pass=admin ansible_ssh_pass=admin ansible_ssh_user=root ansible_ssh_private_key_file=~/.ssh/is_rsa
+
+[evidence_collector]
+192.168.0.12 public_ip=192.168.0.12 ansible_sudo_pass=admin ansible_ssh_pass=admin ansible_ssh_user=root ansible_ssh_private_key_file=~/.ssh/is_rsa
+
+#
+#   Ansible hosts' variables
+#
+
+[wazuh_managers:vars]
+ansible_ssh_common_args='-o StrictHostKeyChecking=no'
+
+[wazuh_agents:vars]
+ansible_ssh_common_args='-o StrictHostKeyChecking=no'
+
+[evidence_collector:vars]
+ansible_ssh_common_args='-o StrictHostKeyChecking=no'
\ No newline at end of file

custom-provision/custom-provision.mk

+CUSTOM_ARGS = -i $(CUSTOM_PROVISIONING_DIR)/custom-inventory.txt \
+					-e ansible_dir=$(ANSIBLE_DIR) \
+					-e environment_dir=$(CUSTOM_PROVISIONING_DIR)
+
+custom-provision:
+	@ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook $(CUSTOM_ARGS) $(ANSIBLE_DIR)/provision.yml
\ No newline at end of file

environments/full-setup/.env

+demo_mode=false
+wazuh_host=192.168.33.10
+wazuh_port=55000
+wazuh_username=wazuh-wui
+wazuh_password=wazuh-wui
+elastic_host=192.168.33.10
+elastic_port=9200
+elastic_username=admin
+elastic_password=changeme
+redis_host=localhost
+redis_port=6379
+redis_queue=low
+clouditor_host=192.168.33.14
+clouditor_port=9090
\ No newline at end of file