Migration from json config to env variables

Migration from json config to env variables

Fixed failing CI job

.dockerignore

@@ -10,4 +10,4 @@ dump.rdb
 test/
 venv/
 env/
-
+.env
\ No newline at end of file

.env

+demo_mode=false
+
+wazuh_host=192.168.33.10
+wazuh_port=55000
+wazuh_username=wazuh-wui
+wazuh_password=wazuh-wui
+
+elastic_host=192.168.33.10
+elastic_port=9200
+elastic_username=admin
+elastic_password=changeme
+
+redis_host=localhost
+redis_port=6379
+redis_queue=low
+
+clouditor_host=192.168.33.14
+clouditor_port=9090
\ No newline at end of file

.gitignore

@@ -4,5 +4,4 @@ __pycache__/
 .idea/
 dump.rdb
 env/
-venv/
-
+venv/
\ No newline at end of file

.gitlab-ci.yml

@@ -25,7 +25,7 @@ test:
   script:
     - apk add bash
     - docker stop $SERVICE || true && docker rm $SERVICE || true
-    - docker run --name $SERVICE -d $REGISTRY/medina/$SERVICE:$VERSION
+    - docker run --env-file .env --name $SERVICE -d $REGISTRY/medina/$SERVICE:$VERSION
     - sleep 5
     - bash test/test.sh
     - docker stop $SERVICE && docker container rm $SERVICE

Dockerfile

@@ -9,6 +9,6 @@ RUN pip3 install -r requirements.txt
 
 COPY . .
 
-RUN apt-get update && apt-get install -y redis-server jq
+RUN apt-get update && apt-get install -y redis-server
 
 ENTRYPOINT ["./entrypoint.sh"]
\ No newline at end of file

Makefile

+build:
+	docker build -t evidence-collector .
+
+run:
+	docker run --env-file .env  evidence-collector
\ No newline at end of file

README.md

@@ -10,25 +10,23 @@ Wazuh evidence collector uses [Wazuh's API](https://documentation.wazuh.com/curr
 
 ### Using docker
 
-> Note: Docker image is not yet complete and might not work due to recent changes around scheduler etc.
-
 1. Set up your Wazuh development environment. Use [Security Monitoring](https://gitlab.xlab.si/medina/security-monitoring) repository to create and deploy Vagrant box with all the required components.
 
 2. Clone this repository.
 
 3. Build Docker image:
 
-```
-$ docker build -t evidence-collector .
-```
+    ```
+    $ make build
+    ```
 
 4. Run the image:
 
-```
-$ docker run evidence-collector
-```
+    ```
+    $ make run
+    ```
 
-> Note: Current simple image runs code from `test.py`. If you wish to test anything else, change this file or edit `Dockerfile`.
+    > Note: See `Environment variables` section for more information about configuration of this component and it's interaction with Wazuh, Clouditor etc.
 
 ### Local environment
 
@@ -38,41 +36,68 @@ $ docker run evidence-collector
 
 3. Install dependencies:
 
-```
-$ pip install -r requirements.txt
+    ```
+    $ pip install -r requirements.txt
+    ```
 
-$ sudo apt-get install jq
-```
+4. Set environment variables:
 
-4. a) Install Redis server locally:
+    ```
+    $ source .env
+    ```
 
-```
-$ sudo apt-get install redis-server
-```
+5. a) Install Redis server locally:
 
-> Note: To stop Redis server use `/etc/init.d/redis-server stop`.
+    ```
+    $ sudo apt-get install redis-server
+    ```
 
-4. b) Run Redis server in Docker container:
+    > Note: To stop Redis server use `/etc/init.d/redis-server stop`.
 
-```
-$ docker run --name my-redis-server -p 6379:6379 -d redis
-```
+    b) Run Redis server in Docker container:
 
-In this case also comment-out server start command in `entrypoint.sh`:
+    ```
+    $ docker run --name my-redis-server -p 6379:6379 -d redis
+    ```
 
-```
-#redis-server &
-```
+    In this case also comment-out server start command in `entrypoint.sh`:
 
-5. Run `entrypoint.sh`:
+    ```
+    #redis-server &
+    ```
 
-```
-$ ./entrypoint.sh
-```
+6. Run `entrypoint.sh`:
 
-> Note: This repository consists of multiple Python modules. When running Python code manually, use of `-m` flag might be necessary.
+    ```
+    $ ./entrypoint.sh
+    ```
 
-## Component configuration 
+    > Note: This repository consists of multiple Python modules. When running Python code manually, use of `-m` flag might be necessary.
+
+## Component configuration
+
+### Environment variables
+
+Required environment variables (if deployed localy) are located and can be set in `.env` file. 
+
+Variables used when deploying to Kubernetes can be edited in `data` section of `/kubernetes/wazuh-vat-evidence-collector-configmap.yaml` file.
+
+All of the following environment variables have to be set (or passed to container) for `evidence-collector` to work:
+
+- `demo_mode`,
+- `wazuh_host`,
+- `wazuh_port`,
+- `wazuh_username`,
+- `wazuh_password`,
+- `elastic_host`,
+- `elastic_port`,
+- `elastic_username`,
+- `elastic_password`,
+- `redis_host`,
+- `redis_port`,
+- `redis_queue`,
+- `clouditor_host`,
+- `clouditor_port`.
 
 ### Generate gRPC code from `.proto` files
 
@@ -115,37 +140,37 @@ $ curl --user admin:changeme --insecure -X GET "https://192.168.33.10:9200/wazuh
 
 1. Install (if needed) and run `redis-server`:
 
-```
-$ sudo apt-get install redis-server
+    ```
+    $ sudo apt-get install redis-server
 
-$ redis-server
-```
+    $ redis-server
+    ```
 
-> Note: By default, server listens on port `6379`. Take this into consideration when starting other components.
+    > Note: By default, server listens on port `6379`. Take this into consideration when starting other components.
 
 2. Install RQ and RQ-scheduler:
 
-```
-$ pip install rq
+    ```
+    $ pip install rq
 
-$ pip install rq-scheduler
-```
+    $ pip install rq-scheduler
+    ```
 
 3. Run both components in 2 terminals:
 
-```
-$ rqworker low
+    ```
+    $ rqworker low
 
-$ rqscheduler --host localhost --port 6379
-```
+    $ rqscheduler --host localhost --port 6379
+    ```
 
-> Note: `low` in the first command references task queue worker will use.
+    > Note: `low` in the first command references task queue worker will use.
 
 4. Run Python script containing RQ commands as usual:
 
-```
-$ python3 -m wazuh_evidence_collector.wazuh_evidence_collector
-```
+    ```
+    $ python3 -m wazuh_evidence_collector.wazuh_evidence_collector
+    ```
 
 ## Known issues
 

constants.json

-{
-    "general": {
-        "demo": false
-    },
-    "wazuh": {
-        "host": "192.168.33.10",
-        "port": 55000,
-        "username": "wazuh-wui",
-        "password": "wazuh-wui"
-    },
-    "elastic": {
-        "host": "192.168.33.10",
-        "port": 9200,
-        "username": "admin",
-        "password": "changeme"
-    },
-    "redis": {
-        "host": "localhost",
-        "port": 6379,
-        "queue": "low"
-    },
-    "clouditor": {
-        "host":"192.168.33.14",
-        "port": 9090
-    }
-}
\ No newline at end of file

entrypoint.sh

 #!/bin/bash
 
-redis_host=$(cat constants.json | jq -r '.redis.host')
-redis_port=$(cat constants.json | jq -r '.redis.port')
-redis_queue=$(cat constants.json | jq -r '.redis.queue')
-
 redis-server --port $redis_port &
 
 rqworker $redis_queue &

forward_evidence/forward_evidence.py

 from grpc_gen.assessment_pb2_grpc import AssessmentStub
 import grpc
+import os
+
+CLOUDITOR_HOST = os.environ.get("clouditor_host")
+CLOUDITOR_PORT = os.environ.get("clouditor_port")
 
 class ForwardEvidence(object):
 
-    def __init__(self, constants, logger):
-        self.channel = grpc.insecure_channel('{}:{}'.format(constants['clouditor']['host'], constants['clouditor']['port']))
+    def __init__(self, logger):
+        self.channel = grpc.insecure_channel('{}:{}'.format(CLOUDITOR_HOST, CLOUDITOR_PORT))
         self.stub = AssessmentStub(self.channel)
         self.logger = logger
 

kubernetes/wazuh-vat-evidence-collector-configmap.yaml

 apiVersion: v1
 kind: ConfigMap
 metadata:
-  name: wazuh-vat-evidence-collector-config
+    name: wazuh-vat-evidence-collector-env
 data:
-  constants.json: |-
-    {
-        "general": {
-            "demo": true
-        },
-        "wazuh": {
-            "host": "localhost",
-            "port": 55000,
-            "username": "wazuh-wui",
-            "password": "wazuh-wui"
-        },
-        "elastic": {
-            "host": "localhost",
-            "port": 9200,
-            "username": "admin",
-            "password": "changeme"
-        },
-        "redis": {
-            "host": "localhost",
-            "port": 6379,
-            "queue": "low"
-        },
-        "clouditor": {
-            "host":"security-assessment-svc",
-            "port": 9090
-        }
-    }
+    demo_mode: true
+    wazuh_host: localhost
+    wazuh_port: 55000
+    wazuh_username: wazuh-wui
+    wazuh_password: wazuh-wui
+
+    elastic_host: localhost
+    elastic_port: 9200
+    elastic_username: admin
+    elastic_password: changeme
+        
+    redis_host: localhost
+    redis_port: 6379
+    redis_queue: low
+        
+    clouditor_host: security-assessment-svc
+    clouditor_port: 9090
\ No newline at end of file

kubernetes/wazuh-vat-evidence-collector-deployment.yaml

@@ -13,18 +13,13 @@ spec:
       labels:
         app: wazuh-vat-evidence-collector
     spec:
-      volumes:
-        - name: config-volume
-          configMap:
-            name: wazuh-vat-evidence-collector-config
       containers:
         - image: optima-medina-docker-dev.artifact.tecnalia.com/wp3/t32/wazuh-vat-evidence-collector:latest
           name: wazuh-vat-evidence-collector
           imagePullPolicy: Always
-          volumeMounts:
-            - name: config-volume
-              mountPath: /evidence-collector/constants.json
-              subPath: constants.json
+          envFrom:
+            - configMapRef:
+              name: wazuh-vat-evidence-collector-env
           env:
             - name: TIME
               value: {{time}}

requirements.txt

@@ -27,4 +27,4 @@ rq-scheduler==0.11.0
 rsa==4.8
 six==1.16.0
 uritemplate==4.1.1
-urllib3==1.25.8
+urllib3==1.25.8
\ No newline at end of file

scheduler/scheduler.py

+import os
 from redis import Redis
 from rq import Queue
 from rq_scheduler import Scheduler
 from wazuh_evidence_collector import wazuh_evidence_collector
-from wazuh_evidence_collector.wazuh_evidence_collector import CONSTANTS, LOGGER
+from wazuh_evidence_collector.wazuh_evidence_collector import LOGGER
+
+REDIS_HOST = os.environ.get("redis_host")
+REDIS_PORT = os.environ.get("redis_port")
+REDIS_QUEUE = os.environ.get("redis_queue")
 
 def remove_jobs(scheduler):
     jobs = scheduler.get_jobs()
@@ -14,8 +19,8 @@ def print_jobs(scheduler):
     for job in jobs:
         LOGGER.info(job)
 
-redis = Redis(CONSTANTS['redis']['host'], CONSTANTS['redis']['port'])
-q = Queue(CONSTANTS['redis']['queue'], connection=redis)
+redis = Redis(REDIS_HOST, REDIS_PORT)
+q = Queue(REDIS_QUEUE, connection=redis)
 scheduler = Scheduler(connection=redis)
 
 # TODO: Remove if needed
@@ -28,7 +33,7 @@ scheduler.cron(
     func=wazuh_evidence_collector.main,
     args=[],
     repeat=None,
-    queue_name=CONSTANTS['redis']['queue'],
+    queue_name=REDIS_QUEUE,
     use_local_timezone=False
 )
 

wazuh_evidence_collector/wazuh_evidence_collector.py

 import json
+import os
 from wazuh_evidence_collector.wazuh_client import WazuhClient
 from elasticsearch import Elasticsearch
 from elasticsearch_dsl import Search
@@ -10,29 +11,35 @@ import uuid
 import configparser
 import logging.config
 
-f = open('constants.json',)
-CONSTANTS = json.load(f)
-f.close()
-
 logging.config.fileConfig('logging.conf')
 LOGGER = logging.getLogger('root')
 
-DEMO = CONSTANTS["general"]["demo"]
+DEMO = os.environ.get("demo_mode")
+
+WAZUH_HOST = os.environ.get("wazuh_host")
+WAZUH_PORT = os.environ.get("wazuh_port")
+WAZUH_USERNAME = os.environ.get("wazuh_username")
+WAZUH_PASSWORD = os.environ.get("wazuh_password")
+
+ELASTIC_HOST = os.environ.get("elastic_host")
+ELASTIC_PORT = os.environ.get("elastic_port")
+ELASTIC_USERNAME = os.environ.get("elastic_username")
+ELASTIC_PASSWORD = os.environ.get("elastic_password")
 
 if not DEMO:
-    wc = WazuhClient(CONSTANTS['wazuh']['host'], CONSTANTS['wazuh']['port'], CONSTANTS['wazuh']['username'], CONSTANTS['wazuh']['password'])
+    wc = WazuhClient(WAZUH_HOST, WAZUH_PORT, WAZUH_USERNAME, WAZUH_PASSWORD)
 
     es = Elasticsearch(
-            CONSTANTS['elastic']['host'],
-            http_auth=(CONSTANTS['elastic']['username'], CONSTANTS['elastic']['password']),
+            ELASTIC_HOST,
+            http_auth=(ELASTIC_USERNAME, ELASTIC_PASSWORD),
             scheme='https',
-            port=CONSTANTS['elastic']['port'],
+            port=ELASTIC_PORT,
             use_ssl=False,
             verify_certs=False,
             ssl_show_warn=False,
         )
 
-forwarder = ForwardEvidence(CONSTANTS, LOGGER)
+forwarder = ForwardEvidence(LOGGER)
 
 # Get ID (UUID)
 def get_id():