-
Zitnik, Anze authoredZitnik, Anze authored
- Evidence Collector
- Wazuh evidence collector
- Installation & use
- Using docker
- Local environment
- Component configuration
- Environment variables
- Medina resource ID mapping
- Generate gRPC code from .proto files
- API User authentication
- Manual Elasticsearch API testin with cURL
- Running RQ and RQ-scheduler locally
- Known issues & debugging
- Debugging gRPC services
- Python Elasticsearch library problems with ODFE
Evidence Collector
Author: XLAB
This project includes modules for collecting evidence regarding Wazuh and VAT and sending it to Clouditor for further processing.
Wazuh evidence collector
Wazuh evidence collector uses Wazuh's API to access information about manager's and agents' system informations and configurations. As an additional measure to ensure correct configuration of ClamAV (if installed on machine) we also make use of Elasticsearch's API to dirrectly access collected logs | Elastic stack is one of the Wazuh's required components (usually installed on the same machine as Wazuh server, but can be stand alone as well).
Installation & use
Using docker
-
Set up your Wazuh development environment. Use Security Monitoring repository to create and deploy Vagrant box with all the required components.
-
Clone this repository.
-
Build Docker image:
$ make build
-
Run the image:
$ make run
Note: See
Environment variables
section for more information about configuration of this component and it's interaction with Wazuh, Clouditor etc.
Local environment
-
Set up your Wazuh development environment. Use Security Monitoring repository to create and deploy Vagrant box with all required components.
-
Clone this repository.
-
Install dependencies:
$ pip install -r requirements.txt
-
Set environment variables:
$ source .env
-
a) Install Redis server locally:
$ sudo apt-get install redis-server
Note: To stop Redis server use
/etc/init.d/redis-server stop
.b) Run Redis server in Docker container:
$ docker run --name my-redis-server -p 6379:6379 -d redis
In this case also comment-out server start command in
entrypoint.sh
:#redis-server &
-
Run
entrypoint.sh
:$ ./entrypoint.sh
Note: This repository consists of multiple Python modules. When running Python code manually, use of
-m
flag might be necessary.
Component configuration
Environment variables
Required environment variables (if deployed locally) are located and can be set in .env
file.
Variables used when deploying to Kubernetes can be edited in data
section of /kubernetes/wazuh-vat-evidence-collector-configmap.yaml
file.
All of the following environment variables have to be set (or passed to container) for evidence-collector
to work:
Variable | Description |
---|---|
dummy_wazuh_manager |
Default value false . Set to true in case Evidence collector runs alone (without security-monitoring framework) locally - generates dummy data. |
wazuh_host |
Wazuh manager host's IP address. |
wazuh_port |
Wazuh manager port. Default value 55000 . |
wazuh_username |
Wazuh manager's username. |
wazuh_password |
Wazuh manager's password. |
elastic_host |
Elasticsearch host's IP address. Usually same as wazuh_host . |
elastic_port |
Elasticsearch port. Default value 9200 . |
elastic_username |
Elasticsearch's username. |
elastic_password |
Elasticsearch's password. |
redis_host |
Redis server host's IP address. Usually localhost . |
redis_port |
Redis server port. Default value 6379 . |
redis_queue |
Redis queue name. |
local_clouditor_deploy |
Default value true . Set to false in case Evidence collector will be using Kubernetes deployed Clouditor. |
clouditor_host |
Clouditor host's IP address. |
clouditor_port |
Clouditor port. Default value 9090 . |
clouditor_oauth2_port |
Clouditor port used for authentication services. Default value 8080 . |
clouditor_client_id |
Clouditor OAuth2 default id. Default value clouditor . |
clouditor_client_secret |
Clouditor OAuth2 default secret. Default value clouditor . |
clouditor_oauth2_scope |
Must be defined if local_clouditor_deploy is set to false . Defines scope used when requesting OAuth2 token. |
wazuh_check_interval |
Interval in seconds (rounded to a minute/60 second intervals); how often should evidence be created and forwarded. Should be the same as the check interval set on Wazuh manager. |
wazuh_rule_level |
Min. Wazuh rule severity level that is required for an event to be counted as a threat. |
Medina resource ID mapping
Resource IDs used to generate evidence resources can be easily mapped to required values. In case ID isn't set, Evidence collector will use name
parameter acquired from Wazuh - which is set to machine's hostname, unless explicitly set to something else.
IDs can be set as key:value
pairs inside resource_id_map.json
file, that is later passed to Docker container:
{
"manager": "wazuh_manager",
"agent1": "test_agent_1",
"agent2": "test_agent_2"
}
Where key
represents Wazuh's name
parameter (machine's hostname) and value
equals to string name
will be mapped to.
.proto
files
Generate gRPC code from pip3 install grpcio-tools # (included in requirements.txt)
python3 -m grpc_tools.protoc --proto_path=proto evidence.proto --python_out=grpc_gen --grpc_python_out=grpc_gen
python3 -m grpc_tools.protoc --proto_path=proto assessment.proto --python_out=grpc_gen --grpc_python_out=grpc_gen
python3 -m grpc_tools.protoc --proto_path=proto metric.proto --python_out=grpc_gen --grpc_python_out=grpc_gen
As we are interacting with Clouditor, .proto files are taken from there.
Because of dependencies on Google APIs, .proto files in proto/google are taken from here.
Note: since we are running the code as a package, we have to modify imports in newly generated code:
import evidence_pb2 as evidence__pb2
-->import grpc_gen.evidence_pb2 as evidence__pb2
(check all generated files)
API User authentication
Current implementation has disabled SSL certificate verification & uses simple username/password verification (defined inside /constants/constants.py
). Production version should change this with cert verification.
Manual Elasticsearch API testin with cURL
Example command for testing the API via CLI:
$ curl --user admin:changeme --insecure -X GET "https://192.168.33.10:9200/wazuh-alerts*/_search?pretty" -H 'Content-Type: application/json' -d'
{"query": {
"bool": {
"must": [{"match": {"predecoder.program_name": "clamd"}},
{"match": {"rule.description": "Clamd restarted"}},
{"match": {"agent.id": "001"}}]
}
}
}'
RQ and RQ-scheduler locally
Running-
Install (if needed) and run
redis-server
:$ sudo apt-get install redis-server $ redis-server
Note: By default, server listens on port
6379
. Take this into consideration when starting other components. -
Install RQ and RQ-scheduler:
$ pip install rq $ pip install rq-scheduler
-
Run both components in 2 terminals:
$ rqworker low $ rqscheduler --host localhost --port 6379
Note:
low
in the first command references task queue worker will use. -
Run Python script containing RQ commands as usual:
$ python3 -m wazuh_evidence_collector.wazuh_evidence_collector
Known issues & debugging
Debugging gRPC services
gRPC can be easily set to verbose debug mode by adding the following variables to .env
file passed to Docker container:
GRPC_VERBOSITY=DEBUG
GRPC_TRACE=http,tcp,api,channel,connectivity_state,handshaker,server_channel
Full list of gRPC environment variables is available here.
Python Elasticsearch library problems with ODFE
Latest versions (7.14.0
& 7.15.0
) of Python Elasticsearch library have problems connecting to Open Distro for Elasticsearch and produce the following error when trying to do so:
elasticsearch.exceptions.UnsupportedProductError: The client noticed that the server is not a supported distribution of Elasticsearch
To resolve this, downgrade to older package version:
$ pip install 'elasticsearch<7.14.0'