From e14f8a5f33fe31a95c52120ed815ae040e7d2160 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?An=C5=BEe=20=C5=BDitnik?= <anze.zitnik@xlab.si> Date: Fri, 15 Jul 2022 15:05:17 +0200 Subject: [PATCH] Update README.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Squashed commit of the following: commit 40d02c489bce9ea1db0685d498a269b18c8cdff6 Author: Anže Žitnik <anze.zitnik@xlab.si> Date: Fri Jul 15 13:04:16 2022 +0000 Update README.md commit e0241dc710b653ef21bfeec5b42a33131a4978f5 Merge: 2499629 04db64f Author: Matevž Eržen <matevz.erzen@xlab.si> Date: Fri May 20 14:10:25 2022 +0000 Merge branch 'redis-version-upgrade' into 'master' Upgraded Redis version See merge request medina/evidence-collector!18 commit 04db64f0ff389a485b11a47ec047a5cf1479edef Author: Matevz Erzen <matevz.erzen@xlab.si> Date: Fri May 20 16:06:59 2022 +0200 Upgraded Redis version commit 249962904cfbcd0e98517ceb141fc2f0d27f7153 Merge: 64ce6c5 7d66a73 Author: Matevž Eržen <matevz.erzen@xlab.si> Date: Fri May 20 12:53:53 2022 +0000 Merge branch 'debug-logging' into 'master' Additional logging See merge request medina/evidence-collector!17 commit 7d66a73307239baa2ad20651087f93fefd8ee405 Author: Matevz Erzen <matevz.erzen@xlab.si> Date: Fri May 20 14:48:56 2022 +0200 Additional logging commit 64ce6c5331d146c3fb7113b3cfb23a0c17d0373d Merge: 641bb58 e9398a1 Author: Anže Žitnik <anze.zitnik@xlab.si> Date: Fri May 20 09:40:20 2022 +0000 Merge branch 'k8s-config-update' into 'master' K8s config update See merge request medina/evidence-collector!16 commit e9398a1e1587b8af4b9b5515559abeb3661e84bc Author: Matevž Eržen <matevz.erzen@xlab.si> Date: Fri May 20 09:40:20 2022 +0000 K8s config update commit 641bb587698893abd8c83fde30353dc1ae96bf6a Merge: 320c3c2 c7f5e0a Author: Anže Žitnik <anze.zitnik@xlab.si> Date: Wed Apr 20 15:52:16 2022 +0000 Merge branch 'oauth2-config-update' into 'master' gRPC config & exception handling update See merge request medina/evidence-collector!15 commit c7f5e0aa7a4efa60542d392b86d80b9f78e8bcc0 Author: Matevž Eržen <matevz.erzen@xlab.si> Date: Wed Apr 20 15:52:16 2022 +0000 gRPC config & exception handling update commit 320c3c2cd4b6b59046d586a5af95d097d2150380 Merge: cfc71e8 6186bb8 Author: Anže Žitnik <anze.zitnik@xlab.si> Date: Tue Apr 19 09:59:10 2022 +0000 Merge branch 'config-update' into 'master' Updated clouditor host & port config See merge request medina/evidence-collector!14 commit 6186bb874c96c8371966ef7f9cf0ef095d0a5017 Author: Matevz Erzen <matevz.erzen@xlab.si> Date: Tue Apr 19 11:55:57 2022 +0200 Updated clouditor host & port config commit cfc71e8dd4875e84863f8caa6d2c89e3ef7c1d4a Author: Anže Žitnik <anze.zitnik@xlab.si> Date: Tue Apr 12 16:58:22 2022 +0200 Fix k8s config commit ec8ddf6a1b249014d33fa86249a7d266c91fff20 Merge: 4664618 60bd257 Author: Anže Žitnik <anze.zitnik@xlab.si> Date: Tue Apr 5 11:05:06 2022 +0000 Merge branch 'wazuh-threat-count' into 'master' Wazuh threat count See merge request medina/evidence-collector!13 commit 60bd257e603974b14dd1e266d6988263612ba9ef Author: Matevž Eržen <matevz.erzen@xlab.si> Date: Tue Apr 5 11:05:05 2022 +0000 Wazuh threat count commit 466461887635f53de24a5acf45a2de5b9ff6117c Merge: 2eb0fa0 4ee3ff1 Author: Anže Žitnik <anze.zitnik@xlab.si> Date: Fri Apr 1 12:31:15 2022 +0000 Merge branch 'cron-interval' into 'master' Custom Cron job interval See merge request medina/evidence-collector!12 commit 4ee3ff18884def9d125f0402c4b3b0bac9a9b842 Author: Matevž Eržen <matevz.erzen@xlab.si> Date: Fri Apr 1 12:31:15 2022 +0000 Custom Cron job interval commit 2eb0fa0683492764ca7ec6afe1019e4be447b7b9 Merge: a8a6550 2a3f602 Author: Matevž Eržen <matevz.erzen@xlab.si> Date: Fri Mar 25 15:10:28 2022 +0000 Merge branch 'resource-id-mapping' into 'master' Resource ID mapping See merge request medina/evidence-collector!11 commit 2a3f602fb67294226cc1fbf9c1650150fce25065 Author: Matevž Eržen <matevz.erzen@xlab.si> Date: Fri Mar 25 15:10:28 2022 +0000 Resource ID mapping commit a8a6550959f313d8ce56e083ebcd24e433ddf941 Merge: 530ddad cb432ad Author: Anže Žitnik <anze.zitnik@xlab.si> Date: Thu Mar 24 11:57:45 2022 +0100 Merge branch 'master' of git.code.tecnalia.com:medina/wp3/task_3.2/evidence-collector commit 530ddad1b8d287c3d019f663337bd3944e13e9ee Merge: a7951bf 037167f Author: Anže Žitnik <anze.zitnik@xlab.si> Date: Thu Mar 24 10:52:27 2022 +0000 Merge branch 'updated-error-handling' into 'master' Updated error handling and tests See merge request medina/evidence-collector!10 commit 037167ff680e9a5ad2fae18f9161b4acb8333a6b Author: Matevž Eržen <matevz.erzen@xlab.si> Date: Thu Mar 24 10:52:27 2022 +0000 Updated error handling and tests commit a7951bf424f532b35c961c1aca24404ad5b257e4 Author: Matevz Erzen <matevz.erzen@xlab.si> Date: Tue Mar 22 16:57:43 2022 +0100 Minor bug fix commit eed1854b7eeb0b7cfd28313918c5a9d7365f00af Merge: 99cf3d5 0ab0d80 Author: Matevž Eržen <matevz.erzen@xlab.si> Date: Tue Mar 22 15:04:16 2022 +0000 Merge branch 'oauth2-implementation' into 'master' Oauth2 implementation See merge request medina/evidence-collector!9 commit 0ab0d80e5e2467c40c22afa1a6de0041b472c658 Author: Matevž Eržen <matevz.erzen@xlab.si> Date: Tue Mar 22 15:04:16 2022 +0000 Oauth2 implementation commit 99cf3d50fbf87b546ebaf32fd244c049255fdd05 Author: Anže Žitnik <anze.zitnik@xlab.si> Date: Wed Mar 9 16:03:41 2022 +0100 Fixed problems with environment variables & updated README commit cb432ad4a0cac358bf7b2eaeaac809402db4ce13 Author: Anže Žitnik <anze.zitnik@xlab.si> Date: Tue Mar 8 10:10:30 2022 +0100 Version 0.0.8 commit 75a950df9ed926861e7b6ec78f3233f2ca85f0d7 Author: Matevz Erzen <matevz.erzen@xlab.si> Date: Mon Mar 7 17:38:19 2022 +0100 Migration from json config to env variables Migration from json config to env variables Fixed failing CI job commit f1f537ffcb1067c4dc8f0ae8be223934917b3846 Author: Matevz Erzen <matevz.erzen@xlab.si> Date: Tue Feb 22 13:15:15 2022 +0100 Add malwareProtectionOutput metric Separate malware protection evidence Data combined in single evidence, updated output struct Fixed Python image in Dockerfile commit 40821f7eaba7f9ffb38be4d8089f25fe49f07160 Merge: a0b4d6d 3aa9664 Author: Anže Žitnik <anze.zitnik@xlab.si> Date: Tue Feb 15 12:42:06 2022 +0000 Merge branch 'elasticsearch-query-update' into 'master' Corrected elasticsearch rule.description query See merge request medina/evidence-collector!5 commit 3aa96646ee135168ee42644411130839f1710d85 Author: Matevz Erzen <matevz.erzen@xlab.si> Date: Tue Feb 15 13:22:58 2022 +0100 Corrected elasticsearch rule.description query commit a0b4d6d5a4f840fb309e53505cd65cb22c54ab95 Author: Anže Žitnik <anze.zitnik@xlab.si> Date: Tue Feb 15 11:45:12 2022 +0100 CI: automate deploy to MEDINA k8s. commit 57b66dc205af552e440c6ba77d6979091911d87f Author: Anže Žitnik <anze.zitnik@xlab.si> Date: Thu Jan 20 17:35:06 2022 +0100 Add basic kubernetes YAML definitions. commit 1cf48f6e5cc4599e9aebd332a7935779dfeba32b Author: Anže Žitnik <anze.zitnik@xlab.si> Date: Thu Jan 20 16:22:15 2022 +0100 Add demo mode function. Squashed commit of the following: commit 718efd4db1fd64e5baed53d27c149219fab47879 Author: Anže Žitnik <anze.zitnik@xlab.si> Date: Thu Jan 20 16:03:58 2022 +0100 Version up commit c574b33acd9c5c131e8a2b53ad69ff1a970d2203 Author: Anže Žitnik <anze.zitnik@xlab.si> Date: Thu Jan 20 16:02:07 2022 +0100 Add demo mode function. Created an option in constants.json where demo mode can be selected. It will not query wazuh or elastic, just output random evidence. commit 7e7ce80679516eb49151a074b0f660ef3c6b5a3e Author: Anže Žitnik <anze.zitnik@xlab.si> Date: Tue Jan 18 13:56:32 2022 +0100 Rename artifact name adding wazuh-vat prefix to evidence-collector commit 06f97ad3166eb355dd80c2a324340aa0ae668de5 Author: Anže Žitnik <anze.zitnik@xlab.si> Date: Tue Jan 18 13:49:44 2022 +0100 Add docker push to MEDINA artifactory to CI script commit 6f3d5cfb517a10d24140063b0cf14e6b46e16c2a Author: Anže Žitnik <anze.zitnik@xlab.si> Date: Tue Dec 14 13:40:12 2021 +0100 Fix some typos in code commit 8904133d4ea66e6305c0e07c09ab799f8a8f2d0a Author: Anže Žitnik <anze.zitnik@xlab.si> Date: Tue Dec 14 11:49:08 2021 +0100 Use AssessEvidence instead of StoreEvidence RPC on Clouditor. Some refactoring. Squashed commit of the following: commit 11ae9a48f6b41c2dc5b3e00de1b808b75cc39013 Author: Anže Žitnik <anze.zitnik@xlab.si> Date: Tue Dec 14 11:40:01 2021 +0100 Change CI script: build and test all branches commit fe84541d50ffc6b25d5fff94b1781345ec2b548d Author: Anže Žitnik <anze.zitnik@xlab.si> Date: Tue Dec 14 11:33:07 2021 +0100 Version up commit b99df078408ea2649ce59cd2d17c247c04c6a992 Author: Anže Žitnik <anze.zitnik@xlab.si> Date: Tue Dec 14 11:25:12 2021 +0100 Update dockerignore: add (v)env folder. commit 20944e6743ce112d558fb0205a9347a46c17da8b Author: Anže Žitnik <anze.zitnik@xlab.si> Date: Tue Dec 14 10:56:41 2021 +0100 Refactoring: move all gRPC-generated files to their own package (grpc_gen). commit f4dce9c9076a1336dc7d0b5e15759b24c4f9bea7 Author: Anže Žitnik <anze.zitnik@xlab.si> Date: Tue Dec 14 10:34:05 2021 +0100 Use AssessEvidence instead of StoreEvidence RPC on Clouditor. Added necessary proto files (removed unneeded) and regenerated python code from them. Also added Google APIs dependencies for grpc code generation. Accordingly updated README. Also updated requirements.txt to include some packages for grpc python code generation and for usage of Google APIs. Note that with the call to AssessEvidence, we need to provide an AssessEvidenceRequest object (instead of simply Evidence as before). Most changes of existing code are because of this. commit 257ce00a35d55720e82e5fdcb68c2a05ba5e2829 Author: Anže Žitnik <anze.zitnik@xlab.si> Date: Tue Dec 14 11:40:01 2021 +0100 Change CI script: build and test all branches commit 4e52cac558c6461379dada01b456edb08b30f9fb Author: Anže Žitnik <anze.zitnik@xlab.si> Date: Fri Dec 10 13:35:36 2021 +0100 Version up commit 313a01396c591996b8fc14b82895b10039bcf732 Author: Anže Žitnik <anze.zitnik@xlab.si> Date: Fri Dec 10 13:34:37 2021 +0100 Logging all exceptions commit 1e714e735fec4ad86d4101f118cc68da181e1cdf Author: Anže Žitnik <anze.zitnik@xlab.si> Date: Thu Dec 9 17:49:41 2021 +0100 Use logging. Replaced all print() calls with logger.info/debug. Printing evidence objects. Logging to /var/log/evidence_collector.log and setting tailf to this file in docker entrypoint. Version 0.0.4. Squashed commit of the following: commit 825d1f95a141f8e11703c27889e53a6e16c3cd66 Author: Anže Žitnik <anze.zitnik@xlab.si> Date: Wed Dec 1 12:21:26 2021 +0100 Output logs to /var/log... and tailf in Dockerfile commit db9b34317d19e42316fc0c5f0a8f60b03b2e4dbc Author: Anže Žitnik <anze.zitnik@xlab.si> Date: Wed Dec 1 12:19:54 2021 +0100 Introduce logger. Replace all prints. commit e6220b090e9e4304ea1f64163cefaf7f0c265150 Merge: 4484a05 1636c41 Author: Anže Žitnik <anze.zitnik@xlab.si> Date: Mon Nov 22 08:46:47 2021 +0000 Merge branch 'develop' into 'master' Updated gRPC message structure See merge request medina/evidence-collector!4 commit 1636c4119b1e7d0c24f6485a0cde0d7ab81b04c4 Author: Matevž Eržen <matevz.erzen@xlab.si> Date: Mon Nov 22 08:46:47 2021 +0000 Updated gRPC message structure commit 4484a05f7f88f2bb67d783a16a96730d3f9749e8 Merge: 9221f03 d4329db Author: Anže Žitnik <anze.zitnik@xlab.si> Date: Mon Nov 15 15:56:17 2021 +0000 Merge branch 'develop' into 'master' gRPC implementation See merge request medina/evidence-collector!3 commit d4329db8e85cb14870ebaf696d467afb9c96ce69 Author: Matevž Eržen <matevz.erzen@xlab.si> Date: Mon Nov 15 15:56:16 2021 +0000 gRPC implementation commit 9221f03bca65d165a05338db2dce8daac1c920d8 Author: Anže Žitnik <anze.zitnik@xlab.si> Date: Fri Nov 5 09:02:24 2021 +0100 Replaced LICENSE (Apache 2.0). commit 5b488ed84d9f0260c151085deea5bb1e4bc605ec Merge: eef54a0 658a7ec Author: Matevž Eržen <matevz.erzen@xlab.si> Date: Tue Oct 19 11:53:52 2021 +0000 Merge branch 'develop' into 'master' Merge CI tests to Master See merge request medina/evidence-collector!2 commit 658a7ec6126e8ed2c4d04eccbb5022d00bc28089 Author: Matevž Eržen <matevz.erzen@xlab.si> Date: Tue Oct 19 11:53:52 2021 +0000 Merge CI tests to Master commit eef54a042c9148be28f3c3baf15c40cdd23bdf0b Merge: c0b92a9 254dd87 Author: Anže Žitnik <anze.zitnik@xlab.si> Date: Fri Oct 8 15:20:36 2021 +0000 Merge branch 'develop' into 'master' Added initial Docker image See merge request medina/evidence-collector!1 commit 254dd879331e97594261531444ba20cf3c6996c7 Author: Matevz Erzen <matevz.erzen@xlab.si> Date: Fri Oct 8 14:35:45 2021 +0200 Added CI config commit fe17530489058514480c2e80b80ea7b9ae34bd22 Author: matevz_erzen <matevz.erzen@xlab.si> Date: Tue Oct 5 10:53:59 2021 +0200 Working Docker image commit e9bb14da18b1f9945b171d45df6d09e77d4c4b93 Author: matevz_erzen <matevz.erzen@xlab.si> Date: Fri Oct 1 12:37:15 2021 +0200 Added scheduling and proto files commit 7a579ff149e4de2f6789d82c6b43c958af8ca2eb Author: matevz_erzen <matevz.erzen@xlab.si> Date: Tue Sep 28 11:33:43 2021 +0200 Added Dockerfile commit 3384ba8d784b9975b5471fc1152c8af6c456bdf7 Author: matevz_erzen <matevz.erzen@xlab.si> Date: Fri Sep 24 12:46:51 2021 +0200 Added ClamAV install verification via Elasticsearch commit c0b92a963801b3cc2739a582173724fca721d36c Author: Anže Žitnik <anze.zitnik@xlab.si> Date: Tue Sep 28 09:46:58 2021 +0200 Empty repo init commit 603b253e8bf53bdcb7c58f1b5946a428ea35dcf4 Author: matevz_erzen <matevz.erzen@xlab.si> Date: Wed Sep 22 15:46:12 2021 +0200 Check ClamAV packages and process commit a0fe1815bef546334ace78de64e8ce07d46f86c2 Author: matevz_erzen <matevz.erzen@xlab.si> Date: Fri Sep 17 14:29:57 2021 +0200 VirusTotal and last scan checks commit 2917af9edf1c7e3701abf9fbc02c1a19f0b7b1c9 Author: matevz_erzen <matevz.erzen@xlab.si> Date: Thu Sep 16 17:24:20 2021 +0200 Basic configuration checks commit d13f5ba70425db9e5d0008cd08a38e33d815a482 Author: Anže Žitnik <anze.zitnik@xlab.si> Date: Wed Sep 15 10:13:39 2021 +0200 initial --- .env | 6 ++--- .gitlab-ci.yml | 59 ++++++++++++++++++++++++++++++++++++++++++++++++++ README.md | 4 ++++ 3 files changed, 66 insertions(+), 3 deletions(-) create mode 100644 .gitlab-ci.yml diff --git a/.env b/.env index 9ae6916..2b5491e 100644 --- a/.env +++ b/.env @@ -3,12 +3,12 @@ dummy_wazuh_manager=false wazuh_host=192.168.33.10 wazuh_port=55000 wazuh_username=wazuh-wui -wazuh_password=wazuh-wui +wazuh_password=password elastic_host=192.168.33.10 elastic_port=9200 elastic_username=admin -elastic_password=changeme +elastic_password=password redis_host=localhost redis_port=6379 @@ -22,7 +22,7 @@ clouditor_port=9090 clouditor_oauth2_host=192.168.33.14 clouditor_oauth2_port=8080 clouditor_client_id=clouditor -clouditor_client_secret=clouditor +clouditor_client_secret=password clouditor_oauth2_scope= ### K8s deployed Clouditor ### diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml new file mode 100644 index 0000000..dc5789d --- /dev/null +++ b/.gitlab-ci.yml @@ -0,0 +1,59 @@ +image: nexus-registry.xlab.si:5001/docker:dind + +variables: + REGISTRY: registry-gitlab.xlab.si + MEDINA_REGISTRY: optima-medina-docker-dev.artifact.tecnalia.com + MEDINA_REG_PATH: wp3/t32 + +before_script: + - export SERVICE=$(grep SERVICE MANIFEST | cut -d '=' -f2) + - export VERSION=$(grep VERSION MANIFEST | cut -d '=' -f2) + +stages: + - build + - test + - push + - deploy + +build: + stage: build + script: + - docker build --no-cache -t $REGISTRY/medina/$SERVICE:$VERSION . + +test: + stage: test + script: + - apk add bash + - docker network create test-ec + - docker run --rm --network=test-ec --env-file .env --name $SERVICE -d $REGISTRY/medina/$SERVICE:$VERSION + - docker run --rm --network=test-ec toschneck/wait-for-it $SERVICE:7890 -t 240 + - bash test/test.sh + after_script: + - SERVICE=$(grep SERVICE MANIFEST | cut -d '=' -f2) + - docker kill $SERVICE || docker network rm test-ec + - docker network rm test-ec + +push: + stage: push + script: + - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $REGISTRY + - docker tag $REGISTRY/medina/$SERVICE:$VERSION $REGISTRY/medina/$SERVICE:latest + - docker push $REGISTRY/medina/$SERVICE:$VERSION + - docker push $REGISTRY/medina/$SERVICE:latest + - docker logout $REGISTRY + - docker login $MEDINA_REGISTRY -u medina.fordevelopers@gmail.com -p AKCp8kqMZkcPRPGZhHBw7uKFsyifF1iHb2ZvbBy5PK88wD8EdeSHZqFsc4h1wp3M2oVYGazhv + - docker tag $REGISTRY/medina/$SERVICE:$VERSION $MEDINA_REGISTRY/$MEDINA_REG_PATH/wazuh-vat-$SERVICE:$VERSION + - docker tag $REGISTRY/medina/$SERVICE:$VERSION $MEDINA_REGISTRY/$MEDINA_REG_PATH/wazuh-vat-$SERVICE:latest + - docker push $MEDINA_REGISTRY/$MEDINA_REG_PATH/wazuh-vat-$SERVICE:$VERSION + - docker push $MEDINA_REGISTRY/$MEDINA_REG_PATH/wazuh-vat-$SERVICE:latest + - docker logout $MEDINA_REGISTRY + only: + - master + +deploy: + stage: deploy + script: + - docker run --rm curlimages/curl -I -X POST "https://xlab:110bb809200c797e6031787b51a049b819@cicd.medina.esilab.org/jenkins/job/medina/job/wp3/job/task_3.2/job/wazuh-vat-evidence-collector-deploy/buildWithParameters?PRJ_ENV=dev&PRJ_IMAGE_TAG=latest&YAMLS_OVERRIDE=" + only: + - master + diff --git a/README.md b/README.md index 1af93c2..831791f 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,9 @@ # Evidence Collector +Author: XLAB + +--- + This project includes modules for collecting evidence regarding Wazuh and VAT and sending it to [Clouditor](https://github.com/clouditor/clouditor) for further processing. ## Wazuh evidence collector -- GitLab