Merge branch 'develop' into severity

Author: kamil-certat

.github/workflows/codespell.yml

@@ -18,7 +18,7 @@ on:
 jobs:
   codespell:
     name: Find and notify about common misspellings
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-latest
     # This should not fail the whole workflow run
     continue-on-error: true
 

.github/workflows/debian-package.yml

@@ -16,7 +16,7 @@ on:
 
 jobs:
   build:
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-latest
     env:
       # Fixes https://github.com/actions/virtual-environments/issues/3080
       STORAGE_OPTS: overlay.mount_program=/usr/bin/fuse-overlayfs

.github/workflows/regexploit.yml

@@ -19,14 +19,16 @@ on:
 jobs:
   regexploit:
     name: Find regular expressions which are vulnerable to ReDoS
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-latest
     # This should not fail the whole workflow run
     continue-on-error: true
 
     steps:
     - name: Checkout repository
       uses: actions/checkout@v3
     - name: Install regexploit
-      run: pip install regexploit
+      #run: pip install regexploit
+      # See https://github.com/doyensec/regexploit/pull/16
+      run: pip install git+https://github.com/sebix/regexploit.git@unsupported-ops-yaml
     - name: Run regexploit
       run: /home/runner/work/intelmq/intelmq/.github/workflows/scripts/regexploit.sh

.github/workflows/scripts/setup-full.sh

@@ -15,10 +15,6 @@ sudo chown -R elasticsearch:elasticsearch /etc/default/elasticsearch
 sudo systemctl start elasticsearch
 
 sudo apt update
-if [ $python_version == '3.8' ]; then
-	# for pymssql there are no wheels for 3.8 https://github.com/certtools/intelmq/issues/2539
-	DEBIAN_FRONTEND="noninteractive" sudo -E apt install -y build-essential freetds-dev libssl-dev libkrb5-dev
-fi
 # for psql (used below)
 DEBIAN_FRONTEND="noninteractive" sudo -E apt install -y postgresql-client
 

.github/workflows/unittests.yml

@@ -18,7 +18,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ['3.8', '3.9', '3.10', '3.11', '3.12', '3.13']
+        python-version: ['3.9', '3.10', '3.11', '3.12', '3.13']
         type: ['full', 'basic']
 
     services:

CHANGELOG.md

@@ -14,8 +14,14 @@ Please refer to the [NEWS](NEWS.md) for a list of changes which have an affect o
 --------------------------------
 
 ### Configuration
+- New parameter `stop_retry_limit` (PR#2598 by Lukas Heindl).
 
 ### Core
+- Drop support for Python 3.8 (fixes #2616, PR#2617 by Sebastian Wagner).
+- `intelmq.lib.splitreports`: Handle bot parameter `chunk_size` values empty string, due to missing parameter typing checks (PR#2604 by Sebastian Wagner).
+- `intelmq.lib.mixins.sql` Add Support for MySQL (PR#2625 by Karl-Johan Karlsson).
+- New parameter `stop_retry_limit` to gracefully handle stopping bots which take longer to shutdown (PR#2598 by Lukas Heindl, fixes #2595).
+- `intelmq.lib.datatypes`: Remove unneeded Dict39 alias (PR#2639 by Nakul Rajpal, fixes #2635)
 
 ### Development
 
@@ -26,24 +32,54 @@ Please refer to the [NEWS](NEWS.md) for a list of changes which have an affect o
 
 ### Bots
 #### Collectors
+- Remove `intelmq.bots.collectors.blueliv` as it uses an unmaintained library, does not work any more and breaks other CI tests (fixes #2593, PR#2632 by Sebastian Wagner).
 
 #### Parsers
+- `intelmq.bots.parsers.cymru.parser_cap_program`: Add mapping for TOR and ipv6-icmp protocol (PR#2621 by Mikk Margus Möll).
+- Remove `intelmq.bots.collectors.blueliv` as it is obsolete with the removed collector (PR#2632 by Sebastian Wagner).
 
 #### Experts
-- `intelmq.bots.experts.asn_lookup.expert`: Print URLs to stdout only in verbose mode (PR#2591 by Sebastian Wagner).
+- `intelmq.bots.experts.asn_lookup.expert`:
+  - Print URLs to stdout only in verbose mode (PR#2591 by Sebastian Wagner).
+  - Check for database file existence and writability (fixes #2566).
+  - Use database path matching to installation type (PR#2606 by Sebastian Wagner).
+- `intelmq.bots.experts.fake.expert`:
+  - Use database path matching to installation type (PR#2606 by Sebastian Wagner).
+  - Add new mode `random_single_value` (PR#2601 by Sebastian Wagner).
+- `intelmq.bots.experts.sieve.expert`: Test for textX dependency in self-check (PR#2605 by Sebastian Wagner).
+- `intelmq.bots.experts.trusted_introducer_lookup.expert`: Change to new TI database URL (fixes #2620, PR#2633 by Sebastian Wagner).
 
 #### Outputs
+- `intelmq.bots.outputs.smtp_batch.output`:
+  - Add new parameter `additional_grouping_keys` for an enhanced email batching feature.
+  - Add new parameter `templating` for additional template variables.
+  - Add new parameter `allowed_fieldnames` for csv field specification.
+  - Add new parameter `fieldnames_translation` for naming csv headers (PR#2610 by Lukas Heindl, fixes #2586).
+- `intelmq.bots.outputs.sql.output`: Add Support for MySQL (PR#2625 by Karl-Johan Karlsson).
 
 ### Documentation
+- Fix and refresh links to mailing lists (PR#2609 by Kamil Mańkowski)
+- `Aggregate Bot`: Add illustration graphics (PR#2612 by Sebastian Wagner).
 
 ### Packaging
 - Replace `/opt/intelmq` example paths in bots with variable `VAR_STATE_PATH` for correct paths in LSB-path setups like with packages (PR#2587 by Sebastian Wagner).
+- New deb-package `intelmq-contrib` with all `contrib/` scripts and documentation (PR#2614 by Sebastian Wagner).
+- New deb-package `intelmq-autostart` containing systemd services and timers to start all enabled IntelMQ bots at boot and periodically (PR#2638 by Sebastian Wagner).
 
 ### Tests
+- `intelmq.tests.lib.test_pipeline.TestAmqp.test_acknowledge`: Skip on all Python versions when running on CI (PR#2602 by Sebastian Wagner).
+- `.github/workflows/codespell.yml`, `debian-package.yml`, `regexploit.yml`: Upgrade to `ubuntu-latest` runners (PR#2602 by Sebastian Wagner).
+- `intelmq.test.test_conf`: With changed behaviour in ruamel.yaml on line wrapping since version 0.18.13, only test the parsabilty of `runtime.yaml` (PR#2619 by Sebastian Wagner).
+- `intelmq.test.BotTestCase.test_static_bot_check_method`: Remove debugging stub raising for all non-empty checks (PR#2622 by Sebastian Wagner).
 
 ### Tools
+- `intelmq.bin.intelmq_psql_initdb`: Use `JSONB` type by default, Postgres supports it since version 9 (PR#2597 by Sebastian Wagner).
+- `intelmq.bin.rewrite_config_files`: Removed obsolete JSON configuration file rewriter (PR#2613 by Sebastian Wagner).
+- `intelmq/lib/bot_debugger.py`: Fix overwriting the runtime logging level by command line parameter (PR#2603 by Sebastian Wagner, fixes #2563).
+- `intelmq.bin.intelmqctl`: Fix bot log level filtering (PR#2607 by Sebastian Wagner, fixes #2596).
 
 ### Contrib
+- Bash Completion: Adapt to YAML-style runtime configuration (PR#2642 by Sebastian Wagner, fixes #2094).
 
 ### Known issues
 
@@ -108,6 +144,7 @@ Please refer to the [NEWS](NEWS.md) for a list of changes which have an affect o
 - `intelmq.tests.lib.test_pipeline.TestAmqp.test_acknowledge`: Also skip on Python 3.11 and 3.12 besides on 3.8 when running on CI (PR#2542 by Sebastian Wagner).
 - Full pytest workflow: Version-independent install of postgres client, for Ubuntu 24.04 (default on GitHub now) test environment compatibility (PR#2557 by Sebastian Wagner).
 - Debian package build workflow: Use artifact upload v4 instead of v3 (PR#2565 by Sebastian Wagner).
+- `intelmq.lib.test.BotTestCase`: Show the warnings and errors of a bot if the amount does not match the expected number (PR#2578 by Kamil Mankowski/Sebastian Wagner, related to #2571).
 
 ### Known issues
 This is short list of the most important known issues. The full list can be retrieved from [GitHub](https://github.com/certtools/intelmq/labels/bug?page=2&q=is%3Aopen+label%3Abug).

MANIFEST.in

@@ -10,7 +10,7 @@ graft intelmq/tests
 include COPYRIGHT
 include LICENSE
 include CHANGELOG.md
-recursive-exclude intelmq/bin intelmq_gen_feeds_docs.py intelmq_gen_harm_docs.py rewrite_config_files.py
+recursive-exclude intelmq/bin intelmq_gen_feeds_docs.py intelmq_gen_harm_docs.py
 exclude .*
 global-exclude *~
 global-exclude *.py[co]

Makefile

@@ -1,7 +1,7 @@
-# SPDX-FileCopyrightText: 2023 Filip Pokorný
+# SPDX-FileCopyrightText: 2023-2025 Filip Pokorný, Sebastian Wagner
 # SPDX-License-Identifier: AGPL-3.0-or-later
 
-.PHONY: coverage docs clean
+.PHONY: coverage docs clean test codespell codestyle
 
 coverage:
 	python -m pytest --cov=intelmq -v
@@ -22,3 +22,6 @@ test:
 
 codestyle:
 	pycodestyle intelmq/{bots,lib,bin}
+
+licenses:
+	reuse lint

NEWS.md

@@ -14,6 +14,7 @@ Please refer to the change log for a full list of changes.
 --------------------------------
 
 ### Requirements
+Python `>=3.9` is now required, which is available on all platforms supported by IntelMQ.
 
 ### Tools
 
@@ -29,6 +30,11 @@ ALTER TABLE events ADD severity varchar(10);
 ### Libraries
 
 ### Postgres databases
+To switch to the more efficient data type `jsonb` instead of `json`, use the following SQL statement. Data is preserved. JSONB also has more query and data manipulation features than plain JSON.
+```sql
+ALTER TABLE events
+   ALTER COLUMN "extra" SET DATA TYPE jsonb;
+```
 
 
 3.4.0 Feature release (2025-03-14)

contrib/autostart/intelmq-periodic-start.timer

@@ -0,0 +1,14 @@
+# SPDX-FileCopyrightText: 2025 Institute for Common Good Technology
+# SPDX-License-Identifier: AGPL-3.0-or-later
+
+[Unit]
+Description=Periodically start all enabled IntelMQ bots
+
+[Timer]
+# run every 5 minutes
+OnUnitActiveSec=5minutes
+Unit=intelmq-start.service
+Persistent=true
+
+[Install]
+WantedBy=multi-user.target timers.target