mirror of
https://github.com/Horizontal42/netsleuth.git
synced 2026-11-10 15:42:10 +03:00
detect ECMP load balancing via N-run traceroute diff (--ecmp)
Shipped half of the original design on purpose: generalizing BSD's alt: same-TTL annotation across backends is impossible - mtr's JSON aggregates cycles and discards per-cycle next-hop variance, icmp_win/icmplib send one probe per TTL by construction, and the Linux system-traceroute parser doesn't model multi-IP hops even though the text can contain them. A finding that only ever fires on macOS/Linux-system-binary would be worse than not having it. New probes/ecmp.py::detect_ecmp() takes N independently-run TraceResults for the same target and diffs which IP showed up at each TTL - pure and backend-agnostic, since it never touches capability detection. Flags path.ecmp (info, multiple next hops exist - normal on a load-balanced network) and path.ecmp_asymmetric (warn, branches differ by >20ms - the diagnostic case, since one degraded branch of a load-balanced pair causes intermittent slowness a single traceroute would never catch). --ecmp is opt-in and deliberately excluded from --full: it multiplies traceroute traffic against the same targets by ecmp.runs (capped at 5) across ecmp.max_targets (capped at 3), sequential and sharing one semaphore, only after the main fan-out has finished. Live-tested against 2 real targets x 3 runs each - ran cleanly, correctly reported no divergence on this network. Part of plans/zesty-zooming-wilkinson.md (gitignored).
This commit is contained in:
parent
e27ccdfab5
commit
a64022b12a
14 changed files with 344 additions and 3 deletions
|
|
@ -39,6 +39,7 @@ src/netsleuth/
|
|||
bgp_path.py --path-diversity: CF-RAY edge colo lookup, international-routing-loop detection.
|
||||
prefix_benchmark.py --prefix-bench: ping the first host of a capped sample of the AS's announced prefixes.
|
||||
dpi_check.py --my-server: single-host TCP RST/filtering self-diagnostic. Not a scanner — see Key decisions.
|
||||
ecmp.py --ecmp: pure N-run traceroute diff to spot per-hop load balancing. Backend-agnostic by design.
|
||||
nat64.py One RFC 7050 AAAA query (ipv4only.arpa) to detect NAT64/464XLAT translation.
|
||||
hop_asn.py Per-hop ASN/AS-name/country/PTR enrichment via Team Cymru DNS, in-run deduped.
|
||||
captive_portal.py Phase-0 gate: three generate_204-style probes for a public-wifi login wall.
|
||||
|
|
@ -161,6 +162,10 @@ The AS prefix benchmark (`probes/prefix_benchmark.py`) gets the same treatment f
|
|||
|
||||
**The webhook payload is a flat JSON body, not shaped for any particular chat platform.** `alerting.build_payload()` returns `{tool, asn, at, previous, current, score, findings, host}` — Slack/Discord-specific formatting is a concern for whatever receives the POST, not for netsleuth. `post_webhook()` swallows every `httpx`/`OSError` failure and returns `False`; a dead webhook endpoint must never interrupt the monitoring loop it's supposed to be watching.
|
||||
|
||||
**ECMP detection diffs N independent traceroute runs, not one run's per-hop annotations.** BSD's `traceparse.py` already records a same-TTL-different-router signal as an `alt:` annotation, but that concept can't generalize: mtr's JSON aggregates cycles and discards per-cycle next-hop variance, `icmp_win`/`icmplib` send exactly one probe per TTL by construction, and the Linux system-traceroute parser doesn't model multi-IP hops even though the text can contain them. A finding that only ever fires on macOS and Linux-with-the-system-binary would be worse than not having it. `probes/ecmp.py::detect_ecmp()` instead takes N independently-run `TraceResult`s for the same target and diffs which IP appeared at each TTL across runs — this works identically regardless of which backend produced each trace, and the function itself never touches capability detection or backend choice at all.
|
||||
|
||||
**`--ecmp` is opt-in and capped, and never joins `--full`.** It multiplies traceroute traffic against the same target by `ecmp.runs` (default 3, capped at 5) — per the trace-concurrency reasoning already established (parallel traces to the *same* target inflate each other's latency worse than parallel traces to different targets), this is deliberately sequential per target and shares one fresh semaphore across `ecmp.max_targets` (capped at 3) targets, run only after the main latency/path/TLS fan-out has already finished.
|
||||
|
||||
**Python 3.14 is the floor, not 3.10.** The project moved off `>=3.10` when this batch of probes was added; none of the new code actually depends on a 3.11+-only construct (`ssl.SSLObject.start_tls()`, `asyncio.timeout()`, `datetime.UTC`) — the two-sequential-connection TLS design above was chosen independently of what the interpreter allows. The version floor moved because there was no longer a reason to hold it back, not because a specific new feature required it.
|
||||
|
||||
## Storage
|
||||
|
|
@ -179,4 +184,4 @@ Five sections back the L4-L7 probes, all opt-in at the CLI level and all with a
|
|||
|
||||
## Tests
|
||||
|
||||
`uv run pytest -q`. Roughly 825 tests covering the parsers, normalizers, verdict engine, DNSBL decoding, throughput math, ping statistics, jitter percentiles, serialization and the report diff. Glue is deliberately untested: live HTTP, real ICMP, Typer wiring, `psutil`/`ctypes` calls and the `--watch` timing loop. Traceroute fixtures live in `tests/fixtures/traceroute/{windows,linux,darwin}/` and include a cp866-encoded Russian Windows sample, because that encoding is exactly what breaks naive parsers. DoH response fixtures (RFC 8484 JSON form) live in `tests/fixtures/api/doh_*.json` alongside the other provider fixtures.
|
||||
`uv run pytest -q`. Roughly 837 tests covering the parsers, normalizers, verdict engine, DNSBL decoding, throughput math, ping statistics, jitter percentiles, serialization and the report diff. Glue is deliberately untested: live HTTP, real ICMP, Typer wiring, `psutil`/`ctypes` calls and the `--watch` timing loop. Traceroute fixtures live in `tests/fixtures/traceroute/{windows,linux,darwin}/` and include a cp866-encoded Russian Windows sample, because that encoding is exactly what breaks naive parsers. DoH response fixtures (RFC 8484 JSON form) live in `tests/fixtures/api/doh_*.json` alongside the other provider fixtures.
|
||||
|
|
|
|||
|
|
@ -39,6 +39,7 @@ src/netsleuth/
|
|||
bgp_path.py --path-diversity: поиск colo по CF-RAY, детект международной петли маршрута.
|
||||
prefix_benchmark.py --prefix-bench: пинг первого хоста ограниченной выборки анонсируемых префиксов AS.
|
||||
dpi_check.py --my-server: одиночная самопроверка TCP RST/фильтрации. Не сканер — см. Ключевые решения.
|
||||
ecmp.py --ecmp: чистый diff N прогонов трассировки для обнаружения балансировки нагрузки по хопам. Не зависит от бэкенда по дизайну.
|
||||
nat64.py Один AAAA-запрос по RFC 7050 (ipv4only.arpa) для детекта трансляции NAT64/464XLAT.
|
||||
hop_asn.py Обогащение хопов ASN/именем AS/страной/PTR через DNS Team Cymru, дедуп в рамках прогона.
|
||||
captive_portal.py Фаза 0: три generate_204-пробы на признаки логин-стены публичного wifi.
|
||||
|
|
@ -161,6 +162,10 @@ src/netsleuth/
|
|||
|
||||
**Полезная нагрузка вебхука — плоское JSON-тело, не заточенное ни под какую конкретную чат-платформу.** `alerting.build_payload()` возвращает `{tool, asn, at, previous, current, score, findings, host}` — форматирование под Slack/Discord — забота того, что принимает POST, а не netsleuth. `post_webhook()` глотает любой отказ `httpx`/`OSError` и возвращает `False`; мёртвый эндпоинт вебхука не должен прерывать цикл мониторинга, за которым он должен следить.
|
||||
|
||||
**Детект ECMP сравнивает N независимых прогонов трассировки, а не аннотации по хопам одного прогона.** `traceparse.py` для BSD уже фиксирует сигнал «разный роутер на том же TTL» как аннотацию `alt:`, но эта концепция не обобщается: JSON от mtr агрегирует циклы и теряет вариацию next-hop по циклам, `icmp_win`/`icmplib` по конструкции шлют ровно одну пробу на TTL, а Linux-парсер системного traceroute не моделирует много-IP хопы, даже если текст их содержит. Находка, которая срабатывает только на macOS и Linux с системным бинарником, хуже, чем её отсутствие. Вместо этого `probes/ecmp.py::detect_ecmp()` берёт N независимо прогнанных `TraceResult` для одной цели и сравнивает, какой IP появился на каждом TTL по прогонам — работает одинаково независимо от того, какой бэкенд дал каждую трассировку, и сама функция вообще не трогает определение возможностей или выбор бэкенда.
|
||||
|
||||
**`--ecmp` опционален и ограничен, и никогда не входит в `--full`.** Он умножает трафик трассировки к одной и той же цели на `ecmp.runs` (по умолчанию 3, потолок 5) — по уже установленной логике конкурентности трассировок (параллельные трассировки к *одной* цели искажают задержку друг друга хуже, чем параллельные трассировки к разным целям), это намеренно последовательно по каждой цели и делит один свежий семафор между `ecmp.max_targets` (потолок 3) целями, выполняясь только после того, как основной веер latency/path/TLS уже завершился.
|
||||
|
||||
**Python 3.14 — минимальная версия, не 3.10.** Проект ушёл с `>=3.10` одновременно с этим набором проб; ни один новый код на деле не зависит от конструкций только для 3.11+ (`ssl.SSLObject.start_tls()`, `asyncio.timeout()`, `datetime.UTC`) — схема TLS с двумя последовательными соединениями выше выбрана независимо от того, что позволяет интерпретатор. Планка версии сдвинулась потому, что не осталось причин её удерживать, а не потому, что это потребовалось для конкретной новой функции.
|
||||
|
||||
## Хранение
|
||||
|
|
@ -179,4 +184,4 @@ src/netsleuth/
|
|||
|
||||
## Тесты
|
||||
|
||||
`uv run pytest -q`. Около 825 тестов: парсеры, нормализаторы, движок вердиктов, декодирование DNSBL, математика пропускной способности, статистика пингов, перцентили джиттера, сериализация и сравнение отчётов. Клей намеренно не тестируется: живой HTTP, настоящий ICMP, обвязка Typer, вызовы `psutil`/`ctypes` и цикл `--watch`. Фикстуры трассировок лежат в `tests/fixtures/traceroute/{windows,linux,darwin}/` и включают образец русской Windows в cp866 — именно эта кодировка ломает наивные парсеры. Фикстуры DoH-ответов (формат RFC 8484 JSON) лежат в `tests/fixtures/api/doh_*.json` рядом с фикстурами других провайдеров.
|
||||
`uv run pytest -q`. Около 837 тестов: парсеры, нормализаторы, движок вердиктов, декодирование DNSBL, математика пропускной способности, статистика пингов, перцентили джиттера, сериализация и сравнение отчётов. Клей намеренно не тестируется: живой HTTP, настоящий ICMP, обвязка Typer, вызовы `psutil`/`ctypes` и цикл `--watch`. Фикстуры трассировок лежат в `tests/fixtures/traceroute/{windows,linux,darwin}/` и включают образец русской Windows в cp866 — именно эта кодировка ломает наивные парсеры. Фикстуры DoH-ответов (формат RFC 8484 JSON) лежат в `tests/fixtures/api/doh_*.json` рядом с фикстурами других провайдеров.
|
||||
|
|
|
|||
|
|
@ -68,6 +68,7 @@ None of this is required — netsleuth degrades gracefully without any of it —
|
|||
- **TLS handshake RTT** (`--tls`) — TCP RTT, TLS 1.3 handshake time and time-to-first-byte to reference services, flagging servers whose handshake is disproportionately slow relative to the network (CPU-bound TLS termination, not a network problem). Also captures each certificate's SHA-256 fingerprint, issuer and days until expiry, flags a validation failure, and checks against an optional pinned fingerprint (`tls.pinned_fingerprints` in `config.yaml`) to catch a corporate/ISP TLS-intercepting middlebox or a genuine MITM.
|
||||
- **DNS: system vs DoH** (`--dns-advanced`) — compares your system resolver's answers and latency against Cloudflare/Google/Quad9 DoH, and probes a bogus resolver IP to detect a transparent DNS proxy intercepting port 53.
|
||||
- **Path diversity / Anycast** (`--path-diversity`) — reads Cloudflare's `CF-RAY` edge code to find which PoP is actually serving your traffic, and flags an international routing loop when it's in a different country than you are.
|
||||
- **ECMP / multipath detection** (`--ecmp`) — traces a couple of reference hosts multiple times and diffs which next hop showed up at each TTL, catching per-hop load balancing that makes traceroutes look inconsistent, and flagging when one branch of a load-balanced pair is measurably worse than the other.
|
||||
- **AS prefix benchmark** (`--prefix-bench`) — pings the first host of a handful of your own AS's announced prefixes to find the lowest-latency PoP. Capped at 256 prefixes and off by default — this samples your own network, it does not scan it.
|
||||
- **DPI / port self-check** (`--my-server <host>`) — a single-host TCP probe across 6 fixed ports on **a server you own**, classifying TCP RST injection vs silent filtering vs a clean response. Not a port scanner: one host per run, a fixed short port list, and it never reads its target from your own egress IP or from BGP data — see [ARCHITECTURE.md](ARCHITECTURE.md) for why.
|
||||
|
||||
|
|
@ -90,6 +91,7 @@ None of this is required — netsleuth degrades gracefully without any of it —
|
|||
| `--dns-advanced` | Compare system DNS vs DoH and probe for a transparent DNS proxy. Included in `--full` |
|
||||
| `--path-diversity` | Compare client geo against the Cloudflare edge PoP actually serving traffic. Included in `--full` |
|
||||
| `--prefix-bench` | Ping the first host of a handful of your own AS's announced prefixes to find the lowest-latency PoP. **Not** included in `--full` — opt in explicitly |
|
||||
| `--ecmp` | Trace a couple of reference hosts multiple times (`ecmp.runs`, capped at 5) to detect per-hop load balancing and measure how unequal the branches are. **Not** included in `--full` — it multiplies traceroute traffic to the same targets, opt in explicitly |
|
||||
| `--my-server <host>` | Check a server **you own** for TCP port blocking / RST injection. Single host only — rejects CIDR input outright. **Not** included in `--full` — opt in explicitly |
|
||||
| `--ndt7` | Opt in to M-Lab NDT7 (publishes your measurement, including your IP, as CC0 open data). On an interactive terminal it prints the consent notice and asks first; on a non-interactive one it proceeds without asking |
|
||||
| `--tcp-trace` | Opt in to a scapy TCP-SYN traceroute through ICMP-filtering middleboxes (needs the `tcptrace` extra plus Npcap on Windows or root on Unix). Falls through to the normal cascade when it cannot run |
|
||||
|
|
|
|||
|
|
@ -68,6 +68,7 @@ netsleuth
|
|||
- **TLS handshake RTT** (`--tls`) — TCP RTT, время TLS 1.3-рукопожатия и время до первого байта к эталонным сервисам; отдельно отмечает сервер, чьё рукопожатие непропорционально медленнее сети (упирается в CPU TLS-терминатора, а не в сеть). Также фиксирует SHA-256-отпечаток сертификата, издателя и дни до истечения, отмечает провал валидации и сверяет с опциональным закреплённым отпечатком (`tls.pinned_fingerprints` в `config.yaml`), чтобы поймать корпоративный/провайдерский TLS-перехватывающий middlebox или настоящий MITM.
|
||||
- **DNS: системный vs DoH** (`--dns-advanced`) — сравнивает ответы и задержку системного резолвера с Cloudflare/Google/Quad9 DoH, а также проверяет фиктивный IP резолвера на признаки прозрачного DNS-прокси, перехватывающего 53 порт.
|
||||
- **Диверсификация маршрута / Anycast** (`--path-diversity`) — по заголовку Cloudflare `CF-RAY` определяет, какая точка (PoP) реально обслуживает трафик, и отмечает международную петлю маршрута, если она в другой стране, чем клиент.
|
||||
- **Детект ECMP / нескольких маршрутов** (`--ecmp`) — трассирует пару эталонных хостов несколько раз и сравнивает, какой следующий хоп появлялся на каждом TTL, ловя балансировку нагрузки по хопам, из-за которой трассировки выглядят непоследовательными, и отмечая, если одна из веток балансировки заметно хуже другой.
|
||||
- **Бенчмарк префиксов AS** (`--prefix-bench`) — пингует первый хост нескольких анонсируемых префиксов вашей собственной AS, чтобы найти точку с наименьшей задержкой. Ограничено 256 префиксами и выключено по умолчанию — это выборочная проба своей же сети, а не её сканирование.
|
||||
- **Самопроверка DPI / портов** (`--my-server <host>`) — одиночная TCP-проба по 6 фиксированным портам **сервера, которым вы владеете**, с классификацией инъекции TCP RST, тихой фильтрации или чистого ответа. Это не портсканер: один хост за запуск, фиксированный короткий список портов, и цель никогда не берётся из вашего собственного внешнего IP или из данных BGP — почему именно так, см. [ARCHITECTURE.ru.md](ARCHITECTURE.ru.md).
|
||||
|
||||
|
|
@ -90,6 +91,7 @@ netsleuth
|
|||
| `--dns-advanced` | Сравнить системный DNS с DoH и проверить прозрачный DNS-прокси. Входит в `--full` |
|
||||
| `--path-diversity` | Сравнить гео клиента с реальной точкой Cloudflare, обслуживающей трафик. Входит в `--full` |
|
||||
| `--prefix-bench` | Пропинговать первый хост нескольких анонсируемых префиксов своей AS в поисках точки с наименьшей задержкой. **Не** входит в `--full` — включается явно |
|
||||
| `--ecmp` | Трассировать пару эталонных хостов несколько раз (`ecmp.runs`, потолок 5) для детекта балансировки нагрузки по хопам и замера, насколько неравноценны ветки. **Не** входит в `--full` — умножает трафик трассировки к тем же целям, включается явно |
|
||||
| `--my-server <host>` | Проверить сервер, **которым вы владеете**, на блокировку портов / инъекцию RST. Только один хост — CIDR отклоняется сразу. **Не** входит в `--full` — включается явно |
|
||||
| `--ndt7` | Включить M-Lab NDT7 (замер публикуется как открытые данные CC0, включая ваш IP). В интерактивном терминале сначала печатает уведомление о согласии и спрашивает подтверждение; в неинтерактивном — продолжает без вопроса |
|
||||
| `--tcp-trace` | Включить TCP-SYN-трассировку через scapy сквозь middlebox'ы, режущие ICMP (нужны extras `tcptrace` и Npcap на Windows или root на Unix). Если не может запуститься — откатывается на обычный каскад |
|
||||
|
|
|
|||
|
|
@ -124,6 +124,13 @@ path_diversity:
|
|||
# Runs first, before any other probe. Checks a handful of known generate_204-style
|
||||
# endpoints for a redirect/tampered response, the classic signature of a public-wifi
|
||||
# login wall. check_urls is capped at 4 in code.
|
||||
# Opt-in via --ecmp. Traces the first few reference hosts multiple times to spot
|
||||
# per-TTL load balancing (different next hops across runs). runs/max_targets are
|
||||
# capped in code: this diffs a handful of runs, not a path-sampling campaign.
|
||||
ecmp:
|
||||
runs: 3
|
||||
max_targets: 2
|
||||
|
||||
captive_portal:
|
||||
enabled: true
|
||||
check_urls:
|
||||
|
|
|
|||
|
|
@ -28,6 +28,7 @@ from netsleuth.interpret import (
|
|||
dns_advanced_findings,
|
||||
dpi_findings,
|
||||
dual_stack_findings,
|
||||
ecmp_findings,
|
||||
gather_vpn_signals,
|
||||
latency_findings,
|
||||
path_asn_findings,
|
||||
|
|
@ -89,6 +90,7 @@ class Options:
|
|||
dns_advanced: bool = False
|
||||
path_diversity: bool = False
|
||||
prefix_bench: bool = False
|
||||
ecmp: bool = False
|
||||
dpi_target: str | None = None
|
||||
formats: frozenset[str] = frozenset({"md"})
|
||||
interface: str | None = None
|
||||
|
|
@ -397,6 +399,35 @@ async def _dpi_section(settings: Settings, options: Options) -> ModuleResult:
|
|||
return ModuleResult(name="dpi_check", status="ok", data=result)
|
||||
|
||||
|
||||
async def _ecmp_section(caps, settings: Settings, options: Options, cycles: int) -> ModuleResult:
|
||||
if not options.ecmp:
|
||||
return ModuleResult(name="ecmp", status="skipped", warnings=["ecmp probe skipped: not requested"])
|
||||
from netsleuth.probes.ecmp import detect_ecmp
|
||||
|
||||
cfg = settings.ecmp
|
||||
targets = [(h.label, h.host) for h in settings.probing.reference_hosts[: cfg.max_targets]]
|
||||
semaphore = asyncio.Semaphore(settings.probing.trace_concurrency)
|
||||
reports = []
|
||||
for _label, host in targets:
|
||||
runs = await asyncio.gather(
|
||||
*(
|
||||
traceroute(
|
||||
host,
|
||||
caps,
|
||||
max_hops=settings.probing.max_hops,
|
||||
cycles=cycles,
|
||||
timeout=settings.timeouts.subprocess_seconds,
|
||||
semaphore=semaphore,
|
||||
tcp_trace=options.tcp_trace,
|
||||
source_ip=options.bind.ipv4 if options.bind else None,
|
||||
)
|
||||
for _ in range(cfg.runs)
|
||||
)
|
||||
)
|
||||
reports.append(detect_ecmp(list(runs)))
|
||||
return ModuleResult(name="ecmp", status="ok", data=reports)
|
||||
|
||||
|
||||
async def _traces(hosts, caps, settings: Settings, cycles: int, options: Options):
|
||||
semaphore = asyncio.Semaphore(settings.probing.trace_concurrency)
|
||||
return list(
|
||||
|
|
@ -608,6 +639,9 @@ async def diagnose(settings: Settings, options: Options) -> tuple[dict, list[Pat
|
|||
modules["dpi_check"] = await run_module(
|
||||
"dpi_check", _dpi_section(settings, options), timeout=timeouts.module_seconds
|
||||
)
|
||||
modules["ecmp"] = await run_module(
|
||||
"ecmp", _ecmp_section(caps, settings, options, cycles), timeout=timeouts.subprocess_seconds
|
||||
)
|
||||
|
||||
# Phase 3c — per-hop ASN/AS-name/country enrichment. Mutates the trace hops
|
||||
# in place; a lookup failure must never downgrade the trace itself, so only
|
||||
|
|
@ -661,6 +695,7 @@ async def diagnose(settings: Settings, options: Options) -> tuple[dict, list[Pat
|
|||
+ cgnat_findings(local, traces)
|
||||
+ dual_stack_findings(bundle.get("nat64_prefix"), local)
|
||||
+ captive_portal_findings(modules["captive_portal"].data or CaptivePortal())
|
||||
+ [f for report in (modules["ecmp"].data or []) for f in ecmp_findings(report)]
|
||||
)
|
||||
|
||||
# Phase 6 — export
|
||||
|
|
@ -796,6 +831,11 @@ def run(
|
|||
"--prefix-bench",
|
||||
help="Ping the first host of a handful of your own AS's announced prefixes to find the lowest-latency PoP.",
|
||||
),
|
||||
ecmp: bool = typer.Option(
|
||||
False,
|
||||
"--ecmp",
|
||||
help="Trace a handful of reference hosts multiple times to detect per-hop load balancing (ECMP).",
|
||||
),
|
||||
my_server: Optional[str] = typer.Option(
|
||||
None,
|
||||
"--my-server",
|
||||
|
|
@ -867,6 +907,7 @@ def run(
|
|||
dns_advanced=dns_advanced or full,
|
||||
path_diversity=path_diversity or full,
|
||||
prefix_bench=prefix_bench,
|
||||
ecmp=ecmp,
|
||||
dpi_target=my_server,
|
||||
formats=formats,
|
||||
webhook=webhook,
|
||||
|
|
|
|||
|
|
@ -222,6 +222,25 @@ class DnsAdvancedConfig(BaseModel):
|
|||
bogus_probe_name: str = "cloudflare.com"
|
||||
|
||||
|
||||
class EcmpConfig(BaseModel):
|
||||
runs: int = 3
|
||||
max_targets: int = 2
|
||||
|
||||
@field_validator("runs")
|
||||
@classmethod
|
||||
def _cap_runs(cls, value: int) -> int:
|
||||
if value > 5:
|
||||
raise ValueError("ecmp.runs may not exceed 5 (this diffs a handful of runs, not a path-sampling campaign)")
|
||||
return value
|
||||
|
||||
@field_validator("max_targets")
|
||||
@classmethod
|
||||
def _cap_max_targets(cls, value: int) -> int:
|
||||
if value > 3:
|
||||
raise ValueError("ecmp.max_targets may not exceed 3")
|
||||
return value
|
||||
|
||||
|
||||
class CaptivePortalConfig(BaseModel):
|
||||
enabled: bool = True
|
||||
check_urls: list[str] = Field(
|
||||
|
|
@ -348,6 +367,7 @@ class Settings(BaseSettings):
|
|||
dns_advanced: DnsAdvancedConfig = DnsAdvancedConfig()
|
||||
path_diversity: PathDiversityConfig = PathDiversityConfig()
|
||||
captive_portal: CaptivePortalConfig = CaptivePortalConfig()
|
||||
ecmp: EcmpConfig = EcmpConfig()
|
||||
|
||||
ipinfo_token: SecretStr | None = Field(default=None, alias="IPINFO_TOKEN")
|
||||
peeringdb_api_key: SecretStr | None = Field(default=None, alias="PEERINGDB_API_KEY")
|
||||
|
|
|
|||
|
|
@ -22,6 +22,7 @@ SECTION_ORDER = (
|
|||
"latency",
|
||||
"tls",
|
||||
"path",
|
||||
"ecmp",
|
||||
"path_diversity",
|
||||
"prefix_benchmark",
|
||||
"dpi_check",
|
||||
|
|
@ -127,6 +128,7 @@ _WARNING_RU = {
|
|||
"speedtest skipped in target mode": "speedtest пропущен в режиме цели",
|
||||
"no throughput measured": "пропускная способность не измерена",
|
||||
"captive portal check disabled in config": "проверка captive portal отключена в конфиге",
|
||||
"ecmp probe skipped: not requested": "проверка ECMP пропущена: не запрошена",
|
||||
}
|
||||
|
||||
|
||||
|
|
@ -255,6 +257,7 @@ _SECTION_TITLES = {
|
|||
"latency": "## Latency",
|
||||
"tls": "## TLS handshake",
|
||||
"path": "## Path",
|
||||
"ecmp": "## ECMP / multipath",
|
||||
"path_diversity": "## Path diversity (Anycast)",
|
||||
"prefix_benchmark": "## AS prefix benchmark",
|
||||
"dpi_check": "## DPI / port check",
|
||||
|
|
@ -272,6 +275,7 @@ _SECTION_TITLES_RU = {
|
|||
"latency": "## Задержки",
|
||||
"tls": "## TLS-рукопожатие",
|
||||
"path": "## Маршрут",
|
||||
"ecmp": "## ECMP / несколько маршрутов",
|
||||
"path_diversity": "## Диверсификация маршрута (Anycast)",
|
||||
"prefix_benchmark": "## Бенчмарк префиксов AS",
|
||||
"dpi_check": "## Проверка DPI / портов",
|
||||
|
|
@ -658,6 +662,33 @@ def _path(report: dict, lang: str = "en") -> list[str]:
|
|||
return lines
|
||||
|
||||
|
||||
def _ecmp(report: dict, lang: str = "en") -> list[str]:
|
||||
title = _section_title("ecmp", lang)
|
||||
note = unavailable(report, "ecmp", lang)
|
||||
if note:
|
||||
return [title, "", note, ""]
|
||||
reports = _data(report, "ecmp") or []
|
||||
lines = [title, ""]
|
||||
for r in reports:
|
||||
target = r.get("target") or "?"
|
||||
if not r.get("divergent_ttls"):
|
||||
lines += [f"- {target}: {_t(lang, 'single path across', 'единый маршрут во всех')} {r.get('runs', 0)} {_t(lang, 'runs', 'прогонах')}"]
|
||||
continue
|
||||
note_text = (r.get("note_ru") or r.get("note") or "") if lang == "ru" else (r.get("note") or "")
|
||||
lines += [f"### {target}", "", note_text, ""]
|
||||
lines += _table(
|
||||
[_t(lang, "TTL", "TTL"), _t(lang, "Next hops", "Следующие хопы"), _t(lang, "RTT spread (ms)", "Разброс RTT (мс)")],
|
||||
[
|
||||
[str(h.get("ttl", "")), ", ".join(h.get("ips") or []), _num(h.get("rtt_spread_ms"))]
|
||||
for h in r.get("hops") or []
|
||||
if len(h.get("ips") or []) > 1
|
||||
],
|
||||
lang,
|
||||
)
|
||||
lines.append("")
|
||||
return lines + [""]
|
||||
|
||||
|
||||
def _speed(report: dict, lang: str = "en") -> list[str]:
|
||||
title = _section_title("speed", lang)
|
||||
note = unavailable(report, "speed", lang)
|
||||
|
|
@ -961,6 +992,7 @@ def render_markdown(
|
|||
lines += _latency(report, lang)
|
||||
lines += _tls(report, lang)
|
||||
lines += _path(report, lang)
|
||||
lines += _ecmp(report, lang)
|
||||
lines += _path_diversity(report, lang)
|
||||
lines += _prefix_benchmark(report, lang)
|
||||
lines += _dpi_check(report, lang)
|
||||
|
|
|
|||
|
|
@ -8,6 +8,7 @@ from netsleuth.models import (
|
|||
DnsAdvanced,
|
||||
DnsLeak,
|
||||
DpiCheckResult,
|
||||
EcmpReport,
|
||||
Finding,
|
||||
IpGeo,
|
||||
LocalNet,
|
||||
|
|
@ -317,6 +318,50 @@ def dual_stack_findings(nat64_prefix: str | None, local: LocalNet) -> list[Findi
|
|||
]
|
||||
|
||||
|
||||
_ECMP_ASYMMETRIC_MS = 20.0
|
||||
|
||||
|
||||
def ecmp_findings(report: EcmpReport) -> list[Finding]:
|
||||
if not report.divergent_ttls:
|
||||
return []
|
||||
findings: list[Finding] = [
|
||||
Finding(
|
||||
id=f"path.ecmp.{report.target}",
|
||||
severity="info",
|
||||
title=f"Multiple paths observed to {report.target}",
|
||||
detail=report.note,
|
||||
metric="divergent_ttls",
|
||||
value=", ".join(str(t) for t in report.divergent_ttls),
|
||||
advice="Normal on a load-balanced network; it just means repeated traceroutes to the same "
|
||||
"target can legitimately show different hops at the same distance.",
|
||||
title_ru=f"Обнаружено несколько маршрутов до {report.target}",
|
||||
detail_ru=report.note_ru or report.note,
|
||||
advice_ru="Нормально для сети с балансировкой нагрузки; означает лишь, что повторные "
|
||||
"трассировки до одной цели могут честно показывать разные хопы на одном расстоянии.",
|
||||
)
|
||||
]
|
||||
worst = max(report.hops, key=lambda h: h.rtt_spread_ms or 0.0)
|
||||
if (worst.rtt_spread_ms or 0.0) > _ECMP_ASYMMETRIC_MS:
|
||||
findings.append(
|
||||
Finding(
|
||||
id=f"path.ecmp_asymmetric.{report.target}",
|
||||
severity="warn",
|
||||
title=f"Load-balanced paths to {report.target} are not equal",
|
||||
detail=f"At TTL {worst.ttl}, next hops {worst.ips} differ by {worst.rtt_spread_ms} ms.",
|
||||
metric="rtt_spread_ms",
|
||||
value=worst.rtt_spread_ms,
|
||||
threshold=_ECMP_ASYMMETRIC_MS,
|
||||
advice="One branch of a load-balanced pair is degraded; this shows up as intermittent "
|
||||
"slowness that a single traceroute run would never catch.",
|
||||
title_ru=f"Балансируемые маршруты до {report.target} неравноценны",
|
||||
detail_ru=f"На TTL {worst.ttl} следующие хопы {worst.ips} отличаются на {worst.rtt_spread_ms} мс.",
|
||||
advice_ru="Одна из веток балансировки нагрузки деградировала; это проявляется как "
|
||||
"прерывистая медлительность, которую один прогон трассировки никогда не поймает.",
|
||||
)
|
||||
)
|
||||
return findings
|
||||
|
||||
|
||||
def path_asn_findings(trace: TraceResult, client_country: str | None) -> list[Finding]:
|
||||
if not client_country or len(trace.hops) < 2:
|
||||
return []
|
||||
|
|
|
|||
|
|
@ -504,6 +504,24 @@ class CaptivePortal:
|
|||
raise ValueError(f"unknown CaptivePortal verdict: {self.verdict!r}")
|
||||
|
||||
|
||||
@dataclass
|
||||
class EcmpHop:
|
||||
ttl: int
|
||||
ips: list[str] = field(default_factory=list)
|
||||
asns: list[str] = field(default_factory=list)
|
||||
rtt_spread_ms: float | None = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class EcmpReport:
|
||||
target: str | None = None
|
||||
runs: int = 0
|
||||
hops: list[EcmpHop] = field(default_factory=list)
|
||||
divergent_ttls: list[int] = field(default_factory=list)
|
||||
note: str = ""
|
||||
note_ru: str | None = None
|
||||
|
||||
|
||||
def to_jsonable(obj: Any) -> Any:
|
||||
if obj is None or isinstance(obj, (str, bool, int)):
|
||||
return obj
|
||||
|
|
|
|||
43
src/netsleuth/probes/ecmp.py
Normal file
43
src/netsleuth/probes/ecmp.py
Normal file
|
|
@ -0,0 +1,43 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from netsleuth.models import EcmpHop, EcmpReport, TraceHop, TraceResult
|
||||
|
||||
|
||||
def detect_ecmp(runs: list[TraceResult]) -> EcmpReport:
|
||||
if not runs:
|
||||
return EcmpReport()
|
||||
target = runs[0].target
|
||||
by_ttl: dict[int, list[TraceHop]] = {}
|
||||
for trace in runs:
|
||||
for hop in trace.hops:
|
||||
if hop.ip:
|
||||
by_ttl.setdefault(hop.ttl, []).append(hop)
|
||||
|
||||
hops: list[EcmpHop] = []
|
||||
divergent: list[int] = []
|
||||
for ttl in sorted(by_ttl):
|
||||
group = by_ttl[ttl]
|
||||
ips: list[str] = []
|
||||
avg_by_ip: dict[str, list[float]] = {}
|
||||
asns: list[str] = []
|
||||
for hop in group:
|
||||
if hop.ip not in ips:
|
||||
ips.append(hop.ip)
|
||||
if hop.avg_ms is not None:
|
||||
avg_by_ip.setdefault(hop.ip, []).append(hop.avg_ms)
|
||||
if hop.asn and hop.asn not in asns:
|
||||
asns.append(hop.asn)
|
||||
spread: float | None = None
|
||||
if len(ips) > 1:
|
||||
divergent.append(ttl)
|
||||
representative = [sum(v) / len(v) for v in avg_by_ip.values() if v]
|
||||
if len(representative) > 1:
|
||||
spread = round(max(representative) - min(representative), 3)
|
||||
hops.append(EcmpHop(ttl=ttl, ips=ips, asns=asns, rtt_spread_ms=spread))
|
||||
|
||||
note = note_ru = ""
|
||||
if divergent:
|
||||
ttl_list = ", ".join(str(t) for t in divergent)
|
||||
note = f"Multiple next hops observed at TTL {ttl_list}."
|
||||
note_ru = f"На TTL {ttl_list} обнаружены разные следующие хопы."
|
||||
return EcmpReport(target=target, runs=len(runs), hops=hops, divergent_ttls=divergent, note=note, note_ru=note_ru)
|
||||
|
|
@ -252,6 +252,20 @@ def test_watch_webhook_on_rejects_an_unknown_entry(tmp_path):
|
|||
load_settings(config_path=path, env_file=tmp_path / "nope.env")
|
||||
|
||||
|
||||
def test_ecmp_runs_are_capped(tmp_path):
|
||||
path = tmp_path / "config.yaml"
|
||||
path.write_text("ecmp:\n runs: 10\n", encoding="utf-8")
|
||||
with pytest.raises(ValidationError, match="not exceed 5"):
|
||||
load_settings(config_path=path, env_file=tmp_path / "nope.env")
|
||||
|
||||
|
||||
def test_ecmp_max_targets_are_capped(tmp_path):
|
||||
path = tmp_path / "config.yaml"
|
||||
path.write_text("ecmp:\n max_targets: 10\n", encoding="utf-8")
|
||||
with pytest.raises(ValidationError, match="not exceed 3"):
|
||||
load_settings(config_path=path, env_file=tmp_path / "nope.env")
|
||||
|
||||
|
||||
def test_history_trend_default_reports_is_capped(tmp_path):
|
||||
path = tmp_path / "config.yaml"
|
||||
path.write_text("history:\n trend_default_reports: 500\n", encoding="utf-8")
|
||||
|
|
|
|||
104
tests/test_ecmp.py
Normal file
104
tests/test_ecmp.py
Normal file
|
|
@ -0,0 +1,104 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from netsleuth.interpret import ecmp_findings
|
||||
from netsleuth.models import TraceHop, TraceResult
|
||||
from netsleuth.probes.ecmp import detect_ecmp
|
||||
|
||||
|
||||
def hop(ttl: int, ip: str | None, avg_ms: float | None = 10.0, asn: str | None = None) -> TraceHop:
|
||||
return TraceHop(ttl=ttl, ip=ip, avg_ms=avg_ms, asn=asn)
|
||||
|
||||
|
||||
def trace(*hops: TraceHop, target: str = "1.1.1.1") -> TraceResult:
|
||||
return TraceResult(target=target, hops=list(hops))
|
||||
|
||||
|
||||
def test_detect_ecmp_on_a_single_run_has_no_divergence():
|
||||
report = detect_ecmp([trace(hop(1, "192.168.1.1"), hop(2, "1.1.1.1"))])
|
||||
assert report.divergent_ttls == []
|
||||
assert report.runs == 1
|
||||
|
||||
|
||||
def test_detect_ecmp_on_three_identical_runs_has_no_divergence():
|
||||
runs = [trace(hop(1, "192.168.1.1"), hop(2, "1.1.1.1")) for _ in range(3)]
|
||||
report = detect_ecmp(runs)
|
||||
assert report.divergent_ttls == []
|
||||
assert report.runs == 3
|
||||
|
||||
|
||||
def test_detect_ecmp_flags_a_ttl_with_different_next_hops():
|
||||
runs = [
|
||||
trace(hop(1, "192.168.1.1"), hop(2, "10.0.0.1")),
|
||||
trace(hop(1, "192.168.1.1"), hop(2, "10.0.0.2")),
|
||||
trace(hop(1, "192.168.1.1"), hop(2, "10.0.0.1")),
|
||||
]
|
||||
report = detect_ecmp(runs)
|
||||
assert report.divergent_ttls == [2]
|
||||
hop2 = next(h for h in report.hops if h.ttl == 2)
|
||||
assert set(hop2.ips) == {"10.0.0.1", "10.0.0.2"}
|
||||
|
||||
|
||||
def test_detect_ecmp_computes_a_large_rtt_gap_between_branches():
|
||||
runs = [
|
||||
trace(hop(1, "10.0.0.1", avg_ms=10.0)),
|
||||
trace(hop(1, "10.0.0.2", avg_ms=80.0)),
|
||||
]
|
||||
report = detect_ecmp(runs)
|
||||
hop1 = report.hops[0]
|
||||
assert hop1.rtt_spread_ms == 70.0
|
||||
|
||||
|
||||
def test_detect_ecmp_handles_runs_of_differing_length():
|
||||
runs = [
|
||||
trace(hop(1, "10.0.0.1"), hop(2, "1.1.1.1")),
|
||||
trace(hop(1, "10.0.0.1")),
|
||||
]
|
||||
report = detect_ecmp(runs)
|
||||
assert report.divergent_ttls == []
|
||||
assert {h.ttl for h in report.hops} == {1, 2}
|
||||
|
||||
|
||||
def test_detect_ecmp_excludes_none_ip_hops():
|
||||
runs = [
|
||||
trace(hop(1, "10.0.0.1")),
|
||||
trace(hop(1, None)),
|
||||
]
|
||||
report = detect_ecmp(runs)
|
||||
assert report.hops[0].ips == ["10.0.0.1"]
|
||||
assert report.divergent_ttls == []
|
||||
|
||||
|
||||
def test_detect_ecmp_on_no_runs():
|
||||
report = detect_ecmp([])
|
||||
assert report.hops == []
|
||||
assert report.runs == 0
|
||||
|
||||
|
||||
def test_ecmp_findings_silent_when_no_divergence():
|
||||
report = detect_ecmp([trace(hop(1, "10.0.0.1"))])
|
||||
assert ecmp_findings(report) == []
|
||||
|
||||
|
||||
def test_ecmp_findings_info_when_divergent_but_close_in_rtt():
|
||||
runs = [
|
||||
trace(hop(1, "10.0.0.1", avg_ms=10.0)),
|
||||
trace(hop(1, "10.0.0.2", avg_ms=12.0)),
|
||||
]
|
||||
report = detect_ecmp(runs)
|
||||
findings = ecmp_findings(report)
|
||||
assert [f.id for f in findings] == [f"path.ecmp.{report.target}"]
|
||||
assert findings[0].severity == "info"
|
||||
|
||||
|
||||
def test_ecmp_findings_warns_on_asymmetric_branches():
|
||||
runs = [
|
||||
trace(hop(1, "10.0.0.1", avg_ms=10.0)),
|
||||
trace(hop(1, "10.0.0.2", avg_ms=80.0)),
|
||||
]
|
||||
report = detect_ecmp(runs)
|
||||
findings = ecmp_findings(report)
|
||||
ids = [f.id for f in findings]
|
||||
assert f"path.ecmp.{report.target}" in ids
|
||||
assert f"path.ecmp_asymmetric.{report.target}" in ids
|
||||
asymmetric = [f for f in findings if f.id.startswith("path.ecmp_asymmetric")][0]
|
||||
assert asymmetric.severity == "warn"
|
||||
|
|
@ -91,6 +91,7 @@ def test_missing_modules_are_rendered_as_skipped_sections_not_dropped():
|
|||
"latency",
|
||||
"tls",
|
||||
"path",
|
||||
"ecmp",
|
||||
"path_diversity",
|
||||
"prefix_benchmark",
|
||||
"dpi_check",
|
||||
|
|
@ -600,6 +601,7 @@ def test_markdown_has_every_section_from_the_spec_in_order():
|
|||
"## Latency",
|
||||
"## TLS handshake",
|
||||
"## Path",
|
||||
"## ECMP / multipath",
|
||||
"## Path diversity (Anycast)",
|
||||
"## AS prefix benchmark",
|
||||
"## DPI / port check",
|
||||
|
|
@ -802,7 +804,7 @@ def test_diagnostics_translates_module_warnings_into_russian():
|
|||
def test_an_all_modules_failed_run_still_renders_every_section():
|
||||
text = render_markdown(dead_report())
|
||||
headings = [line for line in text.splitlines() if line.startswith("## ")]
|
||||
assert len(headings) == 16
|
||||
assert len(headings) == 17
|
||||
assert text.count("Not available") == len(SECTION_ORDER)
|
||||
|
||||
|
||||
|
|
@ -908,6 +910,7 @@ def test_render_markdown_default_call_is_byte_identical_to_before_the_lang_featu
|
|||
"## Latency",
|
||||
"## TLS handshake",
|
||||
"## Path",
|
||||
"## ECMP / multipath",
|
||||
"## Path diversity (Anycast)",
|
||||
"## AS prefix benchmark",
|
||||
"## DPI / port check",
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue