ntopng 5.6
Breakthroughs
Add XL license
Add support Rocky9
Add support to Kafka
Increased max num of exporters
Introduce nTap support
Introduce support to ClickHouse Cluster
Rework Historical Chart Page
Rework pages using VueJS and moving towards responsive client
Improvements
Handle allowed networks for unprivileged users
Improve multitenancy support
Improve thread names
Improve mac formatting
Improve top host sites adding reset method
Improve pcap upload
Improve ports formatting
Improve handling for Cisco NBAR collection
Improve source style
Improve Linux OS detection
Improve Engaged Time Report in Chart
Improve passive DNS hosty resolution
Improve alerts reports
Improve OPNsense installation instruction
Improve host report
Improve support to NDPI_TCP_ISSUES flow risk
Improve layout
Improve ICMP flow handling
Lowered memory consumption due to alert score
Rework pro code directories
Rework lua code
Rework flow aggregation
Rework capabilities support
Socket code cleanup
Use API to build interface report
Update rrd calculations
Update JP localization (courtesy of Yoshihiro Ishikawa)
Changes
Add logo to package
Add missing deps
Add link to host
Add options to send report by email
Add Report class and example
Add internal server error on health/interfaces doc api
Add support for external (REST) host alerts
Add various help and parameters
Add script to create a pdf report from historical API data
Add NXLOG/Active Directory documentation
Add reload button in various pages
Add third party resources
Add flow exporter ips to observation points
Add support for the python API documentation
Add forced offline variable to mantain the --offline option
Add support for Lua host engaged alerts using timeout
Add observation points ts
Add HTTP server in flow details
Add token-based authentication https://www.ntop.org/guides/ntopng/advanced_features/authentication.html?highlight=token#token-based-authentication
Add Flow Risk (Bitmap) Filter in alerts
Add make targets for pip package Updated package classes
Add L7 information in flow object adding
Add CodeQL workflow for GitHub code scanning
Add modal-download-file component and add export timeseries png picture button
Add critical and emergency status to alerts
Add oneway TCP flows counters
Add support for nDPI network handling in flows
Add -n 4 for name resolution
Add IMAP/POP stats
Add Stratosphere Labs Blacklist support
Add support d3v7
Add Requires for RH9 (redhat-lsb-core is deprecated)
Add interfaces stats api and refactor the others health api
Add support to application protocol and master protocol
Add CIDR support in Historical Flows
Add new Aggregated Flows page
Add new Alerts Analysis page
Add support for estimating the number of TCP contacted servers with no reply
Add new Ports Analysis page
Add detection of periodic flows and exported it as flow risk in both flows and alerts
Add REST API to get DB columns and info
Add ability to query alerts from Python
Add Zoom streams handling
Add various checks
Add IP-in-IP decapsulation
Add Host Rules page (possiblity to trigger alerts based on timeseries)
Add the ability to analyze a pcap without creating a new interface
Add Windows timezone handling
Change table definition
Cleanup file names
Disabled host serialization
Enlarged the number of local networks to 1024
Increased upload size to 25 MB
Implement custom script check
Implement support of host filtering with TX traffic sent
Implement unresponsive peers host report
Implement count of incoming tx peers with TCP flows unanswered
Move ts business logic in ts_rest_utils.lua
Patch for handling nicely clock drift at startup
Remove obsolete autogen commands On Linux stay with g++ unless asnitizer is used
Remove REST API v0 (discontinued since ntopng 4.2)
Remove no more used severity
Refactor range-picker query_presets
Rework host packets page and removed dscp page
Rework host ports implementation
Rework Historical class
Rework OPNsense plugin package build
Self test fixes and improvements
Update documentation
Update REST API
Update bootstrap table css
Update various pages to vuejs
Update counter scaling (no gauge)
Update response in service disabled case
nEdge
Add support to multi LAN and fixes DHCP service error
Add VLAN and multi WAN support to nedge
Add routing_policy to nedge configuration callback
Fix netplan configuration error
Update vlan trunk doc
Fix
Df columns error management, table export formatted with % and column reordering now working
Fix missing openssl dependency from MacOS
Fix clang
Fix host sankey minor issues
Fix hyperlinks to historical charts not working
Fix hyperlinks not working correctly
Fix Regex escape
Fix application name resolution on aggregated views
Fix RRD driver for step calaculation
Fix visual bugs with master and app proto
Fix various interface page minor bugs
Fix shortened labels
Fix default sort not working
Fix influxdb retention not updated
Fix name and size of charts
Fix vlan label not mapped
Fix for FreeBSD configure
Fix ip resolution not updating the name
Fix discrepancy in Traffic Calculation (Interface Chart)
Fix measurement units not uniform
Fix crash swap
Fix bug that reported wrong DNS information
Fix build process with opnsense/plugins
Fix validators regexps
Fix ICMP emtropy report Improved HTTP flows report
Fix Telegram Reported alerts contain HTML
Fix multi-series Charts are Unreadable in Dark Mode
Fix invalid reverse host resolution that caused hosts to be labelled with wrong symbolic name
Fix delete obsoleted code from page-stats
Fix for circular dependency js
Fix overlay not working
Fix due to changes to nDPI ALPN handling
Fix CSS Inconsistency Across Browsers
Fix Deep copy also for array of objects
Fix missing modules
Fix NAT handling with nprobe
Fix initialization crash
Removed multiple load from tables
ZMQ encryption key is now reported in hex to avoid escape problems
ntopng 5.2 (February 2022)
Breakthroughs
* New ClickHouse support for storing historical data, replacing nIndex support (data migration available)
* Advanced Historical Flow Explorer, with the ability to define custom queries using JSON-based configurations
* New Historical Data Analysis page (including Score, Applications, Alerts, AS analysis), with the ability to define custom reports with charts
* Enhanced drill down from charts and historical flow data and alerts to PCAP data
* nEdge support for Ubuntu 20
* Enhanced support for Observation Points
Improvements
* Improve CPU utilization and memory footprint
* Improve historical data retention management for flows and timeseries
* Improve periodic activities handling, with support for strict and relaxed (delayed) tasks
* Improve filtering and analysis of the historical flows
* Improve alert explorer and filtering
* Improve Enterprise dashboard look and feel
* Improve the speedtest support and servers selection
* Improve support for ping and continuous ping (ICMP) for active monitoring
* Improve flow-direction handling
* Improve localization (including DE and IT translations)
* Improve IPS policies management
* Add IPS activities logging (e.g. block, unblock)
* Improve SNMP support
* Optimize polling of SNMP devices
* Improve SNMP v3 support
* Add more information including version
* Stateful SNMP alert to detect too many MACs on non-trunk
* Perform fat MIBs poll on average every 15 minutes
* Add preference to disable polling of SNMP fat MIBs
* Add more information to the historical flow data, including Latency, AS, Observation Points, SNMP interface, Host Pools
* Add detailed view of historical flows and alerts
* Add support for nProbe field L7_INFO
* Add ICMP flood alert
* Add Checks exclusion settings for subnets and for hosts and domains globally
* Add CDP support
* Add more regression tests
* Add support for obsolete client SSH version
* Add support for ERSPAN version 2 (type III)
* Add support for all the new nDPI Flow Risks added in nDPI 4.2
* Add extra info to service and periodicity map hosts
* Add Top Sites check
* REST API
* Getter for the bridge MIB
* Getter for LLDP adjacencies
* Check for BPF filters
* Score charts timeseries and analysis
Changes
* Encapsulated traffic is accounted for the lenght of the encapsulated packet and not of the original packet
* Remove nIndex support, including the flow explorer
* Remove MySQL historical flow explorer (export only)
* Hide LDAP password from logs
Fixes
* Fix a few memory leaks, double free, buffer overflow and invalid memory access
* Fix SQLite initialization
* Fix support for fragmented packets
* Fix IP validation in modals
* Fix netplan configuration manager
* Fix blog notifications
* Fix time range picker to support all browsers
* Fix binary application transfer name in alerts
* Fix glitches in chart drag operations
* Fix pools edit/remove
* Fix InfluxDB timeseries export
* Fix ELK memory leak
* Fix TLS version for obsolete TLS alerts when collecting flows
* Fix fields conversion in timeseries charts filters
* Fix some invalid nProbe field mapping
* Fix hosts Geomap
* Fix slow shutdown termination
* Fix wrong Call-ID 0 with RTP streams with no SIP stream associated
* Fix ping support for FreeBSD
* Fix active monitoring interface list
* Fix host names not always shown
* Fix host pools stats
* Fix UTF8 encoding issues in localization tools
* Fix time/timezone in forwarded syslog messages
* Fix unknown process alert
* Fix nil DOM javascript error
* Fix country not always shown in flow alerts
* Fix non-initialized traffic profiles
* Fix traffic profiles not working over ZMQ
* Fix syslog collection
* Fix async SNMP calls blocking the execution
* Fix CPU stats timeseries
* Fix InfluxDB attempts to alwa re-create retention policies
* Fix REST API ts.lua returning 24h data
* Fix processing of DNS packets under certain conditions
* Fix invalid space in SNMP Hostnames
* Fix REST API incompat. (/get/alert/severity/counters.lua, /get/alert/type/counters.lua)
* Fix map layout not saved correctly
* Fix LLDP topology for Juniper routers
* Fix not authorized error when editing SNMP devices
* Fix double 95perc, splitted avg and 95perc in sent/rcvd in charts
* Fix inconsistent local/remote timeseries
* Fix Risks generation in IPS policy configuration
* Fix deletion of sub-interface
* Fix deadline not honored when monitoring SNMP devices
* Fix traffic profiles on L7 protocols
* Fix TCP connection refused check
* Fix failures when the DB is not reacheable
* Fix segfault with View interfaces
* Fix hosts wrongly detected as Local
* Fix missing throughputs in countries
Misc
* Enforces proxy exclusions with env var `no_proxy`
* Move Lua engine to 5.4
* Major code review and cleanup
nEdge
* Add support for Ubuntu 20
* Add ability to logout when using the Captive Portal
* Add per egress interface stats and timeseries
* Add active DHCP leases in UI and REST API
* Add daily/weekly/monthly quotas
* Add service and periodicity maps and alerts
* Fix Captive Portal not working due to invalid allowed interface
* Fix addition of static DHCP leases
* Fix factory reset
* Fix reboot button
ntopng 5.0 (August 2021)
Breakthroughs
* Advanced alerts engine with security features, including the detection of [attackers and victims](https://www.ntop.org/ntopng/how-attackers-and-victims-detection-works-in-ntopng/)
* Integration of 30+ [nDPI security risks](https://www.ntop.org/ndpi/how-to-spot-unsafe-communications-using-ndpi-flow-risk-score/)
* Generation of the `score` [indicator of compromise](https://www.ntop.org/ntopng/what-is-score-and-how-it-can-drive-you-towards-network-issues/) for hosts, interfaces and other network elements
* Ability to collect flows from hundredths of routers by means of [observation points](https://www.ntop.org/nprobe/collecting-flows-from-hundred-of-routers-using-observation-points/)
* Anomaly detection based on Double Exponential Smoothing (DES) to uncover possibly suspicious behaviors in the traffic and in the score
* Encrypted Traffic Analysis (ETA) with special emphasis on the TLS to uncover self-signed, expired, invalid certificates and other issues
New features
* Ability to configure alert exclusions for individual hosts to mitigate false positives
* FreeBSD / OPNsense / pfSense [packages](https://packages.ntop.org/)
* Ability to see the TX/RX traffic breakdown both for physical interfaces and when receiving traffic from nProbe
* Add support for ECS when exporting to Syslog
* Improved TCP analysis, including analysis of TCP flows with zero window and low goodput
* Ability to send alerts to Slack
* Implementation of a token-based REST API access
Improvements
* Reworked the execution of hosts and flows checks (formerly user scripts), yielding a reduced CPU load of about 50%
* Improved 100Kfps+ [NetFlow/sFlow collection performance](https://www.ntop.org/nprobe/netflow-collection-performance-using-ntopng-and-nprobe/)
* Drilldown of [nIndex](https://www.ntop.org/guides/ntopng/advanced_features/flows_dump.html#nindex) historical flows much more flexible
* Migration to Bootstrap 5
* Check malicious JA3 signatures against all TLS-based protocols
* Reworked Doh/DoT handling
Fixes
* Fixes SSRF and stored-XSS injected with malicious SSDP responses
* Fixes several leaks in NetworkInterface
Notes
* To ensure optimal performance and scalability and to prevent uneven resource utilization, the maximum number of interfaces handled by a single ntopng instance has been reduced to
* 16 (Enterprise M)
* 32 (Enterprise L)
* 8 (all other versions)
* REST API v1/ is deprecated and will be dropped in the next stable release in favor of REST API v2/
* The old alerts dashboard has been removed and replaced by an advanced alerts drilldown page with integrated charts
All checksums have been double-checked against existing RMD160 and
SHA512 hashes
Not committed (merge conflicts...):
net/radsecproxy/distinfo
The following distfiles could not be fetched (fetched conditionally?):
./net/citrix_ica/distinfo citrix_ica-10.6.115659/en.linuxx86.tar.gz
./net/djbdns/distinfo dnscache-1.05-multiple-ip.patch
./net/djbdns/distinfo djbdns-1.05-test28.diff.xz
./net/djbdns/distinfo djbdns-1.05-ignoreip2.patch
./net/djbdns/distinfo djbdns-1.05-multiip.diff
./net/djbdns/distinfo djbdns-cachestats.patch
4.2 Stable
Breakthroughs
Flexible Alert Handling
Added recipients and endpoints to send alerts to different recipients on different channels, including email, Discord, Slack and Elasticsearch
Initial SCADA protocol support
Many internal components of ntopng have been rewritten in order to improve the overall ntopng performance, reduce system load, and capable of processing more data while reducing memory usage with respect to 4.0.
Cybersecurity extensions have been greatly enhanced by leveraging on the latest nDPI enhancements that enabled the creation of several user scripts able to supervise many security aspects of modern systems.
Behavioral traffic analysis and lateral traffic movement detection for finding cybersecurity threats in traffic noise.
Initial Scada support with native IEC 60870-5-104 support. We acknowledge switch.ch for having supported this development.
Consolidation of Suricata and external alerts integration to further open ntopng to the integration of commercial security devices.
SNMP support has been enhanced in terms of speed, SNMPv3 protocol support, and variety of supported devices.
New REST API that enabled the integration of ntopng with third party applications such as CheckMK.
New features
Traffic Behavioral Analysis
Periodic Traffic
Lateral Movements
TLS with self-signed certificates, issuerDN, subjectDN
Support for Industrial IOT and Scada with modbus, DNP3 and IEC60870
Support for attack mitigation via SNMP
Active monitoring
Support for ICMP v4/v6, HTTP, HTTPS and Speedtest
Ability to generate alerts upon unreachable or slow hosts or services
Detection of unexpected servers
DHCP, NTP, SMTP, DNS
Services map
nIndex direct to maximixe flows dump performance
MacOS package
Improvements
Implements per-category indicator of compromise score
Flexible configuration import/export/reset
Ability to import/export/reset all the ntopng configurations or parts of it
Increased nIndex dump throughput by a factor 10
Increased user scripts execution throughput
Massive cleanup/simplifications of plugins to ease community contributions
Improved cardinality estimation (e.g., number of contacted hosts, number of contacted ports) using Hyper-Log-Log
Added DSCP information
Reworked handling of dissected virtual hosts to improve speed and reduce memory
nEdge
Support for hardware bypass
Fixes
Fixed race conditions in view interfaces
Fixed crash when restoring serialized hosts in memory
Fixed conditions causing high CPU load
Fixes CSRF vulnerabilities when POSTing JSON
Fixes heap-use-after-free on HTTP dissected last_url
ntopng 4.0:
Breakthroughs
* Plugins engine to tap into flows, hosts and other network elements
* Migration to Bootstrap 4 and Font Awesome 5 for a renewed ntopng look-and-feel with light and dark themes
* Processes and containers monitoring thanks to the eBPF integration via libebpfflow https://github.com/ntop/libebpfflow
* Active monitoring of hosts ICMP/ICMPv6/HTTP/HTTPS Round Trip Times (RTT)
New features
* X.509 client certificate authentication
* ERSPAN transparent ethernet bridging
* Webhook export module for exporting alarms
* Identifications of the hosts in broadcast domain
* Category Lists editor to manage ip/domain lists
* Handling of PEN fields from nProbe
* Added anomalous flows to the looking glass
* Visibility of ICMP port-unreachable flows IPv4
* TCP states filtering (est., connecting, closed and rst)
* Ability to serialize local hosts in the broadcast domain via MAC address
* Japanese, portugese/brazilian localization
* Added process memory, cpu load, InfluxDB, Redis status pages and charts
* Implement ntopng Plugins, self contained modules to extend the ntopng functionalities
* Implement ZMQ/Suricata companion interface
* SSL traffic analysis and alerts via JA3 fingerprint, unsafe ciphers detection
* SSH traffic analysis and alerts via HASSH fingerprint
* Host traffic profile generation via the (MUD) Manufacturer Usage Descriptor
* Experimental Prometheus timeseries export
* Introduce the System interface to manage system wide settings and status
* Read events from Suricata and generate alerts
* SNMP network topology visualization
* Automatic ntopng update check and upgrade
* Calculate host anomaly score and trigger alerts when it exceeds a threshold
* Add ability to extract timeseries data with a click
* Initial Marketplace droplet using Fabric
* Alerts on duplex status change on SNMP interface
Improvements
* View interfaces are now optimized for big networks and use less memory
* Systemd macros are now used to start/restart the ntopng services
* Handles n2disk traffic extractions from recording processes non managed by ntopng
* Interface in/out now available also for non PF_RING interfaces (read from /proc)
* Automatic InfluxDB rollup support
* MDNS discovery improvements
* Rework of the alerts engine and api for efficient engaged alerts triggering
* Faster ZMQ communication to nProbe thanks to the implementation of a binary TLV format
* Stats update for ZMQ interfaces is now based on the idle/active flows timeout
* Timeseries export improvements via queues, detect if InfluxDB is down and stop the export
* Implemented reusable Lua engine to reduce the overhead of periodic scripts
* Improve Lua error handling
* Exclude certain categories from Elephant/Long lived flows alerts
nEdge
* Ability to set up port forwarding
* Support for Ubuntu 18.04
* Fix users and other prefs deleted during nEdge data reset
* Japanese localization
* Block unsupported L3 protocols (currently only ARP and IPv4 are supported)
* DNS mapping port to avoid conflicts with system programs
Fixes
* Fixed export to mysql on shutdown in case of Pcap file in community mode
* Fixed failing SYN-scan detection
* Fixed ZMQ decompression errors with large templates
* Fixed possible XSS in login.lua referer param and `runtime.lua`
* Update geolocation due to changes in the library usage policy
* Fixes to support browsers dark mode
* Option `--zmq-encryption-key <pub key>` can be used with `-I <endpoint>` to encrypt data hi hierarchical mode
* Fixed nIndex missing data while performing some queries and throughput calculation
3.8 Stable
New features
* Remote assistance to temporarily grant encrypted ntopng access to remote
parties
* Custom URLs and IP addresses mappings to traffic categories
* Continuous traffic recording
* User activities logging
* Extended chart metrics
Improvements
* Alerts
* Improved InfluxDB support
* Handles slow and aborted queries
* Uses authentication
* Adds RADIUS and HTTP authenticators
* Options to allow users login via RADIUS and HTTP
* Lua 5.3 support
* Improved performance
* Better memory management
* Native support for 64-bit integers
* Native support for bitwise operations
* Adds the new libmaxminddb geolocation library
* Storage utilization indicators
* Global storage indicator to show the disk used by each interface
* Per-interface storage indicator to show the disk used to store timeseries and flows
* Support for Sonicwall PEN field names
* Option to disable LDAP referrals
* Requests and configures Keepalive support for ZMQ sockets
* Three-way-handshake detection
* Adds SNMP mac addresses to the search function
nEdge
* Implement nEdge policies test page
* Implement device presets
* DNS
Fixes
* Fixes missing flows dump on shutdown
* HTTP dissection fixes
* SNMP
* Properly handles endianness over ZMQ
3.6.1 Stable
Brew formula fixes
3.6 Stable
New features
------------
New pro charts
Ability to compare data with the past (time shift)
Trend lines based on ASAP
Average and percentile lines overlayed on the graph and animated
New color scheme that uses pastel colors for better visualization
https://www.ntop.org/ntopng/ntopng-and-time-series-from-rrd-to-influxdb-new-charts-with-time-shift/
New timeseries API with support for RRD and InfluxDB
Abstracts and handles multiple sources transparently
https://www.ntop.org/guides/ntopng/api/lua/timeseries/index.html
Streaming pcap captures with BPF support
Download live packet captures right from the browser
New SNMP devices caching
Periodically cache information of all the SNMP device configured
Calculate and visualize interfaces throughput
Improvements
------------
Security
Access to the web user interface is controlled with ACLs
Secure ntopng cookies with SameSite and HttpOnly
HTTP cookie authentication
Improved random session id generation
Various SNMP improvemenets
Caching
Interfaces status change alerts
Device interfaces page
Devices and interfaces added to flows
Fixed several library memory leaks
Improved device and interface charts
Interfaces throughput calculation and visualization
Ability to delete all SNMP devices at once
Improved active devices discovery
OS detection via HTTP User-Agent
Alerts
Crypto miners alerts toggle
Detection and alerting of anomalous terminations
Module for sending telegram.org alerts
Slack
Configurable Slack channel names
Added Slack test button
Charts
Active flows vs local hosts chart
Active flows vs interface traffic chart
Ubuntu 18.04 support
Support for ElasticSearch 6 export
Added support for custom categories lists
Added ability to use the non-JIT Lua interpreter
Improved ntopng startup and shutdown time
Support for capturing from interface pairs with PF_RING ZC
Support for variable PPP header lenght
Migrated geolocation to GeoLite2 and libmaxminddb
Configuration backup and restore
Improved IE browser support
Using client SSL certificate for protocol detection
Optimized host/flows purging
