Difference between revisions of "Setting Up ntopng On My Home Network To Monitor Internet Traffic"

From Nearline Storage
Jump to navigation Jump to search
(Created page with "= This is a draft document = == Goal == Monitor network traffic to and from the internet via my cable modem so that I can stay under my data cap. == Architectural Overview...")
 
m
Line 7: Line 7:
 
== Architectural Overview ==
 
== Architectural Overview ==
  
−
<pre>
+
<pre>Cable Modem <-> Router <-> Managed ethernet switch <-> Mirrored port <-> Raspberry Pi w/ntopng
−
Cable Modem <-> Router <-> Managed ethernet switch <-> LAN <-> "netmon" server with ntopng
 
 
                                         ^
 
                                         ^
−
                                         | Mirrored port
+
                                         |  
−
                                         |
+
                                         V
−
                                        +->  Raspebbry Pi w/nprobe
+
                                      LAN</pre>
−
</pre>
 
  
 
* The "Managed ethernet switch" connects all of the various components of my local LAN together with the "Router" that is my gateway to the internet.
 
* The "Managed ethernet switch" connects all of the various components of my local LAN together with the "Router" that is my gateway to the internet.
 
* The "Managed ethernet switch" is configured to mirror all of the packets on the port that the "Router" is plugged into over to the port that the "Raspberry Pi" is plugged into.
 
* The "Managed ethernet switch" is configured to mirror all of the packets on the port that the "Router" is plugged into over to the port that the "Raspberry Pi" is plugged into.
−
* The Raspberry Pi runs the "nprobe" application to forward netflow data from the Pi to the "netmon" virtual server on my LAN which runs ntopng to collect and analyze the netflow data.
+
* The Raspberry Pi runs the "ntopng" application to collect and analyze the netflow data.  It provides a web GUI at port 3000 on my LAN.
  
−
== Installing nprobe On A Raspeberry Pi ==
+
== Installing ntopng On A Raspeberry Pi ==
  
−
# Install Raspian on the Pi.  As of this writing that is the "Buster" version of Raspian.
+
Install Raspian on the Pi.  As of this writing that is the "Buster" version of Raspian.
  
−
# Add the ntopng development repositories
+
Install mariadb-server and redis database packages.
−
<pre>
+
 
−
echo "deb http://apt.ntop.org/buster_pi armhf/" > /etc/apt/sources.list.d/ntop.list
+
Run <code>mysql_secure_installation</code>
−
echo "deb http://apt.ntop.org/buster_pi all/" >> /etc/apt/sources.list.d/ntop.list
+
 
−
</pre>
+
Set up ntop MySQL database:
 +
 
 +
<pre># mysql -u root -p
 +
Enter password:
 +
Welcome to the MySQL monitor.  Commands end with ; or \g.
 +
 
 +
mysql> create database ntopng;
 +
Query OK, 1 row affected (0.00 sec)
 +
 
 +
mysql> CREATE USER 'ntopng'@'localhost' IDENTIFIED BY 'Yoursecretpassword';
 +
Query OK, 0 rows affected (0.00 sec)
 +
 
 +
mysql> GRANT ALL PRIVILEGES ON ntopng . * TO 'ntopng'@'localhost';
 +
Query OK, 0 rows affected (0.00 sec)</pre>
 +
 
 +
Install geoipupdate RPM package, [https://www.maxmind.com/en/geolite2/signup create an account] at MaxMind, [https://www.maxmind.com/en/accounts/current/license-key register for an API key], and edit <code>/etc/GeoIP.conf</code> as follows:
 +
<pre># GeoIP.conf file for `geoipupdate` program, for versions >= 3.1.1.
 +
# Used to update GeoIP databases from https://www.maxmind.com.
 +
# For more information about this config file, visit the docs at
 +
# https://dev.maxmind.com/geoip/geoipupdate/.
 +
 
 +
# `AccountID` is from your MaxMind account.
 +
AccountID YOUR_ACCOUNT_ID_HERE
 +
 
 +
# `LicenseKey` is from your MaxMind account.
 +
LicenseKey YOUR_LICENSE_KEY_HERE
 +
 
 +
# `EditionIDs` is from your MaxMind account.
 +
EditionIDs GeoLite2-ASN GeoLite2-City GeoLite2-Country</pre>
 +
 +
Run <code>sudo geoipupdate</code> command to install the geoip database files.
 +
<br />Install redis and mariadb-server RPM packages
 +
<br />Compile and install the ntopng and nDPI packages from github.com/ntop
 +
 
 +
Compile ntopng from source: https://github.com/ntop
 +
 
 +
== Starting ntopng ==
 +
 
 +
Create <code>/etc/ntopng/ntopng.conf</code>, setting the interface to match the actual name of the network interface and dump-flows to match the MySQL database and userid you set up.
 +
<pre>#        The  configuration  file is similar to the command line, with the exception that an equal
 +
#        sign '=' must be used between key and value. Example:  -i=p1p2  or  --interface=p1p2  For
 +
#        options with no value (e.g. -v) the equal is also necessary. Example: "-v=" must be used.
 +
#
 +
#
 +
#      -G|--pid-path
 +
#        Specifies the path where the PID (process ID) is saved. This option is ignored when
 +
#        ntopng is controlled with systemd (e.g., service ntopng start).
 +
#
 +
# -G=/var/run/ntopng.pid
 +
#
 +
#      -e|--daemon
 +
#        This  parameter  causes ntop to become a daemon, i.e. a task which runs in the background
 +
#        without connection to a specific terminal. To use ntop other than as a casual  monitoring
 +
#        tool, you probably will want to use this option. This option is ignored when ntopng is
 +
#        controlled with systemd (e.g., service ntopng start)
 +
#
 +
# -e=
 +
#
 +
#      -i|--interface
 +
#        Specifies  the  network  interface or collector endpoint to be used by ntopng for network
 +
#        monitoring. On Unix you can specify both the interface name  (e.g.  lo)  or  the  numeric
 +
#        interface id as shown by ntopng -h. On Windows you must use the interface number instead.
 +
#        Note that you can specify -i multiple times in order to instruct ntopng to create  multi-
 +
#        ple interfaces.
 +
#
 +
# -i=eth1
 +
# -i=eth2
 +
-i=ens9
 +
#
 +
#      -w|--http-port
 +
#        Sets the HTTP port of the embedded web server.
 +
#
 +
# -w=3000
 +
#
 +
#      -m|--local-networks
 +
#        ntopng determines the ip addresses and netmasks for each active interface. Any traffic on
 +
#        those  networks  is considered local. This parameter allows the user to define additional
 +
#        networks and subnetworks whose traffic is also considered local in  ntopng  reports.  All
 +
#        other hosts are considered remote. If not specified the default is set to 192.168.1.0/24.
 +
#
 +
#        Commas  separate  multiple  network  values.  Both netmask and CIDR notation may be used,
 +
#        even mixed together, for instance "131.114.21.0/24,10.0.0.0/255.0.0.0".
 +
#
 +
# -m=10.10.123.0/24
 +
# -m=10.10.124.0/24
 +
#
 +
#      -n|--dns-mode
 +
#        Sets the DNS address resolution mode: 0 - Decode DNS responses  and  resolve  only  local
 +
#        (-m)  numeric  IPs  1  -  Decode DNS responses and resolve all numeric IPs 2 - Decode DNS
 +
#        responses and don't resolve numeric IPs 3 - Don't decode DNS responses and don't  resolve
 +
#
 +
-n=1
 +
#
 +
#      -S|--sticky-hosts
 +
#        ntopng  periodically purges idle hosts. With this option you can modify this behaviour by
 +
#        telling ntopng not to purge the hosts specified by -S. This parameter requires  an  argu-
 +
#        ment  that  can  be  "all"  (Keep  all hosts in memory), "local" (Keep only local hosts),
 +
#        "remote" (Keep only remote hosts), "none" (Flush hosts when idle).
 +
#
 +
# -S=
 +
#
 +
#      -d|--data-dir
 +
#        Specifies the data directory (it must be writable by the user that is executing ntopng).
 +
#
 +
# -d=/var/lib/ntopng
 +
#
 +
#      -q|--disable-autologout
 +
#        Disable web interface logout for inactivity.
 +
#
 +
# -q=
 +
#
 +
#      -F|--dump-flows <mode>
 +
#        If ntopng is compiled with sqlite support, flows  can  dumped  persis‐
 +
#        tently  on disk using this option. The mode can be set to es - Dump on
 +
#        ntopng.es queue in Elasticsearch format that be insert on a  ES  data‐
 +
#        base.  In  this  case  the  format  is  "es;<idx  type>;<idx name>;<es
 +
#        URL>;<http auth>". Example:  -F  "es;ntopng;ntopng-%Y.%m.%d;http://lo‐
 +
#        calhost:9200/_bulk;user:pwd".  The  <idx  name> accepts the strftime()
 +
#        format.  mysql - Dump flows in MySQL tables. In this case  the  format
 +
#        is  "<host[@port]|unix  socket>:<dbname>:<table>:<user>:<pw>". Example
 +
#        -F "mysql;localhost;ntopng;flows-%Y.%m.%d;root;".
 +
#
 +
-F="mysql;localhost;ntopng;ntopng_table;ntopng;ntopng"</pre>
 +
 
 +
Create /usr/lib/systemd/system/ntopng.service, setting the correct interface name in the ExecPre ethtool command:
 +
 
 +
<pre>[Unit]
 +
Description=ntopng high-speed web-based traffic monitoring and analysis tool
 +
After=mariadb.service redis.service
 +
Requires=mariadb.service redis.service
 +
Wants=network.target
 +
 
 +
[Service]
 +
Type=simple
 +
 
 +
ExecPre=/usr/sbin/ethtool -K ens9 gro off gso off tso off
 +
ExecStart=/usr/local/bin/ntopng /etc/ntopng/ntopng.conf
 +
 
 +
Restart=on-failure
 +
RestartSec=5
 +
 
 +
[Install]
 +
WantedBy=multi-user.target</pre>
 +
 
 +
== Resetting ntopng password ==
 +
 
 +
<pre>$ sudo redis-cli del ntopng.user.admin.password</pre>
 +
Restart ntopng
 +
<br />Password will revert to default: <code>admin</code>
 +
 
 +
== Deleting ntopng data ==
 +
 
 +
<pre>$ sudo rm -fr /var/lib/ntopng
 +
 
 +
$ mysql -u notopng -p ntopng
 +
> drop table flowsv4, flowsv6;
 +
> quit;</pre>
 +
 
 +
Can also delete data from redis, which will clear settings and passwords as well:
 +
 
 +
<pre>$ sudo redis-cli keys "ntopng.*" | xargs sudo redis-cli del
 +
$ sudo redis-cli keys "ntonpng.*" | xargs sudo redis-cli del</pre>
 +
 
 +
== Notes ==
 +
 
 +
After first installing ntopng, or after deleting data, the six category list files defined by default will not exist.  Error's will appear in the console log to that effect.  These tables can be loaded (updated) manually from the Category Lists menu in the UI or they will get updated automatically as ntopng runs.
 +
 
 +
[[Category:Networking]]

Revision as of 16:17, 6 December 2020

This is a draft document

Goal

Monitor network traffic to and from the internet via my cable modem so that I can stay under my data cap.

Architectural Overview

Cable Modem <-> Router <-> Managed ethernet switch <-> Mirrored port <-> Raspberry Pi w/ntopng
                                        ^
                                        | 
                                        V
                                       LAN
  • The "Managed ethernet switch" connects all of the various components of my local LAN together with the "Router" that is my gateway to the internet.
  • The "Managed ethernet switch" is configured to mirror all of the packets on the port that the "Router" is plugged into over to the port that the "Raspberry Pi" is plugged into.
  • The Raspberry Pi runs the "ntopng" application to collect and analyze the netflow data. It provides a web GUI at port 3000 on my LAN.

Installing ntopng On A Raspeberry Pi

Install Raspian on the Pi. As of this writing that is the "Buster" version of Raspian.

Install mariadb-server and redis database packages.

Run mysql_secure_installation

Set up ntop MySQL database: 

# mysql -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.

mysql> create database ntopng;
Query OK, 1 row affected (0.00 sec)

mysql> CREATE USER 'ntopng'@'localhost' IDENTIFIED BY 'Yoursecretpassword';
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON ntopng . * TO 'ntopng'@'localhost';
Query OK, 0 rows affected (0.00 sec)

Install geoipupdate RPM package, create an account at MaxMind, register for an API key, and edit /etc/GeoIP.conf as follows: 

# GeoIP.conf file for `geoipupdate` program, for versions >= 3.1.1.
# Used to update GeoIP databases from https://www.maxmind.com.
# For more information about this config file, visit the docs at
# https://dev.maxmind.com/geoip/geoipupdate/.

# `AccountID` is from your MaxMind account.
AccountID YOUR_ACCOUNT_ID_HERE

# `LicenseKey` is from your MaxMind account.
LicenseKey YOUR_LICENSE_KEY_HERE

# `EditionIDs` is from your MaxMind account.
EditionIDs GeoLite2-ASN GeoLite2-City GeoLite2-Country

Run sudo geoipupdate command to install the geoip database files.
Install redis and mariadb-server RPM packages
Compile and install the ntopng and nDPI packages from github.com/ntop

Compile ntopng from source: https://github.com/ntop

Starting ntopng

Create /etc/ntopng/ntopng.conf, setting the interface to match the actual name of the network interface and dump-flows to match the MySQL database and userid you set up. 

#         The  configuration  file is similar to the command line, with the exception that an equal
#        sign '=' must be used between key and value. Example:  -i=p1p2  or  --interface=p1p2  For
#        options with no value (e.g. -v) the equal is also necessary. Example: "-v=" must be used.
#
#
#       -G|--pid-path
#        Specifies the path where the PID (process ID) is saved. This option is ignored when
#        ntopng is controlled with systemd (e.g., service ntopng start).
#
# -G=/var/run/ntopng.pid
#
#       -e|--daemon
#        This  parameter  causes ntop to become a daemon, i.e. a task which runs in the background
#        without connection to a specific terminal. To use ntop other than as a casual  monitoring
#        tool, you probably will want to use this option. This option is ignored when ntopng is
#        controlled with systemd (e.g., service ntopng start)
#
# -e=
#
#       -i|--interface
#        Specifies  the  network  interface or collector endpoint to be used by ntopng for network
#        monitoring. On Unix you can specify both the interface name  (e.g.  lo)  or  the  numeric
#        interface id as shown by ntopng -h. On Windows you must use the interface number instead.
#        Note that you can specify -i multiple times in order to instruct ntopng to create  multi-
#        ple interfaces.
#
# -i=eth1
# -i=eth2
-i=ens9
#
#       -w|--http-port
#        Sets the HTTP port of the embedded web server.
#
# -w=3000
#
#       -m|--local-networks
#        ntopng determines the ip addresses and netmasks for each active interface. Any traffic on
#        those  networks  is considered local. This parameter allows the user to define additional
#        networks and subnetworks whose traffic is also considered local in  ntopng  reports.  All
#        other hosts are considered remote. If not specified the default is set to 192.168.1.0/24.
#
#        Commas  separate  multiple  network  values.  Both netmask and CIDR notation may be used,
#        even mixed together, for instance "131.114.21.0/24,10.0.0.0/255.0.0.0".
#
# -m=10.10.123.0/24
# -m=10.10.124.0/24
#
#       -n|--dns-mode
#        Sets the DNS address resolution mode: 0 - Decode DNS responses  and  resolve  only  local
#        (-m)  numeric  IPs  1  -  Decode DNS responses and resolve all numeric IPs 2 - Decode DNS
#        responses and don't resolve numeric IPs 3 - Don't decode DNS responses and don't  resolve
#
-n=1
#
#       -S|--sticky-hosts
#        ntopng  periodically purges idle hosts. With this option you can modify this behaviour by
#        telling ntopng not to purge the hosts specified by -S. This parameter requires  an  argu-
#        ment  that  can  be  "all"  (Keep  all hosts in memory), "local" (Keep only local hosts),
#        "remote" (Keep only remote hosts), "none" (Flush hosts when idle).
#
# -S=
#
#       -d|--data-dir
#        Specifies the data directory (it must be writable by the user that is executing ntopng).
#
# -d=/var/lib/ntopng
#
#       -q|--disable-autologout
#        Disable web interface logout for inactivity.
#
# -q=
#
#       -F|--dump-flows <mode>
#        If ntopng is compiled with sqlite support, flows  can  dumped  persis‐
#        tently  on disk using this option. The mode can be set to es - Dump on
#        ntopng.es queue in Elasticsearch format that be insert on a  ES  data‐
#        base.  In  this  case  the  format  is  "es;<idx  type>;<idx name>;<es
#        URL>;<http auth>". Example:  -F  "es;ntopng;ntopng-%Y.%m.%d;http://lo‐
#        calhost:9200/_bulk;user:pwd".  The  <idx  name> accepts the strftime()
#        format.  mysql - Dump flows in MySQL tables. In this case  the  format
#        is  "<host[@port]|unix  socket>:<dbname>:<table>:<user>:<pw>". Example
#        -F "mysql;localhost;ntopng;flows-%Y.%m.%d;root;".
#
-F="mysql;localhost;ntopng;ntopng_table;ntopng;ntopng"

Create /usr/lib/systemd/system/ntopng.service, setting the correct interface name in the ExecPre ethtool command: 

[Unit]
Description=ntopng high-speed web-based traffic monitoring and analysis tool
After=mariadb.service redis.service
Requires=mariadb.service redis.service
Wants=network.target

[Service]
Type=simple

ExecPre=/usr/sbin/ethtool -K ens9 gro off gso off tso off
ExecStart=/usr/local/bin/ntopng /etc/ntopng/ntopng.conf

Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Resetting ntopng password

$ sudo redis-cli del ntopng.user.admin.password

Restart ntopng
Password will revert to default: admin

Deleting ntopng data

$ sudo rm -fr /var/lib/ntopng

$ mysql -u notopng -p ntopng
> drop table flowsv4, flowsv6;
> quit;

Can also delete data from redis, which will clear settings and passwords as well: 

$ sudo redis-cli keys "ntopng.*" | xargs sudo redis-cli del 
$ sudo redis-cli keys "ntonpng.*" | xargs sudo redis-cli del

Notes

After first installing ntopng, or after deleting data, the six category list files defined by default will not exist. Error's will appear in the console log to that effect. These tables can be loaded (updated) manually from the Category Lists menu in the UI or they will get updated automatically as ntopng runs.

Retrieved from "https://wiki.daveking.com/index.php?title=Setting_Up_ntopng_On_My_Home_Network_To_Monitor_Internet_Traffic&oldid=4422"