Improve docs, and add tool for sending example data (#16)

* config_example.conf: add documentation

* README.md: add more technical details

* add respondd-sim.py script, for sending test data back to FFMan

Also add some setup/usage notes for the script.

The respondd-sim.py script is based on the respondd.py script from
https://github.com/ffnord/mesh-announce .

README.md

@@ -72,9 +72,36 @@ Navigation bar at top of page:
 
 ## Technical Details
 
-List of known access points will be retrieved from [Yanic](https://github.com/FreifunkBremen/yanic) (ie. representing live data from APs).
-Additionally, APs can be added manually by visiting a page like /#/n/apname (where "apname" is the node-id of the new AP), and then setting a hostname.
+List of known nodes will be retrieved with the [respondd](https://github.com/freifunk-gluon/packages/tree/master/net/respondd) protocol (ie. by periodic UDP multicast requests to all reachable nodes). For this, FFMan uses a built-in [Yanic](https://github.com/FreifunkBremen/yanic) instance. The respondd protocol provides configuration details and statistics for each node.
 
-Each browser tab has a websocket connection to the server, so changes made in one tab will appear immediately in other tabs as well
+Alternatively, FFMan can also be configured to just listen to respondd responses (without sending requests); this is useful to "listen in" on the responses requested by a separate Yanic process running on the same network. This mode can be enabled by setting `yanic_collect_interval` to `0s` and settings `yanic.send_no_request` to `true`.
 
-All changes are saved to state file (eg. /tmp/freifunkmanager.json - can be changed in config file).
+Additionally, nodes can be added manually by visiting a page like /#/n/apname (where "apname" is the node-id of the new device), and then setting a hostname.
+
+The web interface displays all nodes that were found (except for nodes which don't respond to SSH - these are blacklisted). The web interface is updated live, by using a websocket connection; this also means that changes made in one tab will appear immediately in other tabs as well.
+
+When node settings are changed in the web interface, an SSH connection is opened to the node to apply the new settings.
+
+All changes are also saved to a state file (eg. /tmp/freifunkmanager.json - can be changed in config file).
+And all of the received node data is also stored in a database (see `db_type` and `db_connection` config options).
+
+## Creating dummy respondd data
+
+- create dummy "eth10" network interface:
+```
+sudo -s
+modprobe dummy
+ifconfig dummy0 hw ether 00:22:22:ff:ff:ff
+ipaddr=`ipv6calc --in prefix+mac --action prefixmac2ipv6 --out ipv6addr fe80:: 00:22:22:ff:ff:ff`
+ip a add dev dummy0 scope link $ipaddr/64
+ip link set dummy0 up
+```
+
+- edit tools/example-response.json to set network addresses for each node that are reachable with SSH (otherwise the nodes will be blacklisted immediately)
+    - also, if necessary adjust the `ssh_ipaddress_prefix` setting in config_example.conf to match the addresses from example-response.json
+- edit config_example.conf: in [yanic] section set ifname="dummy0"
+- edit $GOPATH/src/github.com/FreifunkBremen/yanic/respond/respond.go: change port=10001
+- go to $GOPATH/src/github.com/FreifunkBremen/freifunkmanager/ and run "go build"
+- start FFMan as usual (`./freifunkmanager -config config_example.conf`)
+- in another shell run `./tools/respondd-sim.py -p 10001 -i dummy0 -f tools/example-response.json`
+- FFMan should now display two new nodes with the example hostnames

config_example.conf

@@ -1,27 +1,51 @@
+# Database connection type (note: only "sqlite3" is supported at the moment)
 db_type = "sqlite3"
+# Database connection settings; see https://gorm.io/docs/connecting_to_the_database.html#Supported-Databases for details
 db_connection = "/tmp/freifunkmanager.db"
 
+# Address and port where HTTP server shall listen
 webserver_bind = ":8080"
+# Root directory to serve via HTTP
 webroot = "./webroot/"
 
+# Password required for making changes in the web interface
 secret = "passw0rd"
 
+# How long should a node remain on the blacklist after it has not responded to an SSH connection
+# (nodes are blacklisted if they have sent updated respondd data but are not reachable by SSH)
 blacklist_for = "1w"
 
+# SSH key for logging in on nodes
 ssh_key = "~/.ssh/id_rsa"
+# Only IP addresses starting with this prefix are used for SSH connection
 ssh_ipaddress_prefix = "fd2f:"
+# Timeout for SSH connections
 ssh_timeout = "1m"
 
-# enable receiving
+# If true, built-in Yanic instance will be used to request and collect respondd data from nodes
 yanic_enable = true
+# If set, Yanic startup will be delayed until the next full minute (or hour or whatever is configured here)
 # yanic_synchronize = "1m"
+# How often shall Yanic send respondd requests
 yanic_collect_interval = "10s"
 
+# More settings for the built-in Yanic instance
 [yanic]
+# Interface on which Yanic will send out respondd requests
 ifname = "wlp4s0"
 # e.g. to receive data of real yanic
 #  - please also disable `yanic_collect_interval`
 # ifname = "lo"
+
+# If set, Yanic will listen for response packets on this address only.
 # ip_address = "::1"
+
+# multicast address where respondd requests shall be sent. Default: ff05::2:1001
+# multicast_address = "ff05::2:1001"
+
+# If true, Yanic will not send own respondd request packets but will still listen for response packets
 # send_no_request = true
+
+# Local UDP port where Yanic will listen for response packets. Default: a dynamically selected port
+# (note: request packets to nodes will always be sent to port 1001, regardless of this setting)
 # port = 1001

tools/example-response.json

@@ -0,0 +1,26 @@
+[
+    {
+        "nodeinfo": {
+            "node_id": "002222ff1001",
+            "hostname": "testnode_101",
+            "network" : {
+                "addresses" : [
+                    "1234::5678",
+                    "9012::3456"
+                ]
+            }
+        }
+    },
+    {
+        "nodeinfo": {
+            "node_id": "002222ff1002",
+            "hostname": "testnode_102",
+            "network" : {
+                "addresses" : [
+                    "1234::5678",
+                    "9012::3456"
+                ]
+            }
+        }
+    }
+]

tools/respondd-sim.py

@@ -0,0 +1,77 @@
+#!/usr/bin/env python3
+
+#
+# Return pre-generated respondd responses (read from a JSON file).
+# The JSON file must contain a list of responses (one list entry for each simulated node).
+#
+
+import socketserver
+import argparse
+import socket
+import struct
+import json
+from zlib import compress
+
+
+def get_handler(inputFile):
+    class ResponddUDPHandler(socketserver.BaseRequestHandler):
+        def multi_request(self, providernames, nodeData):
+            ret = {}
+            for name in providernames:
+                if name in nodeData:
+                    ret[name] = nodeData[name]
+
+            print("reponse: %s" % ret)
+            return compress(str.encode(json.dumps(ret)))[2:-4]
+
+        def handle(self):
+            requestString = self.request[0].decode('UTF-8').strip()
+            socket = self.request[1]
+
+            print("got request: '%s' from '%s'" % (requestString, self.client_address))
+            if not(requestString.startswith("GET ")):
+                print("unsupported old-style request")
+                return
+
+            fp = open(inputFile)
+            testData = json.load(fp)
+            fp.close()
+
+            response = None
+            for nodeData in testData:
+                response = self.multi_request(requestString.split(" ")[1:], nodeData)
+                socket.sendto(response, self.client_address)
+    return ResponddUDPHandler
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument('-p', dest='port',
+                        default=1001, type=int, metavar='<port>',
+                        help='port number to listen on (default 1001)')
+    parser.add_argument('-g', dest='group',
+                        default='ff02::2:1001', metavar='<group>',
+                        help='multicast group (default ff02::2:1001)')
+    parser.add_argument('-i', dest='mcast_ifaces',
+                        action='append', metavar='<iface>',
+                        help='interface on which the group is joined')
+    parser.add_argument('-f', dest='input_file',
+                        type=str, metavar='<file>',
+                        required=True,
+                        help='JSON file to read responses from')
+    args = parser.parse_args()
+
+    socketserver.ThreadingUDPServer.address_family = socket.AF_INET6
+    server = socketserver.ThreadingUDPServer(("", args.port), get_handler(args.input_file))
+
+    if args.mcast_ifaces:
+        group_bin = socket.inet_pton(socket.AF_INET6, args.group)
+        for (inf_id, inf_name) in socket.if_nameindex():
+            if inf_name in args.mcast_ifaces:
+                mreq = group_bin + struct.pack('@I', inf_id)
+                server.socket.setsockopt(
+                    socket.IPPROTO_IPV6,
+                    socket.IPV6_JOIN_GROUP,
+                    mreq
+                )
+
+    server.serve_forever()