DXpedition Build – Part 3

Part 3 picks up with the way the Pi Collectors operate and some of the tools I built to test the project along the way

The two Pi Collectors are configured nearly identically to store contact logs, and to upload those records to a remote FTP server for post processing. Custom applications are written in Python. The Pi Collectors are configured with the standard Raspbian linux distribution available from raspberrypi.org.

The Pi Collectors have a wired Ethernet connection to a switch where many laptop systems are also connected. That switch is connected to an standard consumer grade gateway that provides DHCP services and an uplink WAN port. The WAN port is connected to the Satellite Uplink terminal.

Data Collection

The servers have a custom python collector application and a database to store contact information. Every 5 minutes the server runs an application that checks for an active Internet connection and uploads the records to an FTP server.

Laptop workstations have an application (N1MM) that gathers and sends pseudo-XML formatted contact information to the Pi Collectors as a UDP broadcast. The Python application (cqss) uses a UDP socket service that listens for the UDP messages on the broadcast address of the network subnet. Both servers save the data in a log file (/var/log/CQ.log), then parses the XML data, saving selected fields in a local mysql database.

The Python collection application is installed as a service and starts automatically when the Pi boots up. It can be started and stopped at the command line of the server (sudo service cqss start | stop). The application is designed to handle errors gracefully, such as malformed packets, and continue to operate.

Contact Upload

Another Python application (contactUpload) is designed to attempt an upload of contact records only when an active Internet connection is available. The application tries to request a file from the FTP server. If the FTP server responds, the application continues operation.

It queries the database for records that have not yet been uploaded. An abbreviated version of the N1MM records are re-packaged in a pseudo-XML format (ADIF) and saved locally as a file, sans the XML header typically found on XML files. The program then opens a connection to the FTP server and transfers the file. The program updates the status of the contact records to reflect they have been ‘sent’.

The FTP identity and credentials are kept in a local XML file which are read into the application at run time. The values can be edited to change thehost URL, username and password.

This application is registered with the linux cron service to be run every 5 minutes. This can be adjusted by using crontab (crontab -e) to edit the entry for this application. Changes to the crontab table are immediate.

Upon a successful FTP upload an entry is written to an HTML formatted log file called “CQupload.html” which is viewable with a browser using a link on the local apache web server home page.  Another web page can be viewed via a link in the home page that shows the total contacts in the database and lists out the contact records in the order they were recorded.

Management

A few utilities have been created to test the applications and monitor the server behavior.

Web Page

An apache web server is running on both Pi Collectors and any computer, tablet, or smart phone connected to the workstation network or the collection server Wi-Fi network can view the web page with any browser. This browser checks for active processes and displays the results on the home page. The services tested are NTP, the Data Collector Service, the database, and the DHCP service for the Wi-Fi connection.

Database Status Reset

A small Python application (/home/pi/dbreset) will reset the status field in every contact record. The result is the contact upload program will upload all contact records.

Test UDP Client

While a workstation is great test tool for sending sample traffic, a small Python application was written to send sample packets at a high rate of speed. It is configured to send a single sample recordset to a server’s IP address/UDP port repetitively until cancelled (Crtl-C). It uses the WiFi IP address to make is easier for me to use. Haha! It sends 100 records, one record every 1/2 second.

The receive data lines and the resulting print statement are just there for real time debugging, and they can be commented out.

Simple Database Dump

This Python application simply reads all the records in the database and dumps them to the screen. This is useful to see whether records are being collected (status = new) and whether they have been uploaded (status = sent).

#!/usr/bin/python

import socket
import sys
import MySQLdb

# Connect to MySQL data base and create handle...
db = MySQLdb.connect("localhost","root","raspberry","SQ")
cursor = db.cursor()
cursor.execute ("SELECT * FROM records")  

for reading in cursor.fetchall():
    print str(reading[0])+"    "+str(reading[1])+"     "+\
        str(reading[2])+"      "+str(reading[3])

db.close

This Python application tests that the database application (mysql) is running by making a connection and displaying the version on the console.

#!/usr/bin/python
import MySQLdb as mdb
import sys

try:
    con = mdb.connect('localhost', 'root', 'raspberry', 'SQ');
    cur = con.cursor()
    cur.execute("SELECT VERSION()")
    ver = cur.fetchone()
    print ("Database version : %s " % ver)
except mdb.Error, e:
    print ("Error %d: %s" % (e.args[0],e.args[1]))
    sys.exit(1)
finally:    
    if con:    
        con.close()

Logging

The data collection program records raw data, errors and notable successes within the application to the /var/log/CQ.log file that is viewable at the server console. The contact upload program records the success of uploads to the CQupload.html file for viewing by a web browser. There are a number of print statements that are not visible when the programs are invoked programmatically, but are visible when run directly. These statements are primarily for debugging the program.

Networking configuration

The network interfaces are configured such that the wired network interface is using DHCP, and the Wi-Fi interface uses a static IP address. By default th networking default gateway is based on the DHCP process of the wired network interface (eth0).