Getting into it Again & First Project – Presenting Data Over Time

Getting into it Again

It was not the most auspicious of starts.  Two introductory blog posts, a couple of draft posts in the pipeline then nothing… a six year hiatus from the fun.  The draft posts no longer apply so have been deleted.

I’ve got a little more spare time now for my own projects so am back again.  I have plans for a small number of projects that involve a mixture of software, electronics, programmable interface controllers and micro-computers.  I have added a permanently running Linux server to my household and will be using that as well as my Windows PC.  I’ve also installed a Power-Over-Ethernet (POE) switch so I can run Ethernet-connected, USB-powered devices anywhere in the house.

The first hardware projects I want to work on are likely to be devices that collect data from sensors.  When this data is collected, over time, I want to be able to present it.  This very first project is a software solution that can be used to present data over time.

First Project – Presenting Data Over Time

Many years ago, a colleague of mine setup a simple web page that presented the quantity of network traffic going to and from the internet.  My colleague used a tool called the Multi-Router Traffic Grapher (MRTG) that created static image files of graphs.  One graph presented the data for the last 24 hours, a second for the last month and the third covered the last year.  When I asked that same colleague, recently about it, he suggested RRDtool or Cacti.

The requirements I set myself were to create something similar.  My goal is to capture a value at regular intervals and present the data in 3 graphs showing 3 amounts of history.  The most recent 24 hours, most recent month and most recent year.

The data that I am going to capture is the up-time of the PC.  The decimal number of days that it has been turned on and running. It is an easily accessible, changing value and is more useful than presenting a generated random number.  The data-point will be captured every 5 minutes.  I’ve also decided to make this cross-platform so it runs on the Linux server and my work PC (Windows 10) at the office.

To implement this, I broke it down into a number of steps:

1. Examine RRDtool to see if it does what I want.  Explore what would be required to achieve it.
2. Determine how to extract the up-time as a decimal number of days from a Windows 10 PC.
3. Determine how to run this every 5 minutes to load an RRDtool database.
4. Determine how to get RRDtool to write out the graphs I want to see.
5. Implement an html file to present the graphs.
6. Repeat steps 2 to 5 but on the Linux server.

I will not go into much detail of these individual steps as there are plenty of information resources covering the topics in depth.  I’m just going to state what each stage involved and note any problems I encountered.  The code files for the result are available on GitHub some detail about how to install the various elements of the solution are detailed in the project’s README.md file.

1. RRDtool

I’ve chosen RRDtool rather than Cacti as it is at the lower level of the two.  Cacti provides tools and services that extend RRDtool and sit on top of it.  I’ll look into Cacti at a future time.

RRDtool does what I want.  I’ve followed a few online resources and I found the Getting Started with RRDtool at cuddletech.com to be concise and easy to follow.

2. Determine Up-time of the PC

The same software is being used on Windows and Linux so I needed to extract the up-time in the same format.  I chose the decimal number of days because it makes sense on the y-axis of the graphs I want to see.  On Linux, getting the up-time involved piping 3 commands:

1. echo “($(date +%s) – $(date -d “$(uptime -s)” +%s))/(60*60*24)”
   This writes out the calculation for the up-time in days as a string.  It generates a string something like: “(1557700803 – 1557657626)/(60*60*24)”.
2. bc -l
   This takes the previous string and executes the calculation.  -l makes it use the standard maths library.  This yields the value “.49973379629629629629” for the above string.
3. awk ‘{printf “%.15f\n”, $0}’
   This formats the above number to 15 decimal places and includes a leading zero.  It converts the above to “0.499733796296296”.

On Windows, the following commands are used in Powershell:

1. $lb=Get-WmiObject Win32_OperatingSystem | Select-Object LastBootUpTime
   This gets the last boot-up time of the PC.  When “Fast-startup” turned on, the PC will retain the same last boot-up time between shutdowns using the power button (see here).  Turning Fast-startup off will fix this issue.
2. $lbdt = [Management.ManagementDateTimeConverter]::ToDateTime($lb.LastBootUpTime)
   This simply converts the above $lb value to a DateTime value.
3. $ut = (get-date) – $lbdt
   This calculates the difference between now and the last boot-up time.  The resulting value is a TimeSpan object.
4. $strString = “{0}” -f $ut.TotalDays
   This sets $string to the decimal number of days of up time to 15 decimal places.

The calculated up-time value is then passed to RRDtool to be stored in the N time-point (meaning the current time) in the database.

3. Scheduled Execution

Windows and Linux both have excellent scheduled job running tools.  I’ve used these (cron on Linux and Task Scheduler on Windows) to run the uptime-capture script every 5 minutes when the PC is running.

4. Creating the Graphs

The “rrdtool graph” commands is used to generate the graphs.  The only difference between Windows and Linux calls is that Windows does not carry the font file that rrdtool uses by default.  A font file (including location) has to be explicitly stated for the Title, Axis, Legend and Units of the graph.

5. Implement HTML to Present the Graphs

I’ve created a simple html file and css file to present the graphs.

 

As previously mentioned, the code and installation instructions are available on GitHub.

Preamble 2 – An Evening with Arduino and LEDs

In my last post, I mentioned that I had bought an initial electronics play kit consisting of an Arduino Uno, USB cable, breadboard, jump wires and a selection of LEDs and resistors. I also mentioned that I had access to the book Beginning Arduino by Michael McRoberts.

The first chapter of the book takes you through what the Arduino is, how to install the IDE for programming it and how to upload your first sketch to it (a sketch is a compiled program that is uploaded to the Arduino board from the IDE). It then goes through the Arduino IDE in a bit more detail. The second chapter goes straight into the first project which is to make an LED blink.

Using just the Arduino, breadboard, an LED, a suitable resistor and two jumper wires, I made the circuit that would light the LED from port 10 of the Arduino board. (Note: the Beginning Arduino book explained how to calculate what would be a suitable resistor)

Here is a still photo of the setup:

Now you’re possibly thinking, “that’s easy” or “that’s useless”, and you may be right. But that LED was blinking in a manner controlled by a sketch written by me on my laptop. It was an LED in a circuit that didn’t exist until I built it from the bits I had bought. I found this to be a very exciting moment.

I then extended it by adding two more LED circuits on other ports and changed the sketch to light them up in a traffic light sequence. Here’s a video of what I ended up with:

The 3 LEDs are all different because I bought a lucky-dip bag of LEDs and wasn’t lucky enough to get red, green and yellow of the same type and size.

I then spent the rest of the evening playing with timings and sketches to light the LEDs in different patterns.

Preamble 1 – An Introduction & My Initial Purchases

Many years ago, I started my post-secondary school education with an electrical and electronic engineering qualification. After completing that, I changed to Computer Studies and have been working in I.T. and DOS/Windows/Intel PC software development ever since. See the About page for a little more detail.

I don’t know what has made me want to look at electronics again in recent years, I think it is possibly seeing a lot of mash-up and electronics projects appearing on the web as all sorts of wonderful toys become accessible. I had been considering getting a Raspberry Pi to play about with for a while but, recently, when I was in my local electronics store, I bought an Arduino and a few extra bits and pieces to play with:

I picked up the following:

1. Arduino Uno & USB cable
2. A bread board
3. Jump wire kit
4. Lucky-dip bag of LEDs
5. Lucky-dip bag of Resistors

I also had access to the following book:

This book was great at explaining the initial electronics principles that enabled me to use the above kit to get LEDs up and running and controlled by the Arduino. I’ll detail the two little LED projects that I got running in my next post.