In my last post I mentioned that I might be getting some solar heating.  I had checked my fan to make sure it was turning and found that it was stuck.  A small push got it turning again.  However my QC checks still showed signs of solar heating during the daytime.  Then I started seeing elevated temperature readings at night as well.  I suspected the fan bearing had seized causing the fan to get hot and be the source of the erroneous temperature readings.  So I made some time to go outside, take the solar shield apart and see what was going on.

The first thing I did was listen for the fan running.  No fan was heard.  Then I put my hand up inside the solar shield to check the fan wiring to see if it had come loose.  My hand brushed up against something that I didn’t recognize.  I bent down and looked up inside the solar shield to find a small wasp next.  Luckily it appeared it be deserted, otherwise I would most likely have been stung.  I knocked down the nest and started taking the solar shield apart.  A few wasps flew out and i jumped back.  When the coast was clear I carefully started opening up the solar shield and peeked inside.  I saw a larger wasp nest and several wasps flew out toward my face.  I backed up as fast as I could move in preparation to run away.  I ended up falling over backwards and slightly injuring myself.  Luckily the wasps let me escape without stinging me.  I got a can of bug spray put of the garage and that was the end of the wasp nest.

I was now able to complete my dis assembly and confirm that the fan had failed.  I ordered a new fan, which took about a week to receive.  Then I removed the old fan and wired in the replacement.  The old fan was pretty hot when I pulled it out, so I feel pretty confident that ti was the source of my heating.  The station appears to be working so I plan to let it run for a day or two and keep an eye on my QC checks.  I also have a spare fan (I bought two) so I can make repairs faster when I have this problem again.

Just wanted to post an update after my fixes from last month.  The QC page from CWOP shows all green thumbs up for my weather station quality checking now.  That means that the comparison to whichever neighboring stations it uses shows that my station is at least in the ballpark.  That’s a whole lot better than it was before I installed the new humidity sensor.  The wind speed statistics also look good since I made the code change.  I need to go back and clean up an erroneous reading that was recorded prior to making the code change.  I’m not sure if it was caused by the clock bug or not, but it is definitely an error.  I also need to take a look at the fan I’m using to make sure it is still working.  The temperature readings look like I could be getting some unwanted solar heating.  So some maintenance on the solar radiation shield could be called for.  I’ll check it out and post back here if anything needed to be corrected.

My weather data has been flagged as having an issue on the quality checking from CWOP.  Specifically, the humidity readings have been too high compared to neighboring stations.  I had purchased a replacement humidity sensor chip several months ago, but never had the opportunity to install it.  Yesterday that opportunity presented itself.  I shut down the station, took it apart enough to get to the old sensor and brought it inside to the workbench.  It wasn’t very difficult to unsolder the old sensor and solder the new one in its’ place.  Then I reassembled the station and started it back up.  The humidity readings appear to be more reasonable.  I’ll have to wait a few days and see how the quality checking goes.

While i was at it, I modified the code to use System.nanoTime() rather than System.currentTimeMillis() for calculating the wind speed.  The reason for the change wasn’t specifically to gain additional precision (although that is an OK bonus). My concern is that it is possible for the time to be corrected (such as via NTP or by the admin) in the midst of doing the timing for the wind speed.  This could cause an incorrect value to be calculated, and has led to readings of wind speeds in excess of 100 mph in some other weather software packages.  Mine is most likely susceptible to this bug as well.  While this is a rare occurrence and the probability of it happening is slim, the Java recommended method for doing timing is using nanoTime(), which is based on a relative time and is only useful for comparing to other nanoTime() values.  Again, I’ll follow my data for the next few days and see  how it looks.

This past weekend I noticed that my weather Web site was no longer updating with the most recent weather data from my weather station.I check the error logs on the Raspberry Pi running the weather software and saw numerous errors indicating communication problems with the weather station.  Upon closer inspection I discovered that when my son mowed the lawn he somehow managed to sever the cable between the weather server and the weather station hardware.

I promptly shut down the weather station software.  I had to let the station stay offline until I had an opportunity to purchase a new 25 foot long Cat 5 network cable.  In the mean time I made some minor coding modifications to the JavaScript code to better display when there were data errors.  The PHP code displayed the errors how I wanted, but the JavaScript code showed zeroes instead of an error indicator.

Today I had some time to stop by the computer store and purchase a new cable for $14.00.  I removed the severed cable and installed the new one.  Once I restarted the weather software everything came back up fine.  So the weather station is open for business once again.

I’ve been toying with the idea of making the weather station wireless by locating a Raspberry Pi outside with the weather station and use a wireless network to connect to the station.  Right now I have the Pi running from a USB hard drive which I would not want to locate outdoors.  The main reason for the USB hard drive is so that I don’t wear out the SD card from too many writes.  I could split up the software so that the outdoor station reads the hardware and makes the readings available over the network.  This would eliminate many of the writes to the SD card.  An indoor computer could retrieve the raw data and then process it, store it, and update any remote weather services.

The change wouldn’t be difficult to make, and is kind of interesting.  There are a lot of wireless networks in my neighborhood and lots of interference.  Wireless networking isn’t always as reliable in my house as I would like, so I’ve resorted to wired wherever I can.  I may choose to make this change some time, but for now I’m good with the wired connection.

It took me long enough, but I finally moved my weather station software to the Raspberry Pi.  I am happy to say that it is running smoothly, my weather database is updating, all the services I share my weather data with are receiving data, and my Web site is up to date.  I set the software up to run as a service so that I can use the typical Linux tools for managing, starting, and stopping the software.  This is no different than what I was doing on the prior server.  I installed my logrotate scripts and verified that they run by hand, but haven’t conformed that the logs are rotating automatically yet.  The log files need to grow a little more before I will see them being rotated.

I also updated my logretriever software to accept several new command line parameters.  Originally I had hard coded the server hostname, port, and the file name to store the retrieved data in.  Now they can all be specified on the command line.  I also added a usage/help display when the parameters are specified incorrectly.  The downside is that I used the commons cli jar file to do the heavy lifting for this functionality.  That means a new dependency to run the software.

I also made a few script updates.  I had to change the scripts that run logretriever to use the new command line parameters.  While I was in there I also modified the script that updates the weather data to use a loop rather than repeating the same commands multiple times.  Not sure why I never made that change.  It makes for less code to maintain, and also makes it easier to fine tune the script as far as the number of times to run and the delay between each run.

It’s only been one day so far, but everything is working correctly.  The updated software has been uploaded to the Web site.  I will continue to monitor everything closely for the next week or so to make sure there are no issues.

My scripts have been running pretty well these last few weeks. I added some error checking in case something does go wrong. But overall they have been humming along nicely. Time to move on to the next project, which is transitioning my station computer over to one of my Raspberry Pi microcomputers. In case some of you aren’t familiar with the Raspberry Pi, it is a small, single board microcomputer that was originally designed as an educational device. While it serves that purpose well, its’ low price allowed it to be used by hobbyists for many other purposes. For more information on the Raspberry Pi you can browse to raspberrypi.org to get more information on the board.

The Raspberry Pi is similar in many ways to the GuruPlug computer I had used previously as my weather station computer.  That computer, as well as the Pi uses an SD card as its’ primary storage device.   So the operating system and any files it needs are all located on a storage device rather than a traditional hard drive.  While the access speeds are fairly fast compared to a hard drive, the SD card has limited storage and a finite number of write cycles.  This isn’t a big problem for many uses, but my weather station software writes to the file system fairly often.  And since I keep a database with my weather data the limited storage could eventually become a problem.

I had a spare external hard drive with a USB interface I could use to add storage to the Pi.  Utilizing this drive would permit the weather station software to write to a hard drive and avoid wearing out the SD card so quickly.  It also had plenty of storage if I wanted a database to store the weather data.  Sounded like a win/win type of situation.  Then I thought about taking it one step further and installing the operating system on the hard drive and having the boot loader mount the drive and load the operating system from there.  I did some quick Web searching and came up with the solution.

The page I found is The Raspberry Pi Hobbyist: Running From an External Hard Drive.  This page had detailed instructions on how to set everything up.  If you aren’t familiar with Linux the instructions will look like some arcane version of gibberish.  But to somebody familiar with installing hard drives on Linux/Unix it will make sense.  You can look at that page for details, but here are the high level steps.

• Attach the USB drive and make sure it is recognized
• Partition the drive as desired, making sure to at least create a partition for the OS and a swap partition.  I created an additional data partition.
• Format the partitions as needed
• Set up the swap partition on the hard drive
• Copy the OS partition from the SD card to the hard drive partition.  This uses a block by block copy from the SD card to the hard drive, creating a mirror image on the hard drive.
• Check the hard drive OS partition for errors.
• Resize the newly copied OS data to use the entire hard drive partition which is most likely larger than the SD card partition.
• Modify the boot file on the SD card to boot from the hard drive rather than the SD card
• Change fstab to mount the new partitions
• Prevent the system from using the swap partition on the SD card

I made all of these changes to my system and then rebooted.  It still requires the SD card to initiate the boot process, but then mounts up the hard drive and boots from there.  It booted up just fine and when I checked the root partition was on the USB hard drive.  So that step worked just fine.  I also set up Postgresql and configured it so that databases are created on the data partition I created on the USB hard drive.  Then I installed Java and the rxtx library so that my weather application can run.  I don’t really have a whole lot of work left to be able to swap computers and use the Pi for my weather station.  But I have several other unrelated activities that must take precedence over the weather station.  So it may be a little while before I switch over.  I’ll post an update when I make the switch.  Then I need to fix the humidity sensor by replacing the sensor chip once again.  I always seem to be able to find ways to stay busy.

I have decided to post my Powershell scripts that extract and upload the images, and build and upload the time lapse video.  These were some of my very first Powershell scripts, so there will probably be a lot of room for improvement.  But here they are.

The first script retrieves the current image from an HTTP server and save it in a folder named with the current date. The images are given sequence numbers.

########################################################
# Create a sequence of images retrieved from a Web server
#
# Author: Seth Cohen
# Date: 02/17/2014
#
# Input: An http server to retrieve the images.
#
# Output: a folder named YYYY-MM-DD containing a set of jpeg
#       images.  The images are named nnnnn-image.jpg where
#       nnnnn is a sequential number.
#       If the folder doesn't exist, it will be created.
#       When the day changes, a new folder will be created
#               if need be.
#
#       Note: This script must be called each time a new image
#       is to be created.  It can be run from the Windows task
#       scheduler periodically.  I currently run it daily
#       repeated every minute.
#
########################################################

# set destination folder the where images will be stored
#       Use the full path to reference it
$folderPath = "f:\Work"

# set the source URL to retrieve
$url = "http://192.168.123.150:8888/out.jpg"

# create folder name to store individual images, based on the date
$folderName = get-date -format yyyy-MM-dd

# get full path to folder using the folder path specified at the top
$fullFolderPath = join-path -path $folderPath -childpath $folderName

# create folder if it doesn't already exists
if (!(Test-Path -path $fullFolderPath))
{
New-Item $fullFolderPath -type directory
}

# create file name using a sequence number.  The sequence number is
#   calculated by counting the number of jpg files in the destination folder
#   and adding one to it.
$count = (get-childitem $fullFolderPath -filter *-image.jpg).count + 1
$fileName = "{0:D5}" -f $count
# $fileName = get-date -format HHmmss
$fileName = "$fileName-image.jpg"

# Build the file name to grab
$sourceFile = join-path -path $fullFolderPath -childpath "out.jpg"

# retrieve the image from web server using HTTP
invoke-webrequest $url -outfile $sourceFile

# move and rename file
$destPath = join-path -path $fullFolderPath -childPath $fileName
move-item $sourceFile $destPath

The next script creates the time lapse video and uploads the file to an FTP site

########################################################
# Create a time lapse movie from a set of images using ffmpeg
#
# Author: Seth Cohen
# Date: 02/17/2014
#
# Input: a folder named YYYY-MM-DD containing a set of jpeg
#       images.  The images are named nnnnn-image.jpg where
#       nnnnn is a sequential number.
#
# Output: a mpeg movie with a file name of YYYY-MM-DD.mpg
#
########################################################
# set full path to the executable to run (ffmpeg)
$executable = "path_to_ffmpeg.exe_goes_here"

# set destination path where images will be stored
$folderPath = "f:\Work"

# build yesterday's folder name
$yesterday = [DateTime]::Now.AddDays(-1)
$foldername = $yesterday.toString("yyyy-MM-dd")

# get full path to folder
$fullFolderPath = join-path -path $folderPath -childpath $folderName

# make sure yesterday's folder exists
if (Test-Path -path $fullFolderPath)
{
        # we have a folder to work with, so run ffmpeg
        $commandLineArgs = "-i $fullFolderPath\%05d-image.jpg $folderPath\$folderName.mpg"
        invoke-expression "$executable $commandLineArgs"

        # after the movie is created we can perform any clean up deemed necessary
        #   like deleting the source directory.  Do this by hand for now or use
        #   another clean up script to prune old folders

        # copy the video to the  weather web site via ftp
        $server = "your_ftp_site_goes_here"
        $user = "your_username_goes_here"
        $password = "your_password_goes_here"

        echo "open $server
                user $user $password
                binary
                lcd $folderPath
                cd public_html/images
                put $folderName.mpg time-lapse.mpg
                dir time-lapse.mpg
                quit" | ftp -n -v
}

Finally, here is the script that grabs the image from the Webcam and uploads it to Weather Underground via FTP

#################################################
# Powershell script to retrieve the current webcam image
# from Yawcam and upload it to Weather Undeground
# The image is at http://localhost:8888/out.jpg
#
# Author: Seth Cohen
# Date: 02/16/2014
#
#################################################

#Import-Module PSFTP

# first retrieve the image from yawcam
invoke-webrequest http://192.168.123.150:8888/out.jpg -outfile c:\temp\wxwebcam.jpg

# if it was successful then ftp it to the remote site
$server = "webcam.wunderground.com"
$file = "wxwebcam.jpg"
$user = "wxunderground_user_name_goes_here"
$password = "wxunderground_password_goes_here"

echo "open $server
user $user $password
binary
lcd c:\temp
put $file image.jpg
dir
quit" | ftp -n -v

Sharing my data with Weather Underground includes sharing my Webcam images for them to display alongside my weather data.  Recently I checked on their site to verify the Webcam images were being uploaded and kept seeing that my camera was disabled.  They were supposed to be grabbing the image using a URL I supplied them with.  I would check the URL I had given them for my camera image to make sure it was working.  Since it is the same URL used to display the image on my own pages I felt pretty confident that the image was available.  But every time I enabled my camera on their Web site it would become disabled within a few hours to a day.

In order to remedy this problem I chose to change from having them retrieve my camera image to me uploading it via FTP.  At least that way I would have some degree of control over when the images were being delivered.  That led to a couple of issues I needed to address.  The first was that the Webcam software I was using, Yawcam, could only do FTP to one site.  I was already using that capability to upload images to my own Web site.  That required that I create an FTP script to upload the image to Weather Underground.  After creating the script, I set Yawcam to save off the current Webcam image to a folder on my server’s hard drive.  Then I set up Windows task scheduler to run the FTP script every two minutes.  Everything looked good so I left it to run.

When I checked back later that day I noticed that the image had stopped changing on Weather Underground.  I checked my logs and saw that the FTP script was still working.  Checking my server I discovered that Yawcam had stopped updating the still image it was supposed to be capturing from the webcam.  The image being FTP’d to my Web site was fine, and the streaming video was also fine.  I restarted Yawcam and monitored the image being saved.  After several hours it stopped updating again.  This went on for the next few hours before I decided on a different approach to work around this apparent bug in Yawcam.

Yawcam has a feature where it can act as a Web server and serve up the latest image from the Webcam as a JPEG file.  I enabled this feature and set out to find a way to retrieve the image from Yawcam directly.  I ended up writing a Powershell script to retrieve the image and FTP it to the Weather Underground site.  I let it run for a day to make sure it was stable.  The image updated correctly and I was satisfied with how the script was running.

Now I started to get carried away with myself, like I tend to do.  I decided it would be nice to have a time-lapse video of the day’s webcam images.  I modified the script to create a folder using the date and save the images in that folder using a sequence number (the count of images in the folder + 1).  That permitted me to use ffmpeg to build an MPEG video of the images.  I set the script to run once per minute and save off the images in sequence.  I then wrote another script that runs ffmpeg to create the time lapse video and set it to run in the middle of the night on yesterday’s image sequence.  Finally, I had the script FTP the MPEG file it had just created to my weather Web site.  A link on my Web page permits viewing that video.  The Web site only retains one day of time lapse video.  The nightly upload overwrites the video from the prior day.  Everything appears to be running smoothly.  I still plan to write a script to clean up the sequence of images off of my hard drive after an expiration period has passed.

I ordered a new humidity sensor, as mine appears to be suspect.  I probably won’t have an opportunity to install the sensor chip on the humidity circuit board until the weather warms up a bit.  Always something to do.  I also have been preparing a Raspberry Pi to use as the new station computer.  That will come when I get some free time and motivation, or my current station computer quits working for me.  We’ll see which occurs first.

I just posted another update to the Web site software.  Several things have changed, hopefully for the better.

1) I added a configuration file for the JavaScript side of the house.  It is generated by the PHP code, which makes it a lot easier to keep the two in sync.  The new file is config/config.js.php.

2) I did some testing in other browsers and learned that the Knockout “foreach” construct was failing in some browsers.  I tracked it down to incompatibilities between knockout and jQuery.  The site now uses jQuery 1.11.0 and knockout 3.0.  I also had to add jQuery migrate 1.2.1 to the mix to keep some of the browsers happy.  Unfortunately the slide effect and a few other special effects I was using doesn’t work correctly with knockout in all browsers so I stopped using them.  All the browsers I have tested now work, although my browser list and testing is not comprehensive.

3) I revised the data model for the weather forecast to make it simpler to work with.  I updated the forecast display in both PHP and JavaScript, as well as the page that returns the JSON representation of the data.

Enjoy.  I may do some more work with the weather statistics JSON representation to only pass back the data that is actually being displayed on the We site.  That will take some analysis and isn’t a high priority at the moment.  It also looks like my humidity sensor may have an issue as it has been reporting 100% for at least a day now.  It’s too cold to do much about it right now, and the NWS is predicting 6-10 inches of snow in the next 24 hours.  I’ll keep an eye on the sensor and order parts if it keeps looking like I have a problem.

I made a few more changes to the Web site software.  The NWS has a system of color codes for all of their watches, warnings, and alerts.  So I decided to implement that system in my Web site.  Now the banner at the top of the page will display any active alerts in the color chosen by the NWS.  This isn’t critical, and sometimes the colors don’t even look that great.  But I chose to conform, at least for now.  I tested it during our most recent winter weather and the banner did display several different colors as the alerts changed from watch to advisory and warning.

I also made some modifications to the configuration file for the PHP code.  I added more comments to explain what the different parameters are for.  I also added parameters for several items that were hard coded in the application.  I’m still thinking about having a similar configuration file for the JavaScript side of things.  The best thing to do would be to have the PHP code generate the JavaScript code.  I will probably work on that in the next week or so, depending on when I can get some free time.  I also want to do some work on the mobile page as it is pretty bare bones and doesn’t really look all that great.

I posted the updated Web site code as it stands today.  If anybody does download it let me know by email.