First Field Trials or Failure Reconstruction 101

This past Victoria Day weekend we had a big family get-together at my parents' cottage in Shediac, NB. Of course, AMOS (or most of AMOS) was brought along too. I needed to disassemble the solar panel and wooden framework in order to fit it into the van with all our stuff. Unfortunately in the rush to get going I left the solar panel in a box, propped against our garage wall. Half an hour into the two hour drive to Shediac I realized that it wasn't packed and was tempted to drive back for it, but everyone was anxious to get there, so I resigned myself to testing with whatever amount of battery charge there happened to be. Luckily, it turned out that the battery was fully charged and it lasted the entire weekend.

The first attempted test took place on Saturday afternoon. Everything seemed to be working up at the cottage, so my whole family, my parents, sister, brother and his family walked it down to a sandy area of the beach about 10 minutes away. Unfortunately the first test was a dud. For whatever reason the boat could no longer connect to the server program running on my PC.

So back to the cottage, for some more testing, debugging, and modifications of the Boatserver program. That evening AMOS was taken down to an area of beach nearer to the cottage, and this time some of the neighbors came out to watch too. But it was the same sad result; AMOS couldn't connect to the Boatserver program. In retrospect, I think this might have been due to the fact that it was being launched near the base of a 20 foot high rocky cliff, a location that had only spotty cell reception. This time I modified both AMOS's networking software and the BoatCaptain software running on the Android phone to disable the "Nagle" algorithm, so that small individual packets of bytes are not buffered before sending. This definitely improved the responsiveness of communications, so Sunday morning it was time for a new test. This time I decided to also try a text command file with some GPS coordinates in it. Basically just a simple track going north about 50 m, followed by south 50 m, and repeating that loop forever (or until manually stopped).  Unbeknownst to me at the time though, I had entered incorrect latitude values that were fully 1 degree more south than they should have been, i.e. about 150 km to the south of our present location. This caused AMOS to turn rapidly and veer off toward some nearby rocks. Here is a video of the unhappy result:

https://youtu.be/ITgFlJg6kpQ

The wooden platform holding the left propeller was sheared in two by the rock, but I was able to replace it using some scrap wooden flooring. After the repair another trial was attempted, this time under remote (cell phone) control:

https://youtu.be/BmbP_CeS9Ic

(photo credits to Dad for these videos!)

I was able to get the boat moving fairly fast and nearly ran myself over in the process. Although a kayak was available up at the cottage, I was nervous about losing communications and having the boat drift out to sea, so didn't try driving it too far. A couple of things worth noting from the test:

1. Even a moderate amount of wind would probably cause problems with the solar panel installed atop the wooden posts.
2. The algorithm for turning the boat at speed should be modified a bit to allow for greater differential speed between propellers when necessary.
3. The electronic compass was not being properly tilt compensated and gave noticeably bad readings of heading in the presence of waves.

This latter issue was fixed after we got back to Fredericton; the equations for compass tilt compensation that I had borrowed from an application note weren't quite right, or weren't right for the setup I had, and needed to be adjusted a bit. They seem to work well now though.

The testing was continued this morning with a repeat of the GPS waypoints test, this time at Woolastook Park on the St. John River. The day was pretty windy, with a strong wind from the south, so when I released the boat from a small cove it shot towards the first waypoint. Everything looked great! I attempted to keep up with it by following along a small, parallel trail in the woods to the east of the cove.

Looking back at the GPS data that AMOS had logged, it was clear that it hit waypoint #1 exactly as intended, and waited there for a minute, just as it was supposed to. When I was watching it from the shore, I noticed AMOS appear to reverse direction, so I had assumed (wrongly) that it was reversing direction from Waypoint #2. However, it turned out that I had made another error in the GPS waypoints file... the 2nd Waypoint was actually located close to the start, in the corner of the cove (not as shown in the image above). But AMOS was not able to follow the track back to the cove corner correctly. Looking at the log data, it did head out on a correct bearing (214 degrees) which should have taken it back to the starting location, however the wind was blowing from the south and caused AMOS to crash into the western point of land, because AMOS's software did not account for the large differential in propeller speed that would be required to properly change direction. I wasn't close enough to see how serious the impact was, but attempts to manually drive the boat with my phone totally failed. The log from the Boatserver application show that the boat lost its network connection at this time. I suspect my software running on AMOS probably also crashed at this time, since there are no entries in the ship's log file afterward. None of the Raspberry Pi system log files revealed anything about whether or not the application crashed.

So I ran back along the trail, then up onto Route 102, and back down through some scrubby trees to reach the stranded AMOS. The propellers weren't turning at all, another indication that the application had probably crashed. I didn't realize at the time that I could check using a Bluetooth link to verify whether or not AMOS's software was running, but that is something that can be tried next time. AMOS was rebooted, and I figured I would try manually driving it back to Route 102 where it could be easily picked up. But first I wanted to try manually driving the boat around out in the open water to see how difficult it was to turn in the wind. Not surprisingly, a fairly high difference in propeller speeds (40% v.s. 100%) was required to make even very large sweeping turns in the open water with all that wind blowing. In retrospect, this must have been why AMOS could not properly drive itself back to the start: it had been programmed to allow a maximum difference of only 50% between propeller speeds when trimming its heading en route to a GPS waypoint. I should probably modify the algorithm to periodically do fixed location pivots (running the propellers in reversed directions) if the GPS track starts veering too much away from the intended direction.

While driving the boat around out in the open water though, something happened that again resulted in the boat losing communications with the server. Again I suspect the software probably crashed, but I have no proof. This required running back on Route 102 to the launching point, unstrapping the kayak from the van, and paddling madly out to go fetch the boat. I found it on the north shore of the river, bobbing along in the shallows. The trip back to the van was fairly difficult, towing the boat and paddling into the wind, but I was able to get everything packed up and get back home for a quick bite of lunch, before going off to my regular job.

I'll have to repeat this test soon, and this time if communications get lost I'll pull up alongside it in a kayak and check using a Bluetooth link with the Android phone whether or not the software is still running. I should be able to add more file logging commands to AMOS's software too, so that I might get a better idea what happens immediately before a crash or loss of communications.

Route Planning and Business Planning

The printer now seems to be functioning correctly (well most of the time anyway). My lousy C+ grade in 4th year engineering control theory (24 years ago!) notwithstanding, I was able to find some software that ran an automated test on the printer and determined the optimal P, I, and D values for how to apply heat to the new heating cartridge.

This week also saw the development of a simple text file system for passing a series of commands to the boat's program on the Raspberry Pi command line. For example, the test file I was working with this evening had the following commands listed in it:

//sample boat commands script
label: start
heading: 0 //change heading to north
forward: 0:00:05 //go forward for 5 seconds
heading: 90 //change heading to east
forward: 0:00:05 //go forward for 5 seconds
heading: 180 //change heading to south
forward: 0:00:05 //go forward for 5 seconds
heading: 270 //change heading to west
forward: 0:00:05 //go forward for 5 seconds
goto: start

So passing this file to the boat's program tells it to drive the boat north for 5 seconds, then east for 5 seconds, then south for 5 seconds, followed by west for 5 seconds, theoretically ending up where you started. The "goto" statement at the end of it tells it to go back to the start, effectively driving in the same box pattern forever. Here's a video showing the reality of the above script: https://youtu.be/Gf9m5cMF4I4. Since the boat is already processing GPS data, it should be a relatively straightforward task to add some script commands for specifying GPS waypoints, i.e. just comma delimited latitude and longitude values on a line. The boat would compute the necessary heading, turn to it, and then drive forward until it reaches it. Periodic tuning of the heading angle would be required as the waypoint gets closer, but shouldn't be too hard to work out.

Since I have been neglecting / ignoring the necessary business aspects of this project for a while, I attempted to start writing a bit of a business plan, but have not gotten very far with it yet. My rough guess at what the essential development schedule looks like for the next 12 months is something as follows:

• May: Implement file-based command list for GPS self-navigation.
• June: Implement water-sampling system for 12, 100 ml samples.
• July:  Biology sampling / practice counting bacteria.
• August: Do some sampling in field / work on obstacle avoidance algorithms.
• September: Continue work on vision-based obstacle avoidance.
• October: Finish vision-based obstacle avoidance / possibly begin work on alternative obstacle avoidance technology if deemed necessary.
• November to January: Work on continuous monitoring / estimate of bacteria counts.
• February: Begin experiments comparing continuous monitoring vs. absolute counts.
• March: Finish experiments comparing continuous monitoring vs. absolute counts.  
• April: Summarize results of experiments in final report / paper.

Of course there are many smaller, perhaps non-essential development tasks to complete such as adding voltage and current monitoring electronics, cleaning up the wiring of the interior, improving the mounting of the solar panel (i.e. lowering it), improving the mounting of the thrusters (get rid of the wooden-basket mount), improving software user interfaces, etc.

At this point there are a lot of unknowns, but hopefully some aspects of this will be successful. I was thinking it would be really cool to show some or all of this technology at a conference such as "Oceans" (ex: see https://www.oceans19mtsieeemarseille.org/). Or maybe just a "Maker Faire" somewhere, I don't know. It's impossible to know for sure right now; but all of those unknowns do help to make the work exciting and fun!

Printing Problems and Wireless Woes

Before attempting to print a part with a relatively large surface area, I had neglected to do a full leveling test of the bed to ensure that the extrusion tip was neither too close nor too far from the surface of the bed. In some places, it turned out that the tip was about a millimeter or so too close to the surface (i.e. it impacted the surface). This resulted in a sort of "jam" in which the plastic broke off inside the tip and couldn't be easily retrieved. A futile attempt at retrieval only resulted in the breakage of the tip. So 5 new tips were ordered from Amazon for ~ $13. Not too bad. When those arrived, the repair went well, but I accidentally spilled coffee all over my laptop keyboard while reading the assembly instructions. The laptop which had been running at the time immediately powered off or perhaps died; I wasn't sure, and feared the worst. The usual advice in such situations is to immerse the electronic device in some sort of desiccant material or a bag of dry rice, but I feared that would not be sufficient,  and decided more drastic measures were required. I put it in our oven, closed the door and turned the heat on at the lowest possible level (around 170 deg F), then turned off the oven after about a minute or so. Our oven heats quickly, so I think for future spills I won't wait quite that long. After about 5 minutes of leaving the laptop in the turned off oven, I took it out. It was still pretty hot so I waited about 10 more minutes for it to cool and then tried powering it up. Thanks be to God, it worked!

Unfortunately the remainder of the printer repair did not go quite as well that day. After installing the new tip, I powered it up and was surprised to notice the display dim and some curls of smoke waft up into the air... I had accidentally shorted out the two heating element wires near the heating cartridge tip and they were producing a small, intense flame. I unplugged the printer, but it was too late: both heating element wires had burned through. So 5 more heating cartridges were ordered from Amazon, again for $13.

For a few days, this aura of bad luck persisted as I switched over to the wireless communications and tried to get a working cell connection between my phone and the boat. The original plan was to use the cheap Android phone inside the boat as a wireless hotspot, that the boat's Raspberry Pi could use to accept incoming network connections over the Internet; in essence it would be a sort of mobile data server. For whatever reason though, I couldn't get the hotspot feature in the Android operating system to work properly. Support calls were made to Bell and the phone manufacturer (LG) to no avail. The manufacturer believed that because the phone was a US phone, that the hotspot feature would not work in Canada. They recommended doing a factory reset on the phone, which I did, but it didn't accomplish anything. A number of things were attempted, but eventually I just gave up, and decided to use my iPhone as the boat's hotspot instead. It connected to the Raspberry Pi without any issues. I'm a little bit nervous about losing my personal phone at sea, but that's what the "Find My iPhone" app is for right?

Even with the iPhone hotspot connected to the Raspberry Pi though, the server operating on the Pi was unable to accept network connections. My theory is that either the iPhone itself or perhaps Bell was blocking the server port from receiving data. This was pretty discouraging, as I had been assuming that the code that had been written over the last few weeks while testing on WiFi could also be used over a cell connection. I'm a bit out of my element with regards to configuring communications networks, and consulting the general wisdom on Google was not particularly helpful. I sort of felt that the server needed to be moved elsewhere, so I wrote a quick server program for my laptop just this morning that acted as a sort of relay. It accepted connections from both the boat and the "captain" on land, and acted as a sort of go-between, forwarding communications packets back and forth between the two. With some slight modifications to the boat software, this actually worked perfectly! So a good end to a tough week I guess.

AMOS Takes a Dip

I spent about another day trying to get Bluetooth communications to work between AMOS and my iPhone before giving up. I think it can be done, probably using the iOS Core Bluetooth libraries for Bluetooth Low Energy (BLE) communications, but it looks like a lot more work than what I'm willing to dedicate to it at this point. I believe getting Bluetooth working between the Android phone and AMOS won't be too bad; it should be similar to what I did a couple of weeks ago, getting Bluetooth software working between the PC and AMOS.

To that end, I've started writing a version of BoatCaptain for the Android phone. There isn't much functionality in it yet, but here is a cool screen shot to look at:

The PC version of BoatCaptain also now has a stop sign button, for immediately cutting power to the propellers. It makes driving AMOS quite a bit easier, since it gives the driver a sort of "mental" reset so that they immediately know the state of the propellers (i.e. stopped) without needing to look at the propeller speed controls in the graphic user interface.

The PC version now also gets GPS, magnetometer compass, roll & pitch angles, and temperature from AMOS. These were initially set to update every 2 seconds, but it was found that this tended to cause connection problems over Bluetooth, so the interval was reduced to once every 10 seconds for Bluetooth. The data rate should be fairly small, so I'm not sure why the Bluetooth connection was adversely affected by the 2 second update rate. Internal to AMOS, GPS data arrives once every second, and the compass heading, pitch, and roll angles and temperature are collected at 50 Hz, with a 5 sample running average. So this should allow for pretty good real-time navigation later on. The compass data looks reasonable, probably to within 20 degrees or better, and I verified that the GPS latitude and longitude changed slightly when I carried the boat around the backyard.

The iceberg in our pool has almost melted. Here is a video that Kirsten took of me carefully piloting the craft around the perimeter of the berg: https://www.youtube.com/watch?v=p4Y3qhnZCIo  The interior of AMOS also now features a 2.5 kg weight velcroed to the bottom of its interior for ballast; I think it helped a little, although it could probably use even more. The boat is already a bit heavy to carry though, so I'm reluctant to add too much. After this video was made I did some more testing / fooling around and somehow lost the network connection while AMOS had its propellers set to the maximum backward speed. It gyrated a number of times around the pool, out of control, but luckily I was able to grab a hold of it when it came close with one hand, release the cap with the other hand, and then unplug the power wire. Later on I changed the software so that it should kill the motors when the connection gets lost. Should probably also add a master power switch to the exterior of the boat for easier on / off control.

Where AMOS Fits In

In contrast to last week's progress, this week seemed a lot slower. I was hoping to get the BoatCaptain PC software ported over to iOS to make it easier to control the boat from a kayak on the water. I was also hoping to integrate GPS and navigation software into BoatCaptain. Unfortunately I only partly succeeded in porting the software over to iOS and didn't even start the GPS / navigation integration. Here is a screenshot of the working version of BoatCaptain on iOS:

The code for controlling the boat over a network interface works fine, but it's the Bluetooth connectivity stuff that is giving me lots of grief. Unfortunately Apple doesn't make it simple for developers to integrate Bluetooth into their software. Apple's "CoreBluetooth" libraries for low energy Bluetooth devices look promising, but they are quite a bit different than any other Bluetooth software I have done in  the past for Android or the PC, so may require a lot of time to figure out. At this point I'm considering just setting it aside and moving on to other things. 

The North American cell modem that was back ordered and scheduled to ship this week got delayed again, so I canceled that order and decided to look for an alternative. Similar standalone cell modems seemed to be priced at ~ $300 or more, but something like a Bell or Rogers turbo stick (for getting cell data on a tablet or laptop) were less than $100, although they required signing  up for some sort of plan which I didn't look into. Since I already had the SIM card with 100 MBytes / month of data from Bell, I decided to just get an Android smart phone for a little over $100. It can be added to the boat for cell / data connectivity, communicating to the Raspberry Pi Compute module over Bluetooth. It will also afford some gigabytes of data storage.

There was a steering committee report released recently to the New Brunswick government that provides a good summary of the testing and results that have been accomplished in the Parlee Beach and surrounding area. The report itself is available here: http://www2.gnb.ca/content/dam/gnb/Departments/eco-bce/Promo/Parlee_Beach/pdfs/parlee_beach_water_quality_finalreport-e.pdf  

The results indicated that the water quality at the beach location was nearly always within the safe limits specified in the Canadian Recreational Water Quality Guidelines (https://www.canada.ca/en/health-canada/services/publications/healthy-living/guidelines-canadian-recreational-water-quality-third-edition.html). The times when the limits were exceeded seemed to be strongly correlated with wind direction (higher bacteria concentrations were measured at the beach when the wind was from the north or northwest), and possibly some weaker correlation with tides, rainfall, and beach traffic. It was also interesting to note that the bacteria levels measured in the watershed around Shediac Bay were considerably higher than those measured at the beach itself. That could be why winds from the north or northwest resulted in higher concentrations of bacteria at the beach: the bacteria from the watershed were pushed back to the beach by the wind.

So where does AMOS fit in with the testing that has been done so far and the recommendations for future work that the steering committee report described? The second recommendation that the report listed was to "Apply the hydrodynamic model to validate the transport paths related to actual discharges into the Bay (concentrations, volumes, weather conditions)." In addition to collecting samples at various locations in the Bay itself or in the surrounding rivers and streams, AMOS could be allowed to drift at various times and locations in order to estimate probable transport directions that bacteria might follow due to winds, tides, and currents. With a relatively rapid sample evaluation time (less than 12 hours vs. the current 48 hours) this might be able to provide a reasonable estimate of beach water quality, thereby accomplishing the 6th recommendation of the report: "Develop and validate a tool for predicting water quality, based on relevant environmental and meteorological data, which could be used by the Medical Officer of Health (MOH) to issue “No Swimming Advisories” at Parlee Beach. A predictive tool would address concerns with the 48-hour delay related to analyzing water quality samples". Of course, even better than a predictive tool would be a device for obtaining real-time bacteria concentrations on an ongoing, continuous basis. That is the long-term goal of AMOS.

Bluetooth and Blue Skies

A lot of progress was made this past week. After printing several parts for the solar panel framework and bolting them together, it became readily apparent that this method of construction would not result in something rigid enough to work well. So I visited Home Depot and found some 8 foot long bronze tile edge things that looked like they would be suitably stiff and lightweight when bolted to the solar panel. Here is the result with some more scrap wood from the garage added to connect it to the rest of the framework:

The solar panel is elevated above the boat significantly more than it will be in the final design; this is necessary for now so that there is lots of room to stick my big head inside the hull and make electronics modifications. The panel is held to the framework with 6 bolts and wingnuts, so isn't difficult to remove should the need ever arise. Also, I ordered some neat little waterproof connectors (~ $1 each from Amazon) and am using one of them for the solar panel wiring:

Perhaps in a future revision the solar panel could be much closer to the boat and have a sort of hinge / latch mechanism for making it easy to get inside the hull.

The client / server software that I began the week before is also coming along well. The server resides on the boat and waits for incoming network and / or Bluetooth connections. The client (for now) is just a quick and dirty PC program (called "BoatCaptain") that lets you connect over the network (WiFi at this point) or Bluetooth, and has 4 buttons for controlling propeller speed and direction:

Over time, this will expand to include GPS location, magnetic heading, temperature readings, battery voltage, route planning, sample collection, etc. Also, I would like to have a mobile version as well, since it can be tricky to kayak and work a laptop at the same time.

Our recent rainy weather has abated this morning and I just noticed that there is a relatively deep section of the stream bordering our neighbour's property that is free of ice, so I think it's time for a field test...

EDIT: And here is the video of the first field test: https://www.youtube.com/watch?v=H5xzWFsAKhc

Ready For Europe

The 4 volt power supply for the cell modem arrived last week, so on Friday night I soldered it up to the cell modem and connected it to the 5 volts from the new 7-port USB hub which had also recently arrived. The documentation for the 4 volt supply claimed that there were two 4 volt output pins on the little circuit board, so I picked one of them, but after an hour or so of frustration realized that only one of the output pins seemed to be functional. Not sure if that was a fault of the documentation or the board itself, but based on the remainder of the documentation I suspect the former. The documentation for the power supply and cell modem are scattered throughout a number of places on this site: https://itbrainpower.net/. Personally, I much prefer having a single product PDF, rather than having to flip through multiple pages on the web. It just seems so much easier to find what you need later on, if you can open that one document every time.

So once the correct 4 volt output was connected to the cell modem, some LEDs flickered on, and when the power button on the cell modem was pressed, a green light flashed on and off ~ every 1.5 seconds. Something in the documentation indicated that this was as it should be. I then downloaded and installed the Windows drivers, thinking that it might be easier to test out in Windows and confirm that it was working. Indeed a number of new serial COM ports showed up in Device Manager, but the test program the documentation referred to for Windows was not available, and did not seem to be readily available on the Internet. Oh well, back to the Pi for testing then. After a few hours of careful reading and searching, I was able to find out how to correctly (I think) set the Compute Module for use with the modem.  This involved specifying the Bell APN information, and downloading some Python serial libraries to work with the modem sample code. Running the sample code appeared promising at first, as it found the chip device number and SIM card ID, but it was unable to establish any sort of a wireless signal, and did not actually appear to make any sort of wireless connection to the Bell network.

Looking through my purchase records, it turns out that I actually bought a cell modem intended for Europe. So I'm guessing that it works at 900 MHz / 1800 MHz, instead of the frequencies used in North America (850 MHz / 1900 MHz). There is a North American model available, so I ordered it and should have it in a couple of weeks. Hopefully that is the only other obstacle remaining before getting a working cell connection. Maybe the European modem will be useful someday if I ever sell anything overseas!

In the meantime, I can use WiFi for network connectivity software and testing purposes. I'm still working on some client / server software for communications between AMOS and a PC (or phone) and hope to have a simple version of it ready before the massive block of snow and ice in our pool melts. I noticed that there are large sections of the St. John River that are free from ice, but I'm too nervous to do any emergency kayaking at this time of year :-).