Tuesday, May 29, 2018

AMOS Gets a Makeover

The code for AMOS's GPS navigation was tweaked after last week's blog post, and the navigation test at Woolastook was repeated. This time, I think AMOS properly went to each of the GPS waypoints, although unfortunately I don't have definitive proof, since a bug introduced in the software prevented the minute-by-minute saving of GPS coordinates to the ship log file. Although the track of the boat was faster, with less stopping to check bearings, it was still weaving back and forth too much, basically over-compensating the thrust in each direction to maintain a particular heading:


So the software was tweaked some more, the log file bug for GPS coordinates was fixed, and I think it should work a lot better now, but when making some mechanical alterations to the boat this past weekend, I had forgotten to disconnect the boat's +5 and +12 V power supplies. Since there remain a number of small circuit boards that are still loose in the "hold" so to speak, these tend to slide around a bit when I'm rotating the hull, mounting propellers, etc. At first I thought the burning, toxic smell was some chemical that Kelly had added to our pool (I was working on the pool deck at the time) but then the hard truth hit, that some unfortunate circuit had shorted out inside of AMOS. This time it was the GPS circuit board. What appeared to have been a power transistor on the GPS board had been reduced to blackened ash. I'm guessing that one of the GPS power rails probably shorted out on a nearby propeller speed circuit board. So the loss of GPS has prevented field navigation tests for a few days, but a new board is on its way from China and will hopefully be here by week's end.

Speaking of mechanical alterations, check out the makeover that AMOS has undergone this week:

The wooden framework was replaced by threaded rods, nuts, washers, 4 U-bolts, aluminum sheet metal, and lots of JB Weld Epoxy. Accessing the hatch now requires removing 4 wingnuts, but I think a latching mechanism connected between the front two threaded rods, should allow someone to more easily lift the solar panel. Having the propellers closer to the center and a bit further apart has made the boat a bit touchier for turning, and will require some additional refinement to the navigation software.

Tuesday, May 22, 2018

First Field Trials or Failure Reconstruction 101

This past Victoria Day weekend we had a big family get-together at my parents' cottage in Shediac, NB. Of course, AMOS (or most of AMOS) was brought along too. I needed to disassemble the solar panel and wooden framework in order to fit it into the van with all our stuff. Unfortunately in the rush to get going I left the solar panel in a box, propped against our garage wall. Half an hour into the two hour drive to Shediac I realized that it wasn't packed and was tempted to drive back for it, but everyone was anxious to get there, so I resigned myself to testing with whatever amount of battery charge there happened to be. Luckily, it turned out that the battery was fully charged and it lasted the entire weekend.

The first attempted test took place on Saturday afternoon. Everything seemed to be working up at the cottage, so my whole family, my parents, sister, brother and his family walked it down to a sandy area of the beach about 10 minutes away. Unfortunately the first test was a dud. For whatever reason the boat could no longer connect to the server program running on my PC.

So back to the cottage, for some more testing, debugging, and modifications of the Boatserver program. That evening AMOS was taken down to an area of beach nearer to the cottage, and this time some of the neighbors came out to watch too. But it was the same sad result; AMOS couldn't connect to the Boatserver program. In retrospect, I think this might have been due to the fact that it was being launched near the base of a 20 foot high rocky cliff, a location that had only spotty cell reception. This time I modified both AMOS's networking software and the BoatCaptain software running on the Android phone to disable the "Nagle" algorithm, so that small individual packets of bytes are not buffered before sending. This definitely improved the responsiveness of communications, so Sunday morning it was time for a new test. This time I decided to also try a text command file with some GPS coordinates in it. Basically just a simple track going north about 50 m, followed by south 50 m, and repeating that loop forever (or until manually stopped).  Unbeknownst to me at the time though, I had entered incorrect latitude values that were fully 1 degree more south than they should have been, i.e. about 150 km to the south of our present location. This caused AMOS to turn rapidly and veer off toward some nearby rocks. Here is a video of the unhappy result:


The wooden platform holding the left propeller was sheared in two by the rock, but I was able to replace it using some scrap wooden flooring. After the repair another trial was attempted, this time under remote (cell phone) control:


(photo credits to Dad for these videos!)

I was able to get the boat moving fairly fast and nearly ran myself over in the process. Although a kayak was available up at the cottage, I was nervous about losing communications and having the boat drift out to sea, so didn't try driving it too far. A couple of things worth noting from the test:

1. Even a moderate amount of wind would probably cause problems with the solar panel installed atop the wooden posts.
2. The algorithm for turning the boat at speed should be modified a bit to allow for greater differential speed between propellers when necessary.
3. The electronic compass was not being properly tilt compensated and gave noticeably bad readings of heading in the presence of waves.

This latter issue was fixed after we got back to Fredericton; the equations for compass tilt compensation that I had borrowed from an application note weren't quite right, or weren't right for the setup I had, and needed to be adjusted a bit. They seem to work well now though.

The testing was continued this morning with a repeat of the GPS waypoints test, this time at Woolastook Park on the St. John River. The day was pretty windy, with a strong wind from the south, so when I released the boat from a small cove it shot towards the first waypoint. Everything looked great! I attempted to keep up with it by following along a small, parallel trail in the woods to the east of the cove.

Looking back at the GPS data that AMOS had logged, it was clear that it hit waypoint #1 exactly as intended, and waited there for a minute, just as it was supposed to. When I was watching it from the shore, I noticed AMOS appear to reverse direction, so I had assumed (wrongly) that it was reversing direction from Waypoint #2. However, it turned out that I had made another error in the GPS waypoints file... the 2nd Waypoint was actually located close to the start, in the corner of the cove (not as shown in the image above). But AMOS was not able to follow the track back to the cove corner correctly. Looking at the log data, it did head out on a correct bearing (214 degrees) which should have taken it back to the starting location, however the wind was blowing from the south and caused AMOS to crash into the western point of land, because AMOS's software did not account for the large differential in propeller speed that would be required to properly change direction. I wasn't close enough to see how serious the impact was, but attempts to manually drive the boat with my phone totally failed. The log from the Boatserver application show that the boat lost its network connection at this time. I suspect my software running on AMOS probably also crashed at this time, since there are no entries in the ship's log file afterward. None of the Raspberry Pi system log files revealed anything about whether or not the application crashed.

So I ran back along the trail, then up onto Route 102, and back down through some scrubby trees to reach the stranded AMOS. The propellers weren't turning at all, another indication that the application had probably crashed. I didn't realize at the time that I could check using a Bluetooth link to verify whether or not AMOS's software was running, but that is something that can be tried next time. AMOS was rebooted, and I figured I would try manually driving it back to Route 102 where it could be easily picked up. But first I wanted to try manually driving the boat around out in the open water to see how difficult it was to turn in the wind. Not surprisingly, a fairly high difference in propeller speeds (40% v.s. 100%) was required to make even very large sweeping turns in the open water with all that wind blowing. In retrospect, this must have been why AMOS could not properly drive itself back to the start: it had been programmed to allow a maximum difference of only 50% between propeller speeds when trimming its heading en route to a GPS waypoint. I should probably modify the algorithm to periodically do fixed location pivots (running the propellers in reversed directions) if the GPS track starts veering too much away from the intended direction.

While driving the boat around out in the open water though, something happened that again resulted in the boat losing communications with the server. Again I suspect the software probably crashed, but I have no proof. This required running back on Route 102 to the launching point, unstrapping the kayak from the van, and paddling madly out to go fetch the boat. I found it on the north shore of the river, bobbing along in the shallows. The trip back to the van was fairly difficult, towing the boat and paddling into the wind, but I was able to get everything packed up and get back home for a quick bite of lunch, before going off to my regular job.

I'll have to repeat this test soon, and this time if communications get lost I'll pull up alongside it in a kayak and check using a Bluetooth link with the Android phone whether or not the software is still running. I should be able to add more file logging commands to AMOS's software too, so that I might get a better idea what happens immediately before a crash or loss of communications.

Tuesday, May 15, 2018

Route Planning and Business Planning

The printer now seems to be functioning correctly (well most of the time anyway). My lousy C+ grade in 4th year engineering control theory (24 years ago!) notwithstanding, I was able to find some software that ran an automated test on the printer and determined the optimal P, I, and D values for how to apply heat to the new heating cartridge.

This week also saw the development of a simple text file system for passing a series of commands to the boat's program on the Raspberry Pi command line. For example, the test file I was working with this evening had the following commands listed in it:

//sample boat commands script
label: start
heading: 0 //change heading to north
forward: 0:00:05 //go forward for 5 seconds
heading: 90 //change heading to east
forward: 0:00:05 //go forward for 5 seconds
heading: 180 //change heading to south
forward: 0:00:05 //go forward for 5 seconds
heading: 270 //change heading to west
forward: 0:00:05 //go forward for 5 seconds
goto: start

So passing this file to the boat's program tells it to drive the boat north for 5 seconds, then east for 5 seconds, then south for 5 seconds, followed by west for 5 seconds, theoretically ending up where you started. The "goto" statement at the end of it tells it to go back to the start, effectively driving in the same box pattern forever. Here's a video showing the reality of the above script: https://youtu.be/Gf9m5cMF4I4. Since the boat is already processing GPS data, it should be a relatively straightforward task to add some script commands for specifying GPS waypoints, i.e. just comma delimited latitude and longitude values on a line. The boat would compute the necessary heading, turn to it, and then drive forward until it reaches it. Periodic tuning of the heading angle would be required as the waypoint gets closer, but shouldn't be too hard to work out.

Since I have been neglecting / ignoring the necessary business aspects of this project for a while, I attempted to start writing a bit of a business plan, but have not gotten very far with it yet. My rough guess at what the essential development schedule looks like for the next 12 months is something as follows:
  • May: Implement file-based command list for GPS self-navigation.
  • June: Implement water-sampling system for 12, 100 ml samples.
  • July:  Biology sampling / practice counting bacteria.
  • August: Do some sampling in field / work on obstacle avoidance algorithms.
  • September: Continue work on vision-based obstacle avoidance.
  • October: Finish vision-based obstacle avoidance / possibly begin work on alternative obstacle avoidance technology if deemed necessary.
  • November to January: Work on continuous monitoring / estimate of bacteria counts.
  • February: Begin experiments comparing continuous monitoring vs. absolute counts.
  • March: Finish experiments comparing continuous monitoring vs. absolute counts.  
  • April: Summarize results of experiments in final report / paper.
Of course there are many smaller, perhaps non-essential development tasks to complete such as adding voltage and current monitoring electronics, cleaning up the wiring of the interior, improving the mounting of the solar panel (i.e. lowering it), improving the mounting of the thrusters (get rid of the wooden-basket mount), improving software user interfaces, etc.

At this point there are a lot of unknowns, but hopefully some aspects of this will be successful. I was thinking it would be really cool to show some or all of this technology at a conference such as "Oceans" (ex: see https://www.oceans19mtsieeemarseille.org/). Or maybe just a "Maker Faire" somewhere, I don't know. It's impossible to know for sure right now; but all of those unknowns do help to make the work exciting and fun!

Tuesday, May 8, 2018

Printing Problems and Wireless Woes

Before attempting to print a part with a relatively large surface area, I had neglected to do a full leveling test of the bed to ensure that the extrusion tip was neither too close nor too far from the surface of the bed. In some places, it turned out that the tip was about a millimeter or so too close to the surface (i.e. it impacted the surface). This resulted in a sort of "jam" in which the plastic broke off inside the tip and couldn't be easily retrieved. A futile attempt at retrieval only resulted in the breakage of the tip. So 5 new tips were ordered from Amazon for ~ $13. Not too bad. When those arrived, the repair went well, but I accidentally spilled coffee all over my laptop keyboard while reading the assembly instructions. The laptop which had been running at the time immediately powered off or perhaps died; I wasn't sure, and feared the worst. The usual advice in such situations is to immerse the electronic device in some sort of desiccant material or a bag of dry rice, but I feared that would not be sufficient,  and decided more drastic measures were required. I put it in our oven, closed the door and turned the heat on at the lowest possible level (around 170 deg F), then turned off the oven after about a minute or so. Our oven heats quickly, so I think for future spills I won't wait quite that long. After about 5 minutes of leaving the laptop in the turned off oven, I took it out. It was still pretty hot so I waited about 10 more minutes for it to cool and then tried powering it up. Thanks be to God, it worked!

Unfortunately the remainder of the printer repair did not go quite as well that day. After installing the new tip, I powered it up and was surprised to notice the display dim and some curls of smoke waft up into the air... I had accidentally shorted out the two heating element wires near the heating cartridge tip and they were producing a small, intense flame. I unplugged the printer, but it was too late: both heating element wires had burned through. So 5 more heating cartridges were ordered from Amazon, again for $13.

For a few days, this aura of bad luck persisted as I switched over to the wireless communications and tried to get a working cell connection between my phone and the boat. The original plan was to use the cheap Android phone inside the boat as a wireless hotspot, that the boat's Raspberry Pi could use to accept incoming network connections over the Internet; in essence it would be a sort of mobile data server. For whatever reason though, I couldn't get the hotspot feature in the Android operating system to work properly. Support calls were made to Bell and the phone manufacturer (LG) to no avail. The manufacturer believed that because the phone was a US phone, that the hotspot feature would not work in Canada. They recommended doing a factory reset on the phone, which I did, but it didn't accomplish anything. A number of things were attempted, but eventually I just gave up, and decided to use my iPhone as the boat's hotspot instead. It connected to the Raspberry Pi without any issues. I'm a little bit nervous about losing my personal phone at sea, but that's what the "Find My iPhone" app is for right?

Even with the iPhone hotspot connected to the Raspberry Pi though, the server operating on the Pi was unable to accept network connections. My theory is that either the iPhone itself or perhaps Bell was blocking the server port from receiving data. This was pretty discouraging, as I had been assuming that the code that had been written over the last few weeks while testing on WiFi could also be used over a cell connection. I'm a bit out of my element with regards to configuring communications networks, and consulting the general wisdom on Google was not particularly helpful. I sort of felt that the server needed to be moved elsewhere, so I wrote a quick server program for my laptop just this morning that acted as a sort of relay. It accepted connections from both the boat and the "captain" on land, and acted as a sort of go-between, forwarding communications packets back and forth between the two. With some slight modifications to the boat software, this actually worked perfectly! So a good end to a tough week I guess.

Tuesday, May 1, 2018

AMOS Takes a Dip

I spent about another day trying to get Bluetooth communications to work between AMOS and my iPhone before giving up. I think it can be done, probably using the iOS Core Bluetooth libraries for Bluetooth Low Energy (BLE) communications, but it looks like a lot more work than what I'm willing to dedicate to it at this point. I believe getting Bluetooth working between the Android phone and AMOS won't be too bad; it should be similar to what I did a couple of weeks ago, getting Bluetooth software working between the PC and AMOS.

To that end, I've started writing a version of BoatCaptain for the Android phone. There isn't much functionality in it yet, but here is a cool screen shot to look at:

The PC version of BoatCaptain also now has a stop sign button, for immediately cutting power to the propellers. It makes driving AMOS quite a bit easier, since it gives the driver a sort of "mental" reset so that they immediately know the state of the propellers (i.e. stopped) without needing to look at the propeller speed controls in the graphic user interface.

The PC version now also gets GPS, magnetometer compass, roll & pitch angles, and temperature from AMOS. These were initially set to update every 2 seconds, but it was found that this tended to cause connection problems over Bluetooth, so the interval was reduced to once every 10 seconds for Bluetooth. The data rate should be fairly small, so I'm not sure why the Bluetooth connection was adversely affected by the 2 second update rate. Internal to AMOS, GPS data arrives once every second, and the compass heading, pitch, and roll angles and temperature are collected at 50 Hz, with a 5 sample running average. So this should allow for pretty good real-time navigation later on. The compass data looks reasonable, probably to within 20 degrees or better, and I verified that the GPS latitude and longitude changed slightly when I carried the boat around the backyard.

The iceberg in our pool has almost melted. Here is a video that Kirsten took of me carefully piloting the craft around the perimeter of the berg: https://www.youtube.com/watch?v=p4Y3qhnZCIo  The interior of AMOS also now features a 2.5 kg weight velcroed to the bottom of its interior for ballast; I think it helped a little, although it could probably use even more. The boat is already a bit heavy to carry though, so I'm reluctant to add too much. After this video was made I did some more testing / fooling around and somehow lost the network connection while AMOS had its propellers set to the maximum backward speed. It gyrated a number of times around the pool, out of control, but luckily I was able to grab a hold of it when it came close with one hand, release the cap with the other hand, and then unplug the power wire. Later on I changed the software so that it should kill the motors when the connection gets lost. Should probably also add a master power switch to the exterior of the boat for easier on / off control.