Tuesday, May 22, 2018

First Field Trials or Failure Reconstruction 101

This past Victoria Day weekend we had a big family get-together at my parents' cottage in Shediac, NB. Of course, AMOS (or most of AMOS) was brought along too. I needed to disassemble the solar panel and wooden framework in order to fit it into the van with all our stuff. Unfortunately in the rush to get going I left the solar panel in a box, propped against our garage wall. Half an hour into the two hour drive to Shediac I realized that it wasn't packed and was tempted to drive back for it, but everyone was anxious to get there, so I resigned myself to testing with whatever amount of battery charge there happened to be. Luckily, it turned out that the battery was fully charged and it lasted the entire weekend.

The first attempted test took place on Saturday afternoon. Everything seemed to be working up at the cottage, so my whole family, my parents, sister, brother and his family walked it down to a sandy area of the beach about 10 minutes away. Unfortunately the first test was a dud. For whatever reason the boat could no longer connect to the server program running on my PC.

So back to the cottage, for some more testing, debugging, and modifications of the Boatserver program. That evening AMOS was taken down to an area of beach nearer to the cottage, and this time some of the neighbors came out to watch too. But it was the same sad result; AMOS couldn't connect to the Boatserver program. In retrospect, I think this might have been due to the fact that it was being launched near the base of a 20 foot high rocky cliff, a location that had only spotty cell reception. This time I modified both AMOS's networking software and the BoatCaptain software running on the Android phone to disable the "Nagle" algorithm, so that small individual packets of bytes are not buffered before sending. This definitely improved the responsiveness of communications, so Sunday morning it was time for a new test. This time I decided to also try a text command file with some GPS coordinates in it. Basically just a simple track going north about 50 m, followed by south 50 m, and repeating that loop forever (or until manually stopped).  Unbeknownst to me at the time though, I had entered incorrect latitude values that were fully 1 degree more south than they should have been, i.e. about 150 km to the south of our present location. This caused AMOS to turn rapidly and veer off toward some nearby rocks. Here is a video of the unhappy result:

https://youtu.be/ITgFlJg6kpQ

The wooden platform holding the left propeller was sheared in two by the rock, but I was able to replace it using some scrap wooden flooring. After the repair another trial was attempted, this time under remote (cell phone) control:

https://youtu.be/BmbP_CeS9Ic

(photo credits to Dad for these videos!)

I was able to get the boat moving fairly fast and nearly ran myself over in the process. Although a kayak was available up at the cottage, I was nervous about losing communications and having the boat drift out to sea, so didn't try driving it too far. A couple of things worth noting from the test:

1. Even a moderate amount of wind would probably cause problems with the solar panel installed atop the wooden posts.
2. The algorithm for turning the boat at speed should be modified a bit to allow for greater differential speed between propellers when necessary.
3. The electronic compass was not being properly tilt compensated and gave noticeably bad readings of heading in the presence of waves.

This latter issue was fixed after we got back to Fredericton; the equations for compass tilt compensation that I had borrowed from an application note weren't quite right, or weren't right for the setup I had, and needed to be adjusted a bit. They seem to work well now though.

The testing was continued this morning with a repeat of the GPS waypoints test, this time at Woolastook Park on the St. John River. The day was pretty windy, with a strong wind from the south, so when I released the boat from a small cove it shot towards the first waypoint. Everything looked great! I attempted to keep up with it by following along a small, parallel trail in the woods to the east of the cove.


Looking back at the GPS data that AMOS had logged, it was clear that it hit waypoint #1 exactly as intended, and waited there for a minute, just as it was supposed to. When I was watching it from the shore, I noticed AMOS appear to reverse direction, so I had assumed (wrongly) that it was reversing direction from Waypoint #2. However, it turned out that I had made another error in the GPS waypoints file... the 2nd Waypoint was actually located close to the start, in the corner of the cove (not as shown in the image above). But AMOS was not able to follow the track back to the cove corner correctly. Looking at the log data, it did head out on a correct bearing (214 degrees) which should have taken it back to the starting location, however the wind was blowing from the south and caused AMOS to crash into the western point of land, because AMOS's software did not account for the large differential in propeller speed that would be required to properly change direction. I wasn't close enough to see how serious the impact was, but attempts to manually drive the boat with my phone totally failed. The log from the Boatserver application show that the boat lost its network connection at this time. I suspect my software running on AMOS probably also crashed at this time, since there are no entries in the ship's log file afterward. None of the Raspberry Pi system log files revealed anything about whether or not the application crashed.

So I ran back along the trail, then up onto Route 102, and back down through some scrubby trees to reach the stranded AMOS. The propellers weren't turning at all, another indication that the application had probably crashed. I didn't realize at the time that I could check using a Bluetooth link to verify whether or not AMOS's software was running, but that is something that can be tried next time. AMOS was rebooted, and I figured I would try manually driving it back to Route 102 where it could be easily picked up. But first I wanted to try manually driving the boat around out in the open water to see how difficult it was to turn in the wind. Not surprisingly, a fairly high difference in propeller speeds (40% v.s. 100%) was required to make even very large sweeping turns in the open water with all that wind blowing. In retrospect, this must have been why AMOS could not properly drive itself back to the start: it had been programmed to allow a maximum difference of only 50% between propeller speeds when trimming its heading en route to a GPS waypoint. I should probably modify the algorithm to periodically do fixed location pivots (running the propellers in reversed directions) if the GPS track starts veering too much away from the intended direction.

While driving the boat around out in the open water though, something happened that again resulted in the boat losing communications with the server. Again I suspect the software probably crashed, but I have no proof. This required running back on Route 102 to the launching point, unstrapping the kayak from the van, and paddling madly out to go fetch the boat. I found it on the north shore of the river, bobbing along in the shallows. The trip back to the van was fairly difficult, towing the boat and paddling into the wind, but I was able to get everything packed up and get back home for a quick bite of lunch, before going off to my regular job.

I'll have to repeat this test soon, and this time if communications get lost I'll pull up alongside it in a kayak and check using a Bluetooth link with the Android phone whether or not the software is still running. I should be able to add more file logging commands to AMOS's software too, so that I might get a better idea what happens immediately before a crash or loss of communications.

No comments:

Post a Comment