Wednesday, August 10, 2005
Kris Eats Crow
Remember last week, when I complained about the unreliability of the coin acceptor, insisting that it must be a hardware problem because I had done everything possible in software to improve the situation?
Well . . .
A Big Meeting had been scheduled to determine what to do to move forward. In preparation, I wanted to gather extensive statistics on exactly how often the coin acceptor rejected coins, which years and countries of origin had the worst rejection rates, and other stuff like that. The coin acceptor vendor had been asking for such information as well, but I'd been putting it off because I'd already decided it was not my problem, and the hardware guys should deal with it.
I had only been given one Euro coin for my testing. So every time I tested the coin acceptor, I'd drop this coin in, then open the machine, wave my hand around in the coin hopper to find the coin, take the coin out, and then close and lock the machine before inserting the coin again. Thus, it took several seconds for each coin insertion test. I decided this was impractical, and begged for some additional coins. I was given ten more coins, five 1-Euro coins and five 2-Euro coins, giving me a total of eleven coins. I was forced to sign a receipt for the coins, as if they were afraid I'd run off to Europe with my 17-Euro fortune.
I started feeding the coins in one after another, and an interesting pattern emerged. The first one was accepted, the second was rejected, the third was accepted, the fourth was rejected, and so on. It wasn't perfect, but in general, acceptances and rejections alternated. Hmmm, that's weird. I had never seen this before, doing only one coin at a time. Now it made sense that the QA guy was seeing a 50% rejection rate on his machine—he was feeding coins in rapidly as well.
On a hunch, I put a coin in, which was accepted, then I waited ten seconds before dropping in another. It was accepted. It turned out that if I waited ten seconds between insertions, I got a 100% acceptance rate. There was apparently a period of a few seconds after acceptance of a coin when the coin acceptor would reject a subsequent coin.
A possible cause hit me right away. After acceptance of a coin, the machine calculates the new customer balance, then sends a new set of enable/disable commands to the coin acceptor telling it which coins it should accept. For example, if the machine is at the maximum balance, acceptance of all coins will be inhibited; if the machine is one Euro below the maximum, then we will accept a one-Euro coin but not a two-Euro coin. I suspected that sending these commands after every coin acceptance was somehow temporarily inhibiting acceptance.
So I changed the application code to make it stop sending all these commands. Sure enough, when I started pumping coins in again, they were all accepted. The coin acceptor was working perfectly.
I shared my findings with others in the company, inviting them to beat me with a baseball bat after wasting their time with my claims of bad hardware. I also shared my findings with the vendor, but they say that the coin acceptor should work fine even if those commands are sent after each acceptance, so I've still got more work to do to get to the bottom of it.
Lesson learned: When testing a coin acceptor, have a lot of coins on hand. I've been investigating this issue for a few weeks, but as soon as I got a handful of coins, the problem became apparent in just a few minutes.
I'd like to place all the blame on the guy who originally wrote this code, but I can't. The clues were there all along, and I just didn't put them together. I'd also like to blame the guy who decided to give me only one coin, but I should have insisted on receiving more coins sooner.
It was only when I got serious about proving it to be a hardware problem that I discovered it was a software problem after all.
(I'd like to balance this post by also describing something incredibly smart that I've done lately. Unfortunately, nothing comes to mind.)
I'm starting to think we might need an exorcist instead.
I think the vending machines sense my ill feelings toward them.