Timber Spider

Getting there first

My parents are part of a timeshare pool in which they can trade weeks at their location for weeks at other places in the pool. Weeks are submitted to the pool by other owners, at semi-random, and are reserved on a first-come-first-serve basis. Some places are reserved more quickly than others. My dad checks the website that lists the locations almost every day. I thought... why not every hour? It was decided to send an MMS any time a really good week was added, and an email anytime a week of any quality was added or removed. Hopefully by getting alerts sent to our phones we could secure our favorite weeks before anyone else.

Making a robospider

After a few iterations I decided that scrapy was the right tool. It has a lot of stackoverflow stuff, a bunch of automation, processing pipelines, etc.

The availability website is behind a password protected login so I needed to use a spider that could handle the authentication step. I decided to use the InitSpider as it handles authentication before crawling.

Next I told the spider to seek out the "next >" links on the page for creating followup requests. The availabilities are listed in a standard table, so I parse each row as a scrapy Item. I set up an item processing pipeline and activated it to do the alerting.

Alert system

Items gleaned by the spider are passed through a pipeline which assembles them into a dictionary and compares them to an older dictionary for a difference. The older dictionary is shelved when the spider is closed. I decided to shelve the dictionaries as it seemed really easy to do and I thought doing something like a JSON export was overkill. Turns out shelving is really easy (except when it's not).

The comparison creates three lists: additional really good weeks, new weeks, and removed weeks. Later in the pipeline the text message and email is formatted. Both required setup of a dummy gmail account to send to my family. 

Installing and Automating

I had to get scrapy installed in a conda environment (turned out to be a problem) on my pi. I made sure to install the following before attempting to install scrapy.

sudo apt-get install libffi-dev
sudo apt-get install libxml2-dev
sudo apt-get install libxslt1-dev
sudo apt-get install python-dev

It took a while for lxml to build on my little pi (~20 minutes). But after that step scrapy seemed to be available. I loaded my spider and hit go and ran smack into my first issue.

Traceback (most recent call last):

  File "/home/pi/berryconda2/lib/python2.7/site-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks

    result = g.send(result)

  File "/home/pi/berryconda2/lib/python2.7/site-packages/scrapy/crawler.py", line 98, in crawl

    six.reraise(*exc_info)

  File "/home/pi/berryconda2/lib/python2.7/site-packages/scrapy/crawler.py", line 82, in crawl

    yield self.engine.open_spider(self.spider, start_requests)

ImportError: No module named _bsddb

OK so I gotta figure out what this bsddb thing is. Apparently it's some BerkelyDB interface that twisted appears to use.

Getting bsddb3

First thing was to get the BerkeleyDB built for my pi. I followed the install instructions here:

https://github.com/dpapathanasiou/pyBDB

Make sure you install the older version of BDB (5.3.28)

Actually I tried to just pip install bsddb3 but got smacked down.

After a while I decided to drop shelve, as soon as it seemed like it might be causing the bsddb dependency. After dropping it I encountered a new error, ECC Curve.

I went to the Twisted IRC after bashing my skull against the problem for a while and someone confirmed there was some missing stuff from my pyopenssl build. They said they'd try and reproduce the issue. So I waited for a bit.

Forgetting bsddb3, something else is wrong

I tried updating cryptography, twisted, and pyopenssl to the most recent versions I could. I did everything short of updating openssl for the pi to 1.1 (I am was running Jessie). Nothing seemed to work. So I hopped into the IRC for twisted and got some help from runciter. After running some tests they concluded conda was my problem. Specifically berryconda and my old openssl libs. I need to bump to 1.1 but I couldn't easily do that with Jessie. So I decided to upgrade to Stretch. After upgrading to Stretch and uninstalling scrapy, twisted, pyopenssl, and cryptography I installed scrapy again and voila, it works.

Wrapping it up

Last step was to add a simple cron job to execute my script on the hour from 6-23. 

Electronic Component Organizer Android App and Other Progress

On a roll

Lately I've taken it upon myself to really try and push to get this project done. It's been over a year or two in the making and I need to finish it. Unfortunately I keep getting sidetracked by extra things I want to do with it.

For instance, this week I had the idea of using my fancy new bluetooth receiver to serve as the wireless connection to the component organizer. I found some cool projects that people have worked on that provide ways to transfer data from an Android phone to an Arduino using FSK over the audio port (with music as the carrier?). 

So after a quick shopping spree at SF, I have some new toys shipping to me presently. My idea will be to do a pass-thru of the audio signal from the bluetooth receiver, through an Arduino based "box." This box will use an Arduino (or PIC?) to process the signal, and send the results to the box system via 315MHz radio. The results will either be an audio waveform FFT, or the address of a part. So when the box is idle the screen saver will be the FFT of the music passing through the receiver. If an address is requested, the request will be passed via FSK to the "analyzer" and then passed to the organizer.

And just when I thought I was finally finishing this thing...

Django Database and Dajaxice

I did manage to get my Django system running pretty much how I want. I am lazy and didn't want to build my own views for Django when the admin views were all I would ever need, and look so good. So I set to learning about how to modify the Django admin site (extend is the technical word). The framework is pretty easy to deal with. I managed to an ajax based button inside the change_list_results list. And through a bit of hacking managed to get that button to open a serial connection to the organizer Arduino, and send the address of the part. Once that system was running smoothly, I set about organizing my resistors.

Two movies (The Hobbit and There Will Be Blood), and a bunch nachos later and I finally have all my resistors organized. That's not to say I have all the standard 5% resistors in this box system (that would be too many even with each drawer divided into 2), but all the ones I do have are now organized. I don't have to wonder if I have any given resistor because I have then all inventoried.

Looking back, I feel like this project has been one of the most tedious I have ever undertaken. It is taking a lot of will power to not give up until every last resistor I own is properly organized. I mean... I think I went kinda crazy there for a while. Who pays that much attention to their resistor collection? I'm just glad I got it over with...

Android Studio, and Ubuntu 64bit

The next phase of this project is to get the voice search functioning. I am going to use my Android phone for the approach. The idea is to write an app that will use google's speech input API to interpret my speech. Then using tags for each entry, some manually entered, some automatically associated based on component data, search the database for the part. Once a part or list of possible parts is found, provide a "light it up" button that shows where the part is located. Upon clicking the button an address request is sent via the audio connection over bluetooth to the receiver. On it's way out of the bluetooth receiver the audio signal is picked up by the Arduino receiver which is programmed to recognize the FSK message. It relays the address via 315MHz radio to the orgranizer which then lights up the box.

Getting the app ready starts with learning how to write apps for Android, and getting my IDE all set up. I chose to go with Android Studio, because Google's IO conference debut of the IDE looked amazing.

After downloading and "installing" I started it up and started working through a tutorial on the speech recognition API. Android studio has this nice auto compile feature that tells you if your code works. When my setup tried to compile I ran into some issues which I posted, then solved, here.