Sunday, January 7, 2018

Timber Spider

Getting there first

My parents are part of a timeshare pool in which they can trade weeks at their location for weeks at other places in the pool. Weeks are submitted to the pool by other owners, at semi-random, and are reserved on a first-come-first-serve basis. Some places are reserved more quickly than others. My dad checks the website that lists the locations almost every day. I thought... why not every hour? It was decided to send an MMS any time a really good week was added, and an email anytime a week of any quality was added or removed. Hopefully by getting alerts sent to our phones we could secure our favorite weeks before anyone else.

Making a robospider

After a few iterations I decided that scrapy was the right tool. It has a lot of stackoverflow stuff, a bunch of automation, processing pipelines, etc.
The availability website is behind a password protected login so I needed to use a spider that could handle the authentication step. I decided to use the InitSpider as it handles authentication before crawling.
Next I told the spider to seek out the "next >" links on the page for creating followup requests. The availabilities are listed in a standard table, so I parse each row as a scrapy Item. I set up an item processing pipeline and activated it to do the alerting.

Alert system

Items gleaned by the spider are passed through a pipeline which assembles them into a dictionary and compares them to an older dictionary for a difference. The older dictionary is shelved when the spider is closed. I decided to shelve the dictionaries as it seemed really easy to do and I thought doing something like a JSON export was overkill. Turns out shelving is really easy (except when it's not).
The comparison creates three lists: additional really good weeks, new weeks, and removed weeks. Later in the pipeline the text message and email is formatted. Both required setup of a dummy gmail account to send to my family. 

Installing and Automating

I had to get scrapy installed in a conda environment (turned out to be a problem) on my pi. I made sure to install the following before attempting to install scrapy.
sudo apt-get install libffi-dev
sudo apt-get install libxml2-dev
sudo apt-get install libxslt1-dev
sudo apt-get install python-dev

It took a while for lxml to build on my little pi (~20 minutes). But after that step scrapy seemed to be available. I loaded my spider and hit go and ran smack into my first issue.

Traceback (most recent call last):
  File "/home/pi/berryconda2/lib/python2.7/site-packages/twisted/internet/", line 1386, in _inlineCallbacks
    result = g.send(result)
  File "/home/pi/berryconda2/lib/python2.7/site-packages/scrapy/", line 98, in crawl
  File "/home/pi/berryconda2/lib/python2.7/site-packages/scrapy/", line 82, in crawl
    yield self.engine.open_spider(self.spider, start_requests)
ImportError: No module named _bsddb

OK so I gotta figure out what this bsddb thing is. Apparently it's some BerkelyDB interface that twisted appears to use.

Getting bsddb3

First thing was to get the BerkeleyDB built for my pi. I followed the install instructions here:

Make sure you install the older version of BDB (5.3.28)

Actually I tried to just pip install bsddb3 but got smacked down.

After a while I decided to drop shelve, as soon as it seemed like it might be causing the bsddb dependency. After dropping it I encountered a new error, ECC Curve.

I went to the Twisted IRC after bashing my skull against the problem for a while and someone confirmed there was some missing stuff from my pyopenssl build. They said they'd try and reproduce the issue. So I waited for a bit.

Forgetting bsddb3, something else is wrong

I tried updating cryptography, twisted, and pyopenssl to the most recent versions I could. I did everything short of updating openssl for the pi to 1.1 (I am was running Jessie). Nothing seemed to work. So I hopped into the IRC for twisted and got some help from runciter. After running some tests they concluded conda was my problem. Specifically berryconda and my old openssl libs. I need to bump to 1.1 but I couldn't easily do that with Jessie. So I decided to upgrade to Stretch. After upgrading to Stretch and uninstalling scrapy, twisted, pyopenssl, and cryptography I installed scrapy again and voila, it works.

Wrapping it up

Last step was to add a simple cron job to execute my script on the hour from 6-23. 

Friday, January 5, 2018

Electronic Component Organizer Android App and Other Progress

On a roll

Lately I've taken it upon myself to really try and push to get this project done. It's been over a year or two in the making and I need to finish it. Unfortunately I keep getting sidetracked by extra things I want to do with it.

For instance, this week I had the idea of using my fancy new bluetooth receiver to serve as the wireless connection to the component organizer. I found some cool projects that people have worked on that provide ways to transfer data from an Android phone to an Arduino using FSK over the audio port (with music as the carrier?). 

So after a quick shopping spree at SF, I have some new toys shipping to me presently. My idea will be to do a pass-thru of the audio signal from the bluetooth receiver, through an Arduino based "box." This box will use an Arduino (or PIC?) to process the signal, and send the results to the box system via 315MHz radio. The results will either be an audio waveform FFT, or the address of a part. So when the box is idle the screen saver will be the FFT of the music passing through the receiver. If an address is requested, the request will be passed via FSK to the "analyzer" and then passed to the organizer.

And just when I thought I was finally finishing this thing...

Django Database and Dajaxice

I did manage to get my Django system running pretty much how I want. I am lazy and didn't want to build my own views for Django when the admin views were all I would ever need, and look so good. So I set to learning about how to modify the Django admin site (extend is the technical word). The framework is pretty easy to deal with. I managed to an ajax based button inside the change_list_results list. And through a bit of hacking managed to get that button to open a serial connection to the organizer Arduino, and send the address of the part. Once that system was running smoothly, I set about organizing my resistors.

Two movies (The Hobbit and There Will Be Blood), and a bunch nachos later and I finally have all my resistors organized. That's not to say I have all the standard 5% resistors in this box system (that would be too many even with each drawer divided into 2), but all the ones I do have are now organized. I don't have to wonder if I have any given resistor because I have then all inventoried.

Looking back, I feel like this project has been one of the most tedious I have ever undertaken. It is taking a lot of will power to not give up until every last resistor I own is properly organized. I mean... I think I went kinda crazy there for a while. Who pays that much attention to their resistor collection? I'm just glad I got it over with...

Android Studio, and Ubuntu 64bit

The next phase of this project is to get the voice search functioning. I am going to use my Android phone for the approach. The idea is to write an app that will use google's speech input API to interpret my speech. Then using tags for each entry, some manually entered, some automatically associated based on component data, search the database for the part. Once a part or list of possible parts is found, provide a "light it up" button that shows where the part is located. Upon clicking the button an address request is sent via the audio connection over bluetooth to the receiver. On it's way out of the bluetooth receiver the audio signal is picked up by the Arduino receiver which is programmed to recognize the FSK message. It relays the address via 315MHz radio to the orgranizer which then lights up the box.

Getting the app ready starts with learning how to write apps for Android, and getting my IDE all set up. I chose to go with Android Studio, because Google's IO conference debut of the IDE looked amazing.

After downloading and "installing" I started it up and started working through a tutorial on the speech recognition API. Android studio has this nice auto compile feature that tells you if your code works. When my setup tried to compile I ran into some issues which I posted, then solved, here.

Monday, September 2, 2013

Django, Dajax, and Getting Dajax to work (Lite-Em-Up Component Organizer)


After futzing around with all sorts of database application approaches, I finally settled with Django. Here's what I messed with:
  1. Pure sqlite implementation with python application for interface
  2. MS Access Database
  3. Finally, Django
The pure approach at the top is probably what I'd end up going for if I ever wanted to mass produce this, MS Access was painful and nearly useless (it was a dark time in my life). Django has offered the easiest database creation/ application development framework of all three. So it was very attractive for my "just get it done" state of mind. Not only that but most of the interface for working on the database contents has already been made for you (in the admin site). The only thing left for me was figuring out how I was going to use the nice interface to make a box light up... a button perhaps? A button that could run a python script. The tricky part is how do you wire up the button which is in a webpage that Django serves, to run your dead simple python script? And how do you pass the script the number of the box to light up?

The answer is of course, Dajax. Not only will dajax serve me well for this project, but also for all the other projects I have going at work to create database systems for control component management and planning.


Starting from here, I was able to gather that this dajax thing was pretty powerful, and handy. But the example leaves a lot out. I tried going through the setup documentation, but it is also operating from the assumption that you pretty much know what you're doing... Not only that, but there are almost no very basic tutorials for the current (0.9 dajax, 1.5 django) releases. So that's what I am going to write here, somewhat for you, and mostly for me.

How to get a button on a django webpage to execute a python script using dajax:

This tutorial assumes that you already have Django 1.5 setup. I do not claim to be any sort of an expert, my training was in mechatronics and physics, so getting Django to work for me has been an exercise in hacking all the way. That being said, here we go!

Dajax Setup

First install dajax and dajaxice using pip.
 sudo pip install django_dajax  

Then follow these instructions to add dajaxice to your project (required!). Some points for clarification: the to add your dajaxice url to is for the site (not the app!). You probably could have figured that out, but just to save you some questioning.

Finished? Great! Now follow these instructions to install dajax. For clarification this is what I did for the jquery step ( I was confused because I thought you simply add this line to your head:
 {% static "/static/dajax/jquery.dajax.core.js" %}  
 but you need to embed it sensibly like so:
 <script type="text/javascript" src="{% static "/static/dajax/jquery.dajax.core.js" %}"></scrip>  

Also make sure you load the staticfiles lib into the template (place this at the top of the template):
 {% load staticfiles %}  

The meat

Here's my template page all together:
 {% load dajaxice_templatetags %}  
 {% load staticfiles %}  
 <script type="text/javascript" src="{% static "/static/dajax/jquery.dajax.core.js" %}"></script>  
 <script type="text/javascript" src="{% static "/static/eci/jquery-2.0.3.js" %}"></script>  
 {% dajaxice_js_import %}  
 <button onclick="Dajaxice.eci.light_up_box(Dajax.process);">Show me where</button>  

A few things to note:

  • I downloaded the newest jquery from the jquery website here, and placed it in the app folder under static/eci/ where eci is the name of my django app.
  • The onclick call goes as Dajaxice.appname.function in the
Here's my code which resides in my app folder (called eci in my case, for electronics component inventory):

 from dajax.core import Dajax  
 from dajaxice.decorators import dajaxice_register  
 import serial, time  
 def light_up_box(req):  
   dajax = Dajax()  
   dajax.alert("Hello World!")  
   serial_port = serial.Serial('/dev/ttyUSB0', 9600, timeout=None)  
   box_number = 4  
   number = str(box_number)  
   if box_number <10:  
     number = '0'+number+'X'  
     number = number+'X'  
   return dajax.json()  

I use the dajax.alert function to confirm independently that the dajax function was called. I'll probably use it later to confirm that the box was found and the part removed (ask for a quantity or something).

That's the long and short of it. I hope it helps you. I was really sad when I couldn't find a basic tutorial that explained the stupid questions I had, hopefully you don't have the same problem!

Wrap up

So as of last night the electronics component organizer has been improved a bit. I have a larger current power supply connected, which will hopefully help with the power issues I've been having. I have also finally secured my pcbs and power supply to the top of the box so it's not so janky. Also, now that I have this dajax thing working I can finally start putting together the rest of the database viewing pages that will allow me to light up boxes at will.

Thoughts for the future:

I think I might have made a mistake trying to get my computer to stream the FFT data to the arduino over serial. I may end up making a new pcb that has an audio jack pass through so that the FFT can be done directly on the arduino. Once the basic database is done I'll probably go ahead and populate the organizer with parts. That will probably take a good part of a Saturday.

Thursday, May 16, 2013

Lite-It-Up Component Organizer.

So it's been a long time. Actually over a year since I last posted about this project. There has been progress. Unfortunately for the past several months it's kind of been on the backburner. Between working on my house, working at SLAC, and nailing down my rhythm I haven't had mad this project a high priority. Hopefully the blog post means that will change soon. Enough words, let's get down to what's happened.

Where we were

So I had just finished a quick test with some foil to see if the idea of painting the drawers would serve to turn them into an effective light tube. It worked. I had also bought the LEDs and the driver chips. I made a proto-board version of the backplane, and that sucked, so I knew I needed to find a better solution. Enter Halted Electronics.

The backplane and mounting

So I found this awesome proto-board at Halted that was basically tinned copper clad on one side, with predrilled holes in it (just like regular protoboard). I mapped out on the boards where the LEDs would sit and cut out some isolated areas on the board. I forget if the rest of the board was Vcc or Gnd... Then all I had to do was solder the LEDs, and the connector, and run wire back to the connector. The other LED legs were all on a common plane.

And, yes, it took a couple of weekends to do.

For mounting the board to the drawer holder I just made sure to line them up a bit and drilled a hole in the back. Then I used a couple of regular screws and a screw driver and fixed them to the back.

Painting the drawers

Probably the longest and most arduous task in this project. To assist in the process I started out using these pieces of cardboard, but that was too slow. I ended up just using masking tape and spending a few hours taping up all the drawers. 64 drawers later I had them all painted. Then it was time to do it again!

LED Drivers and whatnot

As discussed previously the LEDs are driven using the TLC5940. My prototype uses 4 of them, one for each board, and an arduino to control them. Most of the software was already written so I designed a PCB and sent it out to the board house.


Once all of the wires were soldered, and the boards populated I went about mapping the LEDs in the Arduino. With a properly mapped LED matrix I was able to start working on the screensaver. One really neat thing about this box system is that it's also great for parties.
I wanted to have the drawers light up along with music from my computer. Being that I like to do things the hard way I decided that it would be awesome to have the computer stream the results of a Fourier transform done in soft real-time. After searching around on the internet for a while I found a visualizer that I could easily hack to suit my needs. 
Impulse is a simple music visualizer for Ubuntu that looks good by itself. I liked it because it was already set up to interface with Pulseaudio. Pulseaudio is an audio software layer in Ubuntu, which has the nice feature of automatically creating a soft channel monitor for every output stream it handles. This is the software output monitor that Impulse uses to pass the audio stream to its FFT. From there it was a simple matter of importing impulse, calling the function, and massaging the output into my array -> serial protocol -> serial com link to the arduino(child's play in python).
Once I had the array in the arduino (the array is basically 8 numbers that represent the energy in each frequency band) it was straightforward to control the LEDs.

One issue that I had/have was timeout. Sometimes the serial link will have too much information passed down it. One symptom was that the device would simply quit working. To fix this I increased the baud rate, and turned on the watchdog feature for the arduino. Now if the arduino hangs because of serial issues, the watchdog will notice and restart the arduino. Here's more info on the watchdog features of the arduino: Arduino Watchdog.

I also had to do some tweaking to have the equalizer be more 'active'. Here's a video of what the system behaved like before the tweaking: pre-tweak.

After working with it a bit, this is what I have...

So what's next?

I have some time tonight. With the Maker Faire coming up on Sunday I am reminded that I have a bunch of pretty cool ideas backed up right now because I haven't finished this thing. Motivation is at an all time high, so I think I'll try and ride the wave all the way back into shore. First thing is first. I will need to finish development of my parts database. I want to have a good interface that will allow me to quickly add parts, reorganize the drawers, keep track of quantity, etc... I consider this step done when I have the database somewhat populated, I can complete searches using simple phrases like "1.uF," and the result will be returned via a lit up drawer.

After that is completed (I think about a week or so), I want to start making my android app. The android app is where I'll get voice search capabilities from. 

As a final stroke I will also include a feature that will allow a user to build a BoM and have the system automatically light up drawers so that parts can be retrieved all at once. Additionally this thing will alert you when you run low on a part so you'll know to buy more.

Saturday, May 26, 2012

Remote power button

Visual layout

I recently set up my projector to enjoy movies more comfortably from my couch. To do this I decided it would be awesome to "hide" my projector inside of the closet that comprises almost an entire side of my apartment. I wanted the video quality from my computer to be halfway decent on the projector, so I decided to move the computer into the closet as well so it could sit next to the projector, and maintain a short run of cable. At the same time, I still wanted to be able to sit at my desk and use my keyboard and monitor like a normal computer.

Room Layout
This was accomplished by getting some cables from But I still had the nagging problem of always walking into my closet, and reaching to the shelf my PC is on, to turn on the computer. So I decided to install a remote power button into my desk, using a button I had bought for this exact purpose a couple of years ago. It is a particularly awesome button.

I spend some time this past week drilling a hole into my desk (which is basically extruded steel), into which I could mount the button. My first attempts consisted of using regular drill bits followed by a Dremmel to widen the hole (to .75 in). It only took me about 10 minutes of grinding away at the desk to realize I was using the wrong tool. So I went online and ordered a set of step drill bits. They arrived last night.

I managed to finish the job in another ten minutes. Nothing feels quite like using the right tool for the job. I mounted the button and proceeded to wire it up. Crossing my fingers and praying that the computer circuit just needed a normally open dry contact, I attached the button (I had left my multimeter at work), in parallel with the PC button pins. It worked!

Another lesson learned long ago, is conscientious wiring. I used a screw terminal on the outside of the PC case, followed by jumpers inside. So if I ever need to open my case later, I can just disconnect the wires inside to not be bothered.

I finished the wiring job and cleaned off my desk (killing two birds with one stone), to take some pictures. I really like how clean my setup is now (in both senses).

Friday, May 4, 2012

Light-It-Up Component Organizer

I got the idea for doing this from this guy: . After seeing that video and learning about the TLC5940NT (a 16 bit PWM driver which can be daisychained to even greater orders), I felt inspired to finally clean up my lab supplies. I have too often felt disrupted by not having parts when I thought I did, and not knowing where those parts were when I needed them. With this system I will hopefully solve both problems, as my entire inventory will need to be recorded, and organized.

Designing the PCB's for the control circuit was simple enough. I just followed these diagrams: The main challenge so far in the project is keeping costs down while retaining a higher level of quality than I have adhered to in my projects so far. I have a bit more money to invest in this project, so I intend to make it as clean as possible.

Here's a picture of my proof-of-concept. The trouble with clear drawers is the light cross-talk between drawers when they are lit from the rear. To solve this issue I taped some aluminum foil around the drawer like a hotdog bun.  This solves the issue in the left, right and down directions, but all the drawers above it are still lit. So by adding a second hotdog bun to the drawer above, the original drawer becomes a light-pipe of sorts. I really like how the drawer is fully lit, with the light completely diffused. Of course when the idea is actually implemented I will be using a silver paint and not foil...

I also tested how the concept worked when actually filled with items and I am pleased to say that it works quite well. More on that later.

I started working on the LED backplane last night. I had designed a PCB to accomplish the task, but actually having the PCB made was going to be far too expensive. So I went ahead and tried to etch my own board. The size of the design proved to be too much for the process. I could not get the resist layer to adhere very well to the clad. So I have started making the PCB manually from proto board. It is about as much fun as counting grains of sand. To ease (or add to?) the tedium I bought "Atlas Shrugged" on audiobook. So far, 8 LED's (out of 64) are connected... this project will take some time...

Monday, April 30, 2012

Tesla Coil Overhaul

I spent some time this past weekend fixing my Tesla coil named Zeus. It is a 450W SRSG (synchronous rotary spark gap) coil that can usually produce sparks up to 800cm. Around a month or two ago I was running the coil, having just replaced the transformer (an old Franceformer), when a capacitor failed. The capacitor might have failed due to bad timing on my spark gap (some adjustment will be necessary in the future), or due to my gross oversight in the original design. The original capacitor bank was designed such that the first capacitor and last capacitor in the series were very close to each other. The potential became great enough that the first capacitor failed in a spectacular display of electrical arcing to the leads of the last capacitor.

I have since overhauled the entire capacitor bank, changing the design to keep distances maximized. While I was finishing the capacitor bank I decided to go ahead and overhaul the entire coil's wiring. Prior to the decision I had been using crimp terminals to connect wires on the coil. These shoddy connections caused the coil to be very unreliable. A demonstration item like a Tesla coil should work perfectly every time. To have it fail as a result of bad connectivity is disappointing.

After finishing the overhaul of the capacitor bank and the coil wiring I was ready to test. Usually when testing the coil I can count on the thing to refuse to start for a bit. Often it starts at around 80-90VAC as read on a variac, however, with the recent improvements the coil consistently started around 40V. I was able to achieve sparks at this voltage which was a first.

I expect that I will not have trouble with starting the system. Next I will need to develop a better spark gap but until then...