Coded Structure

"In the beginning, [...] God said 'let there be light'".

Some time later, Man discovered Gallium Arsenide, and eye-piercing LEDs have blighted us ever-since.  (Although they used to be quite dim, actually).  Nevertheless, the 'hello world' of the embedded world is to make an LED flash.

In part 1 I 'covered' installation and test of the libftdi module, but the program didn't actually do anything.  The UM232R module is primarily a UART module (unlike the DLP USB245M, which is a FIFO), but as with most recent FTDI devices it supports a 'BitBang' mode.  BitBang is embedded jargon for running a protocol at the digital bit level, specifying in firmware at what point each of the (potentially many) IO lines goes high or low.

To put these devices in to this mode, an API call is needed to the FTDI driver to establish this mode.  The ftdi_set_bitmode function takes three parameters; the FTDI control context (by reference), a direction bitmap, and a mode indicator (which should be 1 for async bitbang mode).  The interesting is the direction bitmap.  For each of the 8 IO lines, the corresponding bit in this mask can be a '1' to indicate an output, or a '0' for an input.  I'm starting with output, so...

if ((ret = ftdi_set_bitmode(&ftdic, 0xFF, 0x01)) < 0) {
        fprintf(stderr, "unable to set bitmode\n");
        return EXIT_FAILURE;
    }

(this fits into the 'DO STUFF HERE' part of the code from last time)

Once this is done, we can start writing things out to the data port.

unsigned char x = 0;
    while (1) {
        ftdi_write_data(&ftdic, &x, 1);
        x ^= 0xFF;      /* toggle all lines */
        usleep(100000)  /* pause for 100ms  */
    }

This is hackish - it never exits for a start - but it should enable an LED to flash!

In terms of wiring things up, the 'GND' pin needs to go to the shorter leg of the LED (the cathode), and a data line (it doesn't matter which in this example, use DB0 to be safe) goes to the longer lead (anode).  A 1K resistor needs to be in either line to limit the current and prevent burning out the LED. As supplied, the jumpers on the UM232R/UM245R module configure it to be self powered. Anyway, it should look something like this, and the LED should be flashing at approx 5Hz.

To do anything more interesting, let's move to Python.  This is where the fun begins - with ctypes. I remember the first time I used ctypes, using puts from the C standard library on a Solaris machine, and the fact that it just worked. Occasionally there is a need to write the equivalent of a function prototype, but most of the time things are incredibly easy.

First up: loading the shared library. Lets guess its name - libftdi.so?

[GCC 4.1.2 20061115 (prerelease) (Debian 4.1.1-20)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> from ctypes import *
>>>
>>> fdll = CDLL('libftdi.so')
>>>
>>> fdll
<CDLL 'libftdi.so', handle 81da028 at b7cfd4ac>
>>>

So far so good :) Next thing is this ftdi_context struct which is referenced by each function in the C code. It starts as an unpopulated structure, and presumably the first functions (init and usb_open) set it up as required.  While the D2XX interface just gives an integer handle, in libftdi we have to pass pointers to this structure around.  So, how do we define this in ctypes?
First we need its size:

#include "ftdi.h"
#include "stdio.h"
int main()
{
    printf ("%d\n", sizeof(struct ftdi_context));
}

Now compile and run...

/home/user/ftdi> gcc get_size.c
/home/user/ftdi> ./a.out
84

So, we need 84 bytes of storage for this context. ctypes will do that for us:

>>> ftdic = create_string_buffer(84) 

Done that. Now we can rattle it off: only two other things: first, where in C you give &var to give the address of a variable (e.g. as a parameter to a function taking *var), in Python with ctypes, it's byref(var). Second, to create a something which has existence in the C world, (e.g. so can have its address taken), there are a whole bunch of c_XXX functions - we'll use c_byte, which is equivalent to an unsigned char.

from ctypes import *

fdll = CDLL('libftdi.so')

ftdic = create_string_buffer(84)

fdll.ftdi_init(byref(ftdic))

fdll.ftdi_usb_open(byref(ftdic), 0x0403, 0x6001)

fdll.ftdi_set_bitmode(byref(ftdic), 0xFF, 0x01)

import time
x = c_byte(0)
try:
    while True:
        fdll.ftdi_write_data(byref(ftdic), byref(x), 1)
        time.sleep(0.1)
        x.value ^= 0xFF # toggle all bits
except KeyboardInterrupt:
    pass

fdll.ftdi_usb_close(byref(ftdic))

fdll.ftdi_deinit(byref(ftdic))

And we're done! It's surprising how easy it is to translate basic C code using libraries into Python using ctypes - normally it just works.

Next time we'll look at interfacing to an LCD module...

I waited till primary school to read this book, but it seems that 'modern' kids can understand these things sooner - or at least Amazon thinks so...

I'm going to do a series of blog posts on using FTDI devices to access the outside world. There are probably a dozen other similar series out there, so I hope I can introduce enough novelty to make it interesting

I've been interested in low-level programming for as long as I've been programming (anyone else remember this book? - yes, a "children's" book on Machine Code...)

In terms of 'physical computing', things like Arduino are really taking off at the moment, but I'm going to take a step back to the simplicity of simple digital IO based on the BitBang mode of FTDI's latest devices.  There are two reasons for this: firstly it came seem laborious writing two sets of software (for both the host computer and the target micro-controller), and secondly that even if the eventual application is going to be a standalone micro based system, it is still generally quicker to prototype things using only the host computer, avoiding the cross-compile and firmware upload cycle.

Hardware-wise, I'm using a FTDI UM232R (Farnell link) device.  (I've also used one of these, and this either will be usable) This is a DIL module which plugs nicely into a breadboard which can be used to interface with stuff.  If you want to follow along, get a breadboard to plug it into, some LEDs, 1Kohm resistors, and some connecting wire (I like these, but they are waaay over priced).  In a couple of posts I'll be using one of these (N26AZ), too...

On the software side, I'm using Linux on a EEEPC 701 (stock Xandros) with the libftdi drivers, compiled from source found here.  The FTDI supplied drivers are similar and might be a better choice on Windows (not sure if libftdi works on Windows), but some of the function names and interfaces differ somewhat.

First step is getting the simple.c program compiled and working.  The program outline - slightly edited for length - looks something like this.

/* see http://www.intra2net.com/en/developer/libftdi/documentation/ */ 
#include "stdio.h"
#include "ftdi.h"

int main(void)
{
    int ret;
    struct ftdi_context ftdic;
    if (ftdi_init(&ftdic) < 0) {
        return EXIT_FAILURE;
    }

    if ((ret = ftdi_usb_open(&ftdic, 0x0403, 0x6001)) < 0) {
        return EXIT_FAILURE;
    }

    /* DO STUFF HERE */

    if ((ret = ftdi_usb_close(&ftdic)) < 0) {
        return EXIT_FAILURE;
    }

    ftdi_deinit(&ftdic);

    return EXIT_SUCCESS;
}

Compiling this (gcc -o simple simple.c -lftdi) and running the resulting executable should not cause any errors, and should return a successful error code if a FTDI device is attached.

This post is getting long enough for now, so I'll leave it at that. Next time will be configuring bitbang mode, where we can use the device as a configurable 8-bit IO port, blinking lights, and the wonders of Python's ctypes module...

When I was a teenager, I discovered something amazing. A pen that made writing fun, desirable, and (almost) exciting. A Pilot V5. They never lasted long, but maybe that was because I'd actually write stuff with them. Time moved on, and I grew a penchant for nice Parker pens (Frontier rollerball please...), except that I'd get one for Christmas and have lost it by my birthday only a month later, and need another one. Anyways... Moving on more than a few years, I've come to realise that having something which makes writing attractive is more than just a nice incidental thing, but an important consideration.

At least for me, writing is a major way of organising and clarifying my thoughts, and I quite enjoy writing random streams of consciousness on a subject and later reviewing it to see if any interesting things are in there. If writing is fun, I'll do more of it, and having a decent writing instrument does make it fun.

Nowadays, I don't use a pen much (and my handwriting seems to be getting worse from lack of practice), but I still use a writing instrument - my keyboard. And just as there is a great difference between a free biro and a decent pen, so there is with keyboards, if not more so. A decent keyboard makes 'writing' fun, and that is a good thing. My current choice is a small Apple keyboard which is nice and portable, so I can plug it into whichever computer I happen to be using (it's even worth the annoyance of not having a delete key on a PC), and enjoy fine typing, perhaps even leading to some fine thoughts...

I'm playing with some midi based Python stuff at the moment, doing raw file access to /dev/midi1 (midisport uno), and most of the time it's just sitting on a file.read(1). In Windows, the user can break into this with Ctrl-Break (Ctrl-C doesn't work on Windows or Linux), but there doesn't seem to be an equivalent to Ctrl-Break in the Linux world. So it's Ctrl-Z to put it in the background, ps -a, find the program, then terminate it hastily with sudo kill -9. All of which is a bit excessive. Rather than put something proper in there to terminate it (is select() the way to do these things on Linux?), I decided to get the wrong way a little less wrong.
I now have a half-way house to getting it a bit simpler:

alias killpy="sudo kill -9 \`ps -a | grep python | head -c 6\`"

And it's then just Ctrl-Z, killpy. I'm sure this isn't the best way of doing things, but I like the fact that it does things the same way I do, but using pipes to communicate, instead of the eye-brain-hand-keyboard loop. And it's the first time I've used the -c option on head (number of bytes to copy to stdout), so I've learnt something!

Of course it doesn't work if there is more than one Python process, or whatever, and yes I do need the sudo and -9.

Names are important, and doubly so in programming. So when a programmer sees 'list', they think they know what that means, and similarly when they see 'array'.
The fundamental difference would be something like this:

array performance:
random access - O(1)
insertion / deletion of known element - O(n)

list performance:
random access - O(n)
insertion / deletion of known element - O(1)

The performance guarantees of a programming languages' data structures form part of the functional specification of that type, not just some incidental extra information.

I bumped into this when using streams of bytes (represented as lists of integers each < 256) with the following codec-like code:

def process(packet):
    itemlen = work_out_length(packet)
    item, packet = packet[:itemlen], packet[itemlen:]
    # do something with item
    return packet

packet = some data
while packet:
    packet = process(packet)

which is equivalent to this...

a = some data
while a:
    head, a = a[0], a[1:]
    process(head)

(The actual problem wasn't as easy to solve as the above case, as this assumes that the 'head' item is always a single byte; in reality it could be any number of bytes, and the packet would have to be streamed using multiple recursive loops like the above to process it.  But the fundamentals are the same.)

Anyway, it all works fine until a large packet arrives.  And then the interactive program suddenly stops; what took on the order of a millisecond suddenly takes half-an-hour, which to any user looks like the program has crashed.

This is a functional programming idiom, but it just doesn't work with Python lists in the general case of large lists.  It didn't just slow it down, it completely broke it.

Solutions in this specific case are deques (collections.deque) or using iterators. But that's for another time...


In the C++ STL, part of the specification is a performance guarantee for each algorithm on each container type (http://www.sgi.com/tech/stl/complexity.html).  In anything other than toy programs this information is critical, and they give the C++ developer an additional criteria in selecting the appropriate collection types.  It changes 'worse/better' into 'wrong/right'.  'If [these specifications] are ignored [in an STL implementation], the performance of the resulting program will often render it useless.' - from previous link.  The very separation of algorithms and data structures which the C++ STL enables (see the Elements of Programming book for a up-to-date discussion of the underlying paradigm of the STL - without STL-specific code), makes possible the enumeration of performance guarantees (other than specifying it for every function in every types' API). So while the Python documentation for lists warns that inserts and deletes at the beginning are O(n), this information isn't part of a coherent bigger picture which guides me to the right methods and data structures.

So it is 1:00 in the morning, and our youngest son wakes up screaming. At 18 months it isn't always clear what the problem is, but with a bit of attention he soon settles. Except the same thing happened an hour ago, though my wife got up then. And, almost exactly an hour later, he wakes again, and does his 'muuuuu-meeeeee' type noises in-between crying. Except I've woken up first, which sort of means it's my job to go to him, again... And at this point I remember that vital software principle: don't repeat youself (DRY). The tempting thing is to give him back his dummy, give him a cuddle, and within 2 minutes he could be back asleep - and in 3 so could I. This is the bet I am making: there's a small chance he'll sleep through the rest of the night. It's always possible. But far more likely is that he'll wake again, and I won't get much sleep at all tonight. Because if he woke 3 hours on the trot when he normally sleeps through without problem, there's probably a reason - maybe even a reason I could fix (bets on a dirty nappy?) But I'm tired, and tiredness makes me even more lazy than usual, and... I hope he settles and I go back to bed.

As in many areas of life, software developers continually have to make the choice between short-term ease against the risk of long-term disaster. If the disaster was certain, the choice would be clear, if not easy. But there is always the chance that it will never happen, and if the cost of averting that potential disaster is significant (e.g. lost business due to competition in time-to-market), it is no longer clear-cut. But each time the risk is seen and ignored, the likelihood of getting it done right decreases. If I get up at each hour from 1am till 5am to settle my son, am I really going to bother doing anything different at 6am?

So what are we to do? Recognise the need early, when the cost is least and the confidence of knowing that the potential disaster has already been averted can have the longest effect. Make the commitment early, not counting the short-term effort as a cost, but as a decision well-made.

I leave the quantitative analysis to Ian Fleming:
'Once is happenstance, Twice is coincidence, The third time is enemy action'
- Ian Fleming, Goldfinger

Enemy action must be countered with force of will, or we shall be defeated.